Explanation
While using an all-purpose cluster is ok during development but anytime you don't need to interact with a notebook, especially for a scheduled job it is less expensive to use a job cluster. Using an all-purpose cluster can be twice as expensive as a job cluster.
Please note: The compute cost you pay the cloud provider for the same cluster type and size be-tween an all-purpose cluster and job cluster is the same the only difference is the DBU cost.
The total cost of cluster = Total cost of VM compute(Azure or AWS or GCP) + Cost per DBU The per DBU cost varies between all-purpose and Job Cluster Here is the recent cost estimate from AWS between Jobs Cluster and all-purpose Cluster, for jobs compute its
$0.15 cents per DBU v$0.55 cents per DBU for all-purpose
Graphical user interface Description automatically generated

How do I check how much the DBU cost for my cluster?
When you click on an exister cluster or when you look at the cluster details you will see this in the top right corner Graphical user interface, text, application, email Description automatically generated

Explanation
The answer is
CREATE DATABASE sales LOCATION 'dbfs:/mnt/delta/databases/sales.db/'
Note: with the introduction of the Unity catalog and three-layer namespace usage of SCHEMA and DATABASE is interchangeable

Explanation
The answer is cloudfiles.schemalocation, checkpointlocation
When reading the data cloudfiles.schemalocation is used to store the inferred schema of the incoming data.
When writing a stream to recover from failures checkpointlocation is used to store the offset of the byte that was most recently processed.

Explanation
1. CREATE OR REPLACE GLOBAL TEMPORARY VIEW view_name
2. AS SELECT * FROM table_name
There are two types of temporary views that can be created Local and Global
*A session-scoped temporary view is only available with a spark session, so another note-book in the same cluster can not access it. if a notebook is detached and reattached local temporary view is lost.
*A global temporary view is available to all the notebooks in the cluster but if a cluster re-starts a global temporary view is lost.

Explanation
Auto Loader supports two modes when ingesting new files from cloud object storage Directory listing: Auto Loader identifies new files by listing the input directory, and uses a directory polling approach.
File notification: Auto Loader can automatically set up a notification service and queue service that subscribe to file events from the input directory.
Diagram Description automatically generated

File notification is more efficient and can be used to process the data in real-time as data arrives in cloud object storage.
Choosing between file notification and directory listing modes | Databricks on AWS