The function date_sub(start, days) subtracts the number of days from the start date. If a negative number is passed, the behavior becomes a date addition.
Example:
SELECT date_sub('2024-05-01', -5)
-- Returns: 2024-05-06
So, a negative value effectively adds the absolute number of days to the date.

To use a Python function as a UDF (User Defined Function) in Spark SQL, it must first be registered using spark.udf.register().
Correct usage:
spark.udf.register("cube_func", cube_func)
num_df.selectExpr("cube_func(num)").show()
This registers cube_func as a callable SQL function available in expressions or queries.
Why the other options are incorrect:
B: You must wrap with udf() or selectExpr; calling plain Python functions won't work.
C: createDataFrame is for building DataFrames, not calling UDFs.
D: DataFrames cannot directly register UDFs.
Reference:
PySpark SQL Functions - spark.udf.register() and selectExpr().
Databricks Exam Guide (June 2025): Section "Using Spark SQL" - user-defined functions and Spark SQL integration.

The default broadcast join threshold in Spark is:
spark.sql.autoBroadcastJoinThreshold = 10MB
Since df2 is only 8 MB (less than 10 MB), Spark will automatically apply a broadcast join without requiring explicit hints.
From the Spark documentation:
"If one side of the join is smaller than the broadcast threshold, Spark will automatically broadcast it to all executors." A is incorrect because Spark does support auto broadcast even with static plans.
B is correct: Spark will automatically broadcast df2.
C and D are incorrect because Spark's default logic handles this optimization.
Final answer: B

Adaptive Query Execution (AQE) is a powerful optimization framework introduced in Apache Spark 3.0 and enabled by default since Spark 3.2. It dynamically adjusts query execution plans based on runtime statistics, leading to significant performance improvements. The key benefits of AQE include:
Dynamic Join Strategy Selection: AQE can switch join strategies at runtime. For instance, it can convert a sort-merge join to a broadcast hash join if it detects that one side of the join is small enough to be broadcasted, thus optimizing the join operation .
Handling Skewed Data: AQE detects skewed partitions during join operations and splits them into smaller partitions. This approach balances the workload across tasks, preventing scenarios where certain tasks take significantly longer due to data skew .
Coalescing Post-Shuffle Partitions: AQE dynamically coalesces small shuffle partitions into larger ones based on the actual data size, reducing the overhead of managing numerous small tasks and improving overall query performance .
These runtime optimizations allow Spark to adapt to the actual data characteristics during query execution, leading to more efficient resource utilization and faster query processing times.

In Structured Streaming, checkpoints store state information (offsets, progress, and metadata) needed to resume a stream after a failure or restart.
Correct usage:
Set the checkpointLocation option when writing the streaming output:
streaming_df.writeStream \
.format("delta") \
.option("checkpointLocation", "/path/to/checkpoint/dir") \
.start("/path/to/output")
Spark uses this checkpoint directory to recover progress automatically and maintain exactly-once semantics.
Why the other options are incorrect:
A/D: recoveryLocation is not a valid Spark configuration option.
B: Checkpointing must be configured in writeStream, not during readStream.
Reference:
PySpark Structured Streaming Guide - Checkpointing and recovery.
Databricks Exam Guide (June 2025): Section "Structured Streaming" - explains checkpointing and fault-tolerant streaming recovery.