Comprehensive and Detailed Explanation From Exact Extract:
To enable fault tolerance and ensure that Spark can resume from the last committed offset after failure, you must configure a checkpoint location using the correct option key:"checkpointLocation".
From the official Spark Structured Streaming guide:
"To make a streaming query fault-tolerant and recoverable, a checkpoint directory must be specified using.
option("checkpointLocation", "/path/to/dir")."
Explanation of options:
Option A uses an invalid option name:"checkpoint"(should be"checkpointLocation") Option B is correct: it setscheckpointLocationproperly Option C lacks checkpointing and won't resume after failure Option D also lacks checkpointing configuration Reference: Apache Spark 3.5 Documentation # Structured Streaming # Fault Tolerance Semantics

Comprehensive and Detailed Explanation From Exact Extract:
Shuffling occurs in operations likegroupBy,reduceByKey, orjoin-which cause data to be moved across partitions. Therepartition()operation can also cause a shuffle, but in this context, it follows an aggregation.
InOption D, thegroupByfollowed byaggresults in a shuffle due to grouping across nodes.
Therepartition(10)is a partitioning transformation but does not involve a new shuffle since the data is already grouped.
This sequence - shuffle (groupBy) followed by non-shuffling (repartition) - is correct.
Option A does the opposite: thefilterdoes not cause a shuffle, butgroupBydoes - this makes it the wrong order.

Comprehensive and Detailed Explanation:
Broadcast joins work by sending the smaller DataFrame to all executors, eliminating the shuffle of the larger DataFrame.
From Spark documentation:
"Broadcast joins are efficient when one DataFrame is small enough to fit in memory. Spark avoids shuffling the larger table." DataFrame B (1 GB) fits within the default threshold and should be broadcasted.
It eliminates the need to shuffle the large DataFrame A.
Final Answer: B

Comprehensive and Detailed Explanation From Exact Extract:
To flatten an array of structs into individual rows and access fields within each struct, you must:
Useexplode()to expand the array so each struct becomes its own row.
Access the struct fields via dot notation (e.g.,record_exploded.sensor_id).
Option C does exactly that:
First, explode therecordarray column into a new columnrecord_exploded.
Then, access fields of the struct using the dot syntax inselect.
This is standard practice in PySpark for nested data transformation.
Final Answer: C

To read each file as a single record, use:
spark.read.text(path, wholetext=True)
This ensures that Spark reads the entire file contents into one row.
Reference:Spark read.text() with wholetext