Comprehensive and Detailed Explanation From Exact Extract:
The default broadcast join threshold in Spark is:
spark.sql.autoBroadcastJoinThreshold = 10MB
Sincedf2is only 8 MB (less than 10 MB), Spark will automatically apply a broadcast join without requiring explicit hints.
From the Spark documentation:
"If one side of the join is smaller than the broadcast threshold, Spark will automatically broadcast it to all executors." A is incorrect because Spark does support auto broadcast even with static plans.
B is correct: Spark will automatically broadcast df2.
C and D are incorrect because Spark's default logic handles this optimization.
Final Answer: B

Comprehensive and Detailed Explanation From Exact Extract:
The provided code defines a Pandas UDF of type Series-to-Series, where a new instance of the language modelis created on each call, which happens per batch. This is inefficient and results in significant overhead due to repeated model initialization.
To reduce the frequency of model loading, the engineer should convert the UDF to an iterator-based Pandas UDF (Iterator[pd.Series] -> Iterator[pd.Series]). This allows the model to be loaded once per executor and reused across multiple batches, rather than once per call.
From the official Databricks documentation:
"Iterator of Series to Iterator of Series UDFs are useful when the UDF initialization is expensive... For example, loading a ML model once per executor rather than once per row/batch."
- Databricks Official Docs: Pandas UDFs
Correct implementation looks like:
python
CopyEdit
@pandas_udf("string")
def translate_udf(batch_iter: Iterator[pd.Series]) -> Iterator[pd.Series]:
model = get_translation_model(target_lang='es')
for batch in batch_iter:
yield batch.apply(model)
This refactor ensures theget_translation_model()is invoked once per executor process, not per batch, significantly improving pipeline performance.

Comprehensive and Detailed Explanation From Exact Extract:
To sort a PySpark DataFrame by multiple columns with mixed sort directions, the correct usage is:
python
CopyEdit
df.orderBy("age","salary", ascending=[True,False])
agewill be sorted in ascending order
salarywill be sorted in descending order
TheorderBy()andsort()methods in PySpark accept a list of booleans to specify the sort direction for each column.
Documentation Reference:PySpark API - DataFrame.orderBy

To persist a table and specify the save path, use:
users.write.option("path","/some/path").saveAsTable("default_table")
The .option("path", ...) must be applied before calling saveAsTable.
Option A uses invalid syntax (write(path=...)).
Option B applies.option()after.saveAsTable()-which is too late.
Option D uses incorrect syntax (no path parameter in saveAsTable).
Reference:Spark SQL - Save as Table

Temp views likecreateOrReplaceTempVieware session-scoped.
They disappear once the Spark session ends.
To retain data across sessions, it must be persisted:
df.write.saveAsTable("users_vw")
Thus, the view needs to be persisted as a table to survive session termination.
Reference:Databricks - Temp vs Global vs Permanent Views