The approx_percentile function in Spark is a performance-optimized alternative to percentile. It takes an optional accuracy parameter:
approx_percentile(column, percentage, accuracy)
Higher accuracy values → more precise results, but increased memory/computation.
Lower values → faster but less accurate.
From the documentation:
"Increasing the accuracy improves precision but increases memory usage." Final answer: D

To sort a PySpark DataFrame by multiple columns with mixed sort directions, the correct usage is:
python
CopyEdit
df.orderBy("age", "salary", ascending=[True, False])
age will be sorted in ascending order
salary will be sorted in descending order
The orderBy() and sort() methods in PySpark accept a list of booleans to specify the sort direction for each column.
Documentation Reference: PySpark API - DataFrame.orderBy

To persist a table and specify the save path, use:
users.write.option("path", "/some/path").saveAsTable("default_table")
The .option("path", ...) must be applied before calling saveAsTable.
Option A uses invalid syntax (write(path=...)).
Option B applies .option() after .saveAsTable()-which is too late.
Option D uses incorrect syntax (no path parameter in saveAsTable).

The correct approach to perform a parallelized groupBy operation across Spark worker nodes using Pandas API is via applyInPandas. This function enables grouped map operations using Pandas logic in a distributed Spark environment. It applies a user-defined function to each group of data represented as a Pandas DataFrame.
As per the Databricks documentation:
"applyInPandas() allows for vectorized operations on grouped data in Spark. It applies a user-defined function to each group of a DataFrame and outputs a new DataFrame. This is the recommended approach for using Pandas logic across grouped data with parallel execution." Option A is correct and achieves this parallel execution.
Option B (mapInPandas) applies to the entire DataFrame, not grouped operations.
Option C uses built-in aggregation functions, which are efficient but not customizable with Pandas logic.
Option D creates a scalar Pandas UDF which does not perform a group-wise transformation.
Therefore, to run a groupBy with parallel Pandas logic on Spark workers, Option A using applyInPandas is the only correct answer.

Adaptive Query Execution (AQE) in Spark 3.x automatically optimizes query plans at runtime based on the actual data characteristics observed during job execution.
Key features of AQE include:
Dynamic switching of join strategies: Changes between sort-merge join and broadcast join based on actual shuffle sizes.
Coalescing shuffle partitions: Reduces small tasks and improves parallelism efficiency.
Handling skew joins: Dynamically splits large partitions to avoid data skew.
Thus, the most accurate answer describing AQE's function is "dynamically switching join strategies." Why the other options are incorrect:
B: Table statistics are collected manually or by the metastore, not by AQE.
C: AQE benefits multi-stage jobs involving shuffles, not single-stage jobs.
D: Delta file optimization is handled by Databricks utilities, not AQE.
Reference:
Databricks Exam Guide (June 2025): Section "Troubleshooting and Tuning Apache Spark DataFrame API Applications" - covers AQE and its benefits.
Spark 3.5 Release Notes - Adaptive Query Execution dynamic optimizations.