In Spark SQL and DataFrame operations, the configuration parameter spark.sql.shuffle.partitions defines the number of partitions created during shuffle operations such as join, groupBy, and distinct.
The default value (in Spark 3.5) is 200.
If this number is too low, Spark creates fewer tasks, leading to idle executors and poor cluster utilization.
Increasing this value allows Spark to create more tasks that can run in parallel across executors, effectively using more cluster resources.
Correct approach:
spark.conf.set("spark.sql.shuffle.partitions", 400)
This increases the parallelism level of shuffle stages and improves overall resource utilization.
Why the other options are incorrect:
B: Reducing partitions further would decrease parallelism and worsen the underutilization issue.
C: Dynamic resource allocation scales executors up or down based on workload, but it doesn't fix low task parallelism caused by insufficient shuffle partitions.
D: Increasing dataset size is not a tuning solution and doesn't address task-level under-parallelization.
Reference (Databricks Apache Spark 3.5 - Python / Study Guide):
Spark SQL Configuration: spark.sql.shuffle.partitions - controls the number of shuffle partitions.
Databricks Exam Guide (June 2025): Section "Troubleshooting and Tuning Apache Spark DataFrame API Applications" - tuning strategies, partitioning, and optimizing cluster utilization.

In Structured Streaming, only transformation operations are allowed on streaming DataFrames. These include select(), filter(), where(), groupBy(), withColumn(), etc.
Example of supported transformation:
filtered_df = streaming_df.filter("count < 30")
However, actions such as count(), show(), and collect() are not supported directly on streaming DataFrames because streaming queries are unbounded and never finish until stopped.
To perform aggregations, the query must be executed through writeStream and an output sink.
Why the other options are incorrect:
A: count() is an action, not allowed directly on streaming DataFrames.
C: countDistinct() is a stateful aggregation, not supported outside of a proper streaming query.
D: show() is also an action, unsupported on streaming queries.
Reference:
PySpark Structured Streaming Programming Guide - supported transformations and actions.
Databricks Exam Guide (June 2025): Section "Structured Streaming" - performing operations on streaming DataFrames and understanding supported transformations.

The number of tasks is controlled by the number of partitions. By default, spark.sql.shuffle.partitions is 200. If stages are showing very few tasks (less than total cores), you may not be leveraging full parallelism.
From the Spark tuning guide:
"To improve performance, especially for large clusters, increase spark.sql.shuffle.partitions to create more tasks and parallelism." Thus:
A is correct: increasing shuffle partitions increases parallelism
B is wrong: it further reduces parallelism
C is invalid: increasing dataset size doesn't guarantee more partitions D is irrelevant to task count per stage Final answer: A

Spark Connect enables decoupling of the client and Spark driver processes, allowing remote access. Spark supports configuring the remote Spark Connect server in multiple ways:
From Databricks and Spark documentation:
Option B (--remote "sc://localhost") is a valid command-line argument for the pyspark shell to connect using Spark Connect.
Option C (setting SPARK_REMOTE environment variable) is also a supported method to configure the remote endpoint.
Option A is incorrect because Spark Connect uses the sc:// protocol, not https://.
Option D requires modifying the code, which the question explicitly avoids.
Option E configures the port on the server side but doesn't start a client connection.
Final Answers: B and C

When reading CSV files using Spark SQL or the DataFrame API, Spark by default assumes that the first line of the file is data, not headers. To interpret the first line as column names, the header option must be set to true.
Correct syntax:
CREATE TABLE locations
USING csv
OPTIONS (
path '/data/locations.csv',
header 'true'
);
This tells Spark to read the first row as column headers and correctly map columns like city, lat, and long.
Why the other options are incorrect:
B (header 'false'): Default behavior; would keep reading header as data.
C / D (sep): Used to specify the delimiter; not relevant unless the file uses a different separator (e.g., |).
Reference (Databricks Apache Spark 3.5 - Python / Study Guide):
PySpark SQL Data Sources - CSV options (header, inferSchema, sep).
Databricks Exam Guide (June 2025): Section "Using Spark SQL" - Reading data from files with different formats using Spark SQL and DataFrame APIs.