To read from a specific Kafka topic using Structured Streaming, the correct syntax is:
python
CopyEdit
.option("subscribe", "feed")
This is explicitly defined in the Spark documentation:
"subscribe - The Kafka topic to subscribe to. Only one topic can be specified for this option." (Source: Apache Spark Structured Streaming + Kafka Integration Guide)
"subscribe - The Kafka topic to subscribe to. Only one topic can be specified for this option." (Source: Apache Spark Structured Streaming + Kafka Integration Guide) B . "subscribe.topic" is invalid.
C . "kafka.topic" is not a recognized option.
D . "topic" is not valid for Kafka source in Spark.

In Apache Spark, the execution hierarchy is structured as follows:
Application: The highest-level unit, representing the user program built on Spark.
Job: Triggered by an action (e.g., collect(), count()). Each action corresponds to a job.
Stage: A job is divided into stages based on shuffle boundaries. Each stage contains tasks that can be executed in parallel.
Task: The smallest unit of work, representing a single operation applied to a partition of the data.
When the collect() action is invoked, Spark initiates a job. This job is then divided into stages at points where data shuffling is required (i.e., wide transformations). Each stage comprises tasks that are distributed across the cluster's executors, operating on individual data partitions.
This hierarchical execution model allows Spark to efficiently process large-scale data by parallelizing tasks and optimizing resource utilization.

In Spark, the default join type is an inner join, which returns only the rows with matching keys in both DataFrames. To retain all records from the left DataFrame (purch_df) and include matching records from the right DataFrame (cust_df), a left outer join should be used.
By specifying the join type as 'left', the modified code ensures that all records from purch_df are preserved, and matching records from cust_df are included. Records in purch_df without a corresponding match in cust_df will have null values for the columns from cust_df.
This approach is consistent with standard SQL join operations and is supported in PySpark's DataFrame API.

When queries frequently filter on certain columns, partitioning by those columns ensures partition pruning, allowing Spark to scan only relevant directories instead of the entire dataset.
Correct code:
final.write.partitionBy("event_year", "event_month").parquet("events.liveLatest") This improves read performance dramatically for filters like:
SELECT * FROM events.liveLatest WHERE event_year = 2024 AND event_month = 5; bucketBy() helps in clustering and joins, not in partition pruning for file-based tables.
Why the other options are incorrect:
B: Bucket count changes parallelism, not query pruning.
C: sortBy organizes data within files, not across partitions.
D: Partitioning by only one column limits pruning benefits.
Reference:
Spark SQL DataFrameWriter - partitionBy() for partitioned tables.
Databricks Exam Guide (June 2025): Section "Using Spark SQL" - partitioning vs. bucketing and query optimization.

In Spark, a shuffle occurs when data needs to be redistributed across partitions or nodes, typically due to operations that require grouping, joining, or sorting.
groupBy triggers a shuffle because it aggregates data based on key values, requiring data from multiple partitions to be moved across the cluster.
Operations like filter and select are narrow transformations (no shuffle).
coalesce can reduce the number of partitions without a full shuffle (unlike repartition).
Why the other options are incorrect:
A (filter) - Narrow transformation, no shuffle.
B (select) - Simple projection, no data movement.
D (coalesce) - Reduces partitions locally without shuffle.
Reference:
Spark Architecture - Transformations and Actions; narrow vs. wide transformations.
Databricks Exam Guide (June 2025): Section "Apache Spark Architecture and Components" - explains shuffle operations and execution hierarchy.