Online Access Free Associate-Developer-Apache-Spark-3.5 Exam Questions

38 of 55.
A data engineer is working with Spark SQL and has a large JSON file stored at /data/input.json.
The file contains records with varying schemas, and the engineer wants to create an external table in Spark SQL that:
Reads directly from /data/input.json.
Infers the schema automatically.
Merges differing schemas.
Which code snippet should the engineer use?

• A.CREATE EXTERNAL TABLE users
  USING json
  OPTIONS (path '/data/input.json', mergeAll 'true');
• B.CREATE EXTERNAL TABLE users
  USING json
  OPTIONS (path '/data/input.json', inferSchema 'true');
• C.CREATE TABLE users
  USING json
  OPTIONS (path '/data/input.json');
• D.CREATE EXTERNAL TABLE users
  USING json
  OPTIONS (path '/data/input.json', mergeSchema 'true');

A data engineer is reviewing a Spark application that applies several transformations to a DataFrame but notices that the job does not start executing immediately.
Which two characteristics of Apache Spark's execution model explain this behavior?
Choose 2 answers:

• A.The Spark engine optimizes the execution plan during the transformations, causing delays.
• B.The Spark engine requires manual intervention to start executing transformations.
• C.Only actions trigger the execution of the transformation pipeline.
• D.Transformations are executed immediately to build the lineage graph.
• E.Transformations are evaluated lazily.

35 of 55.
A data engineer is building a Structured Streaming pipeline and wants it to recover from failures or intentional shutdowns by continuing where it left off.
How can this be achieved?

• A.By configuring the option checkpointLocation during readStream.
• B.By configuring the option recoveryLocation during writeStream.
• C.By configuring the option checkpointLocation during writeStream.
• D.By configuring the option recoveryLocation during SparkSession initialization.

A developer is trying to join two tables, sales.purchases_fct and sales.customer_dim, using the following code:

fact_df = purch_df.join(cust_df, F.col('customer_id') == F.col('custid')) The developer has discovered that customers in the purchases_fct table that do not exist in the customer_dim table are being dropped from the joined table.
Which change should be made to the code to stop these customer records from being dropped?

• A.fact_df = purch_df.join(cust_df, F.col('customer_id') == F.col('custid'), 'right_outer')
• B.fact_df = cust_df.join(purch_df, F.col('customer_id') == F.col('custid'))
• C.fact_df = purch_df.join(cust_df, F.col('customer_id') == F.col('custid'), 'left')
• D.fact_df = purch_df.join(cust_df, F.col('cust_id') == F.col('customer_id'))

A data engineer is asked to build an ingestion pipeline for a set of Parquet files delivered by an upstream team on a nightly basis. The data is stored in a directory structure with a base path of "/path/events/data". The upstream team drops daily data into the underlying subdirectories following the convention year/month/day.
A few examples of the directory structure are:

Which of the following code snippets will read all the data within the directory structure?

• A.df = spark.read.parquet("/path/events/data/")
• B.df = spark.read.option("inferSchema", "true").parquet("/path/events/data/")
• C.df = spark.read.parquet("/path/events/data/*")
• D.df = spark.read.option("recursiveFileLookup", "true").parquet("/path/events/data/")