Online Access Free Associate-Developer-Apache-Spark Exam Questions

Which of the following statements about storage levels is incorrect?

• A.The cache operator on DataFrames is evaluated like a transformation.
• B.In client mode, DataFrames cached with the MEMORY_ONLY_2 level will not be stored in the edge node's memory.
• C.DISK_ONLY will not use the worker node's memory.
• D.Caching can be undone using the DataFrame.unpersist() operator.
• E.MEMORY_AND_DISK replicates cached DataFrames both on memory and disk.

Which of the following code blocks reads in the JSON file stored at filePath as a DataFrame?

• A.spark.read.json(filePath)
• B.spark.read.path(filePath)
• C.spark.read.path(filePath, source="json")
• D.spark.read().json(filePath)
• E.spark.read().path(filePath)

The code block displayed below contains multiple errors. The code block should remove column transactionDate from DataFrame transactionsDf and add a column transactionTimestamp in which dates that are expressed as strings in column transactionDate of DataFrame transactionsDf are converted into unix timestamps. Find the errors.
Sample of DataFrame transactionsDf:
1.+-------------+---------+-----+-------+---------+----+----------------+
2.|transactionId|predError|value|storeId|productId| f| transactionDate|
3.+-------------+---------+-----+-------+---------+----+----------------+
4.| 1| 3| 4| 25| 1|null|2020-04-26 15:35|
5.| 2| 6| 7| 2| 2|null|2020-04-13 22:01|
6.| 3| 3| null| 25| 3|null|2020-04-02 10:53|
7.+-------------+---------+-----+-------+---------+----+----------------+ Code block:
1.transactionsDf = transactionsDf.drop("transactionDate")
2.transactionsDf["transactionTimestamp"] = unix_timestamp("transactionDate", "yyyy-MM-dd")

• A.Column transactionDate should be dropped after transactionTimestamp has been written. The string indicating the date format should be adjusted. The withColumn operator should be used instead of the existing column assignment. Operator to_unixtime() should be used instead of unix_timestamp().
• B.Column transactionDate should be dropped after transactionTimestamp has been written. The string indicating the date format should be adjusted. The withColumn operator should be used instead of the existing column assignment.
• C.Column transactionDate should be wrapped in a col() operator.
• D.The string indicating the date format should be adjusted. The withColumnReplaced operator should be used instead of the drop and assign pattern in the code block to replace column transactionDate with the new column transactionTimestamp.
• E.Column transactionDate should be dropped after transactionTimestamp has been written. The withColumn operator should be used instead of the existing column assignment. Column transactionDate should be wrapped in a col() operator.

In which order should the code blocks shown below be run in order to return the number of records that are not empty in column value in the DataFrame resulting from an inner join of DataFrame transactionsDf and itemsDf on columns productId and itemId, respectively?
1. .filter(~isnull(col('value')))
2. .count()
3. transactionsDf.join(itemsDf, col("transactionsDf.productId")==col("itemsDf.itemId"))
4. transactionsDf.join(itemsDf, transactionsDf.productId==itemsDf.itemId, how='inner')
5. .filter(col('value').isnotnull())
6. .sum(col('value'))

• A.3, 5, 2
• B.4, 1, 2
• C.4, 6
• D.3, 1, 2
• E.3, 1, 6

Which of the following describes the role of the cluster manager?

• A.The cluster manager allocates resources to the DataFrame manager.
• B.The cluster manager schedules tasks on the cluster in client mode.
• C.The cluster manager schedules tasks on the cluster in local mode.
• D.The cluster manager allocates resources to Spark applications and maintains the executor processes in remote mode.
• E.The cluster manager allocates resources to Spark applications and maintains the executor processes in client mode.