A team of data engineer are adding tables to a DLT pipeline that contain repetitive expectations for many of the same data quality checks. One member of the team suggests reusing these data quality rules across all tables defined for this pipeline. What approach would allow them to do this?
Correct Answer: A
Maintaining data quality rules in a centralized Delta table allows for the reuse of these rules across multiple DLT (Delta Live Tables) pipelines. By storing these rules outside the pipeline's target schema and referencing the schema name as a pipeline parameter, the team can apply the same set of data quality checks to different tables within the pipeline. This approach ensures consistency in data quality validations and reduces redundancy in code by not having to replicate the same rules in each DLT notebook or file. Reference: Databricks Documentation on Delta Live Tables: Delta Live Tables Guide
A data engineer is using Lakeflow Declarative Pipeline to propagate row deletions from a source bronze table (user_bronze) to a target silver table (user_silver). The engineer wants deletions in user_bronze to automatically delete corresponding rows in user_silver during pipeline execution. Which configuration ensures deletions in the bronze table are propagated to the silver table?
Correct Answer: B
Comprehensive and Detailed Explanation From Exact Extract of Databricks Data Engineer Documents: According to Databricks documentation, Change Data Feed (CDF) allows pipelines to read incremental data changes, including inserts, updates, and deletes, from a Delta table. When deletions occur in the source table, reading the CDF stream ensures downstream consumers receive the deletion records. The Lakeflow Declarative Pipelines API provides the apply_changes() function (or auto-CDC pipelines) with the apply_as_deletes parameter to correctly apply those deletions to the target table. This enables automatic synchronization between bronze and silver layers. Options A and D either require manual handling or complete rebuilds, and C incorrectly applies CDF to the target rather than the source. Therefore, enabling CDF on the bronze table and using apply_as_deletes=True is the correct, Databricks-supported configuration.
Which of the following technologies can be used to identify key areas of text when parsing Spark Driver log4j output?
Correct Answer: A
Regex, or regular expressions, are a powerful way of matching patterns in text. They can be used to identify key areas of text when parsing Spark Driver log4j output, such as the log level, the timestamp, the thread name, the class name, the method name, and the message. Regex can be applied in various languages and frameworks, such as Scala, Python, Java, Spark SQL, and Databricks notebooks. Reference: https://docs.databricks.com/notebooks/notebooks-use.html#use-regular-expressions https://docs.databricks.com/spark/latest/spark-sql/udf-scala.html#using-regular-expressions-in-udfs https://docs.databricks.com/spark/latest/sparkr/functions/regexp_extract.html https://docs.databricks.com/spark/latest/sparkr/functions/regexp_replace.html
A data engineering team uses Databricks Lakehouse Monitoring to track the percent_null metric for a critical column in their Delta table. The profile metrics table (prod_catalog.prod_schema.customer_data_profile_metrics) stores hourly percent_null values. The team wants to: Trigger an alert when the daily average of percent_null exceeds 5% for three consecutive days. Ensure that notifications are not spammed during sustained issues. Options:
Correct Answer: B
The key requirement is to detect when the daily average of percent_null is greater than 5% for three consecutive days. Option A only checks the last 24 hours, not consecutive days. It would trigger too frequently and cause spam. Option C calculates an average across all records in the last 3 days, but this could be skewed by one high or low day - it does not ensure consecutive daily violations. Option D simply counts days where the threshold was exceeded, but it does not guarantee that those days were consecutive. This could incorrectly trigger on non-adjacent violations. Option B is correct: It aggregates hourly values into daily averages. It checks that the last 3 consecutive days all had averages above 5%. It avoids redundant alerts by using Notification Frequency: Just once. This matches Databricks Lakehouse Monitoring best practices, where SQL alerts should be designed to aggregate metrics to the correct granularity (daily here) and ensure consecutive threshold violations before triggering. Reference (Databricks Lakehouse Monitoring, SQL Alerts Best Practices): Use DATE_TRUNC to compute metrics at the correct time granularity. To detect consecutive-day issues, filter the last N daily aggregates and check conditions across all rows. Always configure alerts with controlled notification frequency to prevent alert fatigue.
A transactions table has been liquid clustered on the columns product_id, user_id, and event_date. Which operation lacks support for cluster on write?
Correct Answer: A
Delta Lake's Liquid Clustering is an advanced feature that improves query performance by dynamically clustering data without requiring costly compaction steps like traditional Z-ordering. When performing writes to a Liquid Clustered table, some write operations automatically maintain clustering, while others do not. Explanation of Each Option: (A) spark.writestream.format('delta').mode('append') (Correct Answer) Reason: Streaming writes (writestream) do not support Liquid Clustering because streaming data arrives in micro-batches. Since Liquid Clustering needs efficient global reorganization of files, streaming append operations don't provide sufficient data volume at a time to be effectively clustered. Delta Lake documentation states that Liquid Clustering is only supported for batch writes. (B) CTAS and RTAS statements Reason: CREATE TABLE AS SELECT (CTAS) and REPLACE TABLE AS SELECT (RTAS) are batch operations and can enforce Liquid Clustering. These operations create or replace a table based on a query result, and since they are batch-based, Liquid Clustering applies. (C) INSERT INTO operations Reason: INSERT INTO is supported for Liquid Clustering because it is a batch operation. While it may not be as efficient as MERGE or COPY INTO, clustering is applied upon execution. (D) spark.write.format('delta').mode('append') Reason: Batch append operations are supported for Liquid Clustering. Unlike streaming append, batch writes allow the optimizer to re-cluster data efficiently. Conclusion: Since streaming append operations do not support Liquid Clustering, option (A) is the correct answer. Reference: Liquid Clustering in Delta Lake - Databricks Documentation
