A data engineer is managing a data pipeline in Databricks, where multiple Delta tables are used for various transformations. The team wants to track how data flows through the pipeline, including identifying dependencies between Delta tables, notebooks, jobs, and dashboards. The data engineer is utilizing the Unity Catalog lineage feature to monitor this process. How does Unity Catalog's data lineage feature support the visualization of relationships between Delta tables, notebooks, jobs, and dashboards?
Correct Answer: D
Databricks documentation for Unity Catalog lineage states that lineage is captured across queries run on Databricks , is supported for all languages , and is tracked down to the column level . It also explicitly says that lineage data includes related notebooks, jobs, and dashboards , and that the lineage can be visualized in Catalog Explorer in near real time as an interactive graph. This directly matches option D. The value of this feature is that it helps engineers understand where a table's data came from, which upstream notebooks or jobs produced it, and which downstream assets, including dashboards, depend on it. Column-level lineage adds another layer of traceability by showing how specific fields are derived through transformations, which is especially useful for impact analysis, governance, and troubleshooting. Options A, B, and C each remove documented capabilities that Unity Catalog lineage actually supports. Therefore, the Databricks-documented answer is the full interactive graph across Delta tables and related assets, with column-level tracking included. =========
A data engineer needs to develop integration tests for an ETL process and deploy a version-controlled, packaged workflow into production using an external job scheduler. Which tool should the data engineer use for this job?
Correct Answer: B
Databricks Asset Bundles provide a modern, declarative approach to defining, testing, and deploying Databricks workflows as version-controlled code artifacts . Using YAML configuration files, engineers can describe jobs, pipelines, dependencies, and environments in a structured and reproducible way. Asset Bundles integrate well with CI/CD pipelines and external schedulers, enabling teams to package workflows, run integration tests, and deploy consistently across environments such as development, staging, and production. This aligns directly with the requirement for version control and external orchestration. The Databricks CLI (A) is primarily used as a deployment interface but does not provide the structured packaging and workflow definition capabilities of Asset Bundles. Databricks Connect (C) is intended for local development and testing of Spark code, not workflow deployment. The Databricks SDK (D) provides programmatic access to APIs but lacks the declarative, packaged deployment model required here. Databricks documentation positions Asset Bundles as the recommended solution for production-grade workflow deployment and testing within modern data engineering practices.
In order for Structured Streaming to reliably track the exact progress of the processing so that it can handle any kind of failure by restarting and/or reprocessing, which of the following two approaches is used by Spark to record the offset range of the data being processed in each trigger?
Correct Answer: A
Structured Streaming uses checkpointing and write-ahead logs to record the offset range of the data being processed in each trigger. This ensures that the engine can reliably track the exact progress of the processing and handle any kind of failure by restarting and/or reprocessing. Checkpointing is the mechanism of saving the state of a streaming query to fault-tolerant storage (such as HDFS) so that it can be recovered after a failure. Write-ahead logs are files that record the offset range of the data being processed in each trigger and are written to the checkpoint location before the processing starts. These logs are used to recover the query state and resume processing from the last processed offset range in case of a failure. References: Structured Streaming Programming Guide, Fault Tolerance Semantics
A data engineer needs to apply custom logic to string column city in table stores for a specific use case. In order to apply this custom logic at scale, the data engineer wants to create a SQL user-defined function (UDF). Which of the following code blocks creates this SQL UDF?
A data engineer is attempting to drop a Spark SQL table my_table and runs the following command: DROP TABLE IF EXISTS my_table; After running this command, the engineer notices that the data files and metadata files have been deleted from the file system. Which of the following describes why all of these files were deleted?
Correct Answer: A
The reason why all of the data files and metadata files were deleted from the file system after dropping the table is that the table was managed. A managed table is a table that is created and managed by Spark SQL. It stores both the data and the metadata in the default location specified by the spark.sql.warehouse. dir configuration property. When a managed table is dropped, both the data and the metadata are deleted from the file system. Option B is not correct, as the size of the table's data does not affect the behavior of dropping the table. Whether the table's data is smaller or larger than 10 GB, the data files and metadata files will be deleted if the table is managed, and will be preserved if the table is external. Option C is not correct, for the same reason as option B. Option D is not correct, as an external table is a table that is created and managed by the user. It stores the data in a user-specified location, and only stores the metadata in the Spark SQL catalog. When an external table is dropped, only the metadata is deleted from the catalog, but the data files are preserved in the file system. Option E is not correct, as a table must have a location to store the data. If the location is not specified by the user, it will use the default location for managed tables. Therefore, a table without a location is a managed table, and dropping it will delete both the data and the metadata. Managing Tables [Databricks Data Engineer Professional Exam Guide]
