Online Access Free Certified-Data-Engineer-Professional Exam Questions

A data engineer, while designing a Pandas UDF to process financial time-series data with complex calculations that require maintaining state across rows within each stock symbol group, must ensure the function is efficient and scalable. Which approach will solve the problem with minimum overhead while preserving data integrity?

• A.Use a grouped_agg Pandas UDF that processes each stock symbol group independently, maintaining state through intermediate aggregation results that get passed between successive UDF calls via broadcast variables.
• B.Use applyInPandas() on a Spark DataFrame that receives all rows for each stock symbol as a Pandas DataFrame, allowing processing within each group while maintaining state variables local to each group's processing function.
• C.Use a SCALAR Pandas UDF that processes the entire dataset at once, implementing custom partitioning logic within the UDF to group by stock symbol and maintain state using global variables shared across all executor processes.
• D.Use a SCALAR_ITER Pandas UDF with iterator-based processing, implementing state management through persistent storage (Delta tables) that gets updated after each batch to maintain continuity across iterator chunks.

The data architect has mandated that all tables in the Lakehouse should be configured as external Delta Lake tables.
Which approach will ensure that this requirement is met?

• A.When configuring an external data warehouse for all table storage. leverage Databricks for all ELT.
• B.Whenever a table is being created, make sure that the location keyword is used.
• C.Whenever a database is being created, make sure that the location keyword is used
• D.When tables are created, make sure that the external keyword is used in the create table statement.
• E.When the workspace is being configured, make sure that external cloud object storage has been mounted.

The data engineer is using Spark's MEMORY_ONLY storage level. Which indicators should the data engineer look for in the spark UI's Storage tab to signal that a cached table is not performing optimally?

• A.On Heap Memory Usage is within 75% of off Heap Memory usage
• B.The RDD Block Name included the '' annotation signaling failure to cache
• C.Size on Disk is> 0
• D.The number of Cached Partitions> the number of Spark Partitions
• E.Size on Disk is < Size in Memory

A data architect has heard about lake's built-in versioning and time travel capabilities. For auditing purposes they have a requirement to maintain a full of all valid street addresses as they appear in the customers table.
The architect is interested in implementing a Type 1 table, overwriting existing records with new values and relying on Delta Lake time travel to support long-term auditing. A data engineer on the project feels that a Type 2 table will provide better performance and scalability. Which piece of information is critical to this decision?

• A.Delta Lake time travel does not scale well in cost or latency to provide a long-term versioning solution.
• B.Delta Lake only supports Type 0 tables; once records are inserted to a Delta Lake table, they cannot be modified.
• C.Delta Lake time travel cannot be used to query previous versions of these tables because Type 1 changes modify data files in place.
• D.Data corruption can occur if a query fails in a partially completed state because Type 2 tables requires setting multiple fields in a single update.
• E.Shallow clones can be combined with Type 1 tables to accelerate historic queries for long-term versioning.

A Data Engineer is building a simple data pipeline using Lakeflow Declarative Pipelines (LDP) in Databricks to ingest customer data. The raw customer data is stored in a cloud storage location in JSON format. The task is to create Lakeflow Declarative Pipelines that read the raw JSON data and write it into a Delta table for further processing. Which code snippet will correctly ingest the raw JSON data and create a Delta table using LDP?

• A.import dlt
  @dlt.table
  def raw_customers():
  return spark.read.format("csv").load("s3://my-bucket/raw-customers/")
• B.import dlt
  @dlt.view
  def raw_customers():
  return spark.format.json("s3://my-bucket/raw-customers/")
• C.import dlt
  @dlt.table
  def raw_customers():
  return spark.read.format("parquet").load("s3://my-bucket/raw-customers/")
• D.import dlt
  @dlt.table
  def raw_customers():
  return spark.read.json("s3://my-bucket/raw-customers/")