Explanation

Layer 1: Azure Data Factory
Layer 2: Azure Databricks
Azure Databricks is fully integrated with Azure Data Factory .

References:
https://docs.microsoft.com/en-us/azure/architecture/data-guide/big-data/

Explanation

Health Review: Azure SQL Database
Scenario: ADatum identifies the following requirements for the Health Review application:
* Ensure that sensitive health data is encrypted at rest and in transit.
* Tag all the sensitive health data in Health Review. The data will be used for auditing.
Health Interface: Azure Cosmos DB
ADatum identifies the following requirements for the Health Interface application:
* Upgrade to a data storage solution that will provide flexible schemas and increased throughput for writing data. Data must be regionally located close to each hospital, and reads must display be the most recent committed version of an item.
* Reduce the amount of time it takes to add data from new hospitals to Health Interface.
* Support a more scalable batch processing solution in Azure.
* Reduce the amount of development effort to rewrite existing SQL queries.
Health Insights: Azure SQL Data Warehouse
Azure SQL Data Warehouse is a cloud-based enterprise data warehouse that leverages massively parallel processing (MPP) to quickly run complex queries across petabytes of data. Use SQL Data Warehouse as a key component of a big data solution.
You can access Azure SQL Data Warehouse (SQL DW) from Databricks using the SQL Data Warehouse connector (referred to as the SQL DW connector), a data source implementation for Apache Spark that uses Azure Blob Storage, and PolyBase in SQL DW to transfer large volumes of data efficiently between a Databricks cluster and a SQL DW instance.
Scenario: ADatum identifies the following requirements for the Health Insights application:
* The new Health Insights application must be built on a massively parallel processing (MPP) architecture that will support the high performance of joins on large fact tables References:
https://docs.databricks.com/data/data-sources/azure/sql-data-warehouse.html

Explanation

Box 1: Always Encrypted with deterministic encryption
Deterministic encryption always generates the same encrypted value for any given plain text value. Using deterministic encryption allows point lookups, equality joins, grouping and indexing on encrypted columns.
However, it may also allow unauthorized users to guess information about encrypted values by examining patterns in the encrypted column, especially if there is a small set of possible encrypted values, such as True/False, or North/South/East/West region. Deterministic encryption must use a column collation with a binary2 sort order for character columns.
Box 2: Always Encrypted with Randomized encryption
* Randomized encryption uses a method that encrypts data in a less predictable manner. Randomized encryption is more secure, but prevents searching, grouping, indexing, and joining on encrypted columns.
Note: With Always Encrypted the Database Engine never operates on plaintext data stored in encrypted columns, but it still supports some queries on encrypted data, depending on the encryption type for the column. Always Encrypted supports two types of encryption: randomized encryption and deterministic encryption.
Use deterministic encryption for columns that will be used as search or grouping parameters, for example a government ID number. Use randomized encryption, for data such as confidential investigation comments, which are not grouped with other records and are not used to join tables.
References:
https://docs.microsoft.com/en-us/sql/relational-databases/security/encryption/always-encrypted-database-engine

Explanation

Box 1: Eventual
Traffic data insertion rate must be maximized.
Sensor data must be stored in a Cosmos DB named treydata in a collection named SensorData With Azure Cosmos DB, developers can choose from five well-defined consistency models on the consistency spectrum. From strongest to more relaxed, the models include strong, bounded staleness, session, consistent prefix, and eventual consistency.
Box 2: License plate
This solution reports on all data related to a specific vehicle license plate. The report must use data from the SensorData collection.
References:
https://docs.microsoft.com/en-us/azure/cosmos-db/consistency-levels

Azure Data Lake Storage is optimized performance for parallel analytics workloads A key mechanism that allows Azure Data Lake Storage Gen2 to provide file system performance at object storage scale and prices is the addition of a hierarchical namespace. This allows the collection of objects/files within an account to be organized into a hierarchy of directories and nested subdirectories in the same way that the file system on your computer is organized.
Reference:
https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-namespace