Your company is planning to migrate their on-premises Hadoop environment to the cloud.
Increasing storage cost and maintenance of data stored in HDFS is a major concern for your company. You also want to make minimal changes to existing data analytics jobs and existing architecture.
How should you proceed with the migration?

Your team runs a Python job that reads millions of customer record files stored in a Cloud Storage bucket. To comply with regulatory requirements, you need to ensure that customer data is immediately deleted once the job is completed. You want to minimize the time required to complete this task. What should you do?

You are planning to add unit tests to your application. You need to be able to assert that published Pub/Sub messages are processed by your subscriber in order. You want the unit tests to be cost-effective and reliable. What should you do?

Case Study 1 - HipLocal
Company Overview
HipLocal is a community application designed to facilitate communication between people in close proximity. It is used for event planning and organizing sporting events, and for businesses to connect with their local communities. HipLocal launched recently in a few neighborhoods in Dallas and is rapidly growing into a global phenomenon. Its unique style of hyper-local community communication and business outreach is in demand around the world.
Executive Statement
We are the number one local community app; it's time to take our local community services global. Our venture capital investors want to see rapid growth and the same great experience for new local and virtual communities that come online, whether their members are 10 or 10000 miles away from each other.
Solution Concept
HipLocal wants to expand their existing service, with updated functionality, in new regions to better serve their global customers. They want to hire and train a new team to support these regions in their time zones. They will need to ensure that the application scales smoothly and provides clear uptime data.
Existing Technical Environment
HipLocal's environment is a mix of on-premises hardware and infrastructure running in Google Cloud Platform. The HipLocal team understands their application well, but has limited experience in global scale applications. Their existing technical environment is as follows:
* Existing APIs run on Compute Engine virtual machine instances hosted in GCP.
* State is stored in a single instance MySQL database in GCP.
* Data is exported to an on-premises Teradata/Vertica data warehouse.
* Data analytics is performed in an on-premises Hadoop environment.
* The application has no logging.
* There are basic indicators of uptime; alerts are frequently fired when the APIs are unresponsive.
Business Requirements
HipLocal's investors want to expand their footprint and support the increase in demand they are seeing. Their requirements are:
* Expand availability of the application to new regions.
* Increase the number of concurrent users that can be supported.
* Ensure a consistent experience for users when they travel to different regions.
* Obtain user activity metrics to better understand how to monetize their product.
* Ensure compliance with regulations in the new regions (for example, GDPR).
* Reduce infrastructure management time and cost.
* Adopt the Google-recommended practices for cloud computing.
Technical Requirements
* The application and backend must provide usage metrics and monitoring.
* APIs require strong authentication and authorization.
* Logging must be increased, and data should be stored in a cloud analytics platform.
* Move to serverless architecture to facilitate elastic scaling.
* Provide authorized access to internal apps in a secure manner.
HipLocal's data science team wants to analyze user reviews.
How should they prepare the data?

Case Study 2 - HipLocal
Company Overview
HipLocal is a community application designed to facilitate communication between people in close proximity. It is used for event planning and organizing sporting events, and for businesses to connect with their local communities. HipLocal launched recently in a few neighborhoods in Dallas and is rapidly growing into a global phenomenon. Its unique style of hyper-local community communication and business outreach is in demand around the world.
Executive Statement
We are the number one local community app; it's time to take our local community services global. Our venture capital investors want to see rapid growth and the same great experience for new local and virtual communities that come online, whether their members are 10 or 10000 miles away from each other.
Solution Concept
HipLocal wants to expand their existing service, with updated functionality, in new regions to better serve their global customers. They want to hire and train a new team to support these regions in their time zones. They will need to ensure that the application scales smoothly and provides clear uptime data, and that they analyze and respond to any issues that occur.
Existing Technical Environment
HipLocal's environment is a mix of on-premises hardware and infrastructure running in Google Cloud Platform. The HipLocal team understands their application well, but has limited experience in global scale applications. Their existing technical environment is as follows:
- Existing APIs run on Compute Engine virtual machine instances hosted in GCP.
- State is stored in a single instance MySQL database in GCP.
- Release cycles include development freezes to allow for QA testing.
- The application has no logging.
- Applications are manually deployed by infrastructure engineers during periods of slow traffic on weekday evenings.
- There are basic indicators of uptime; alerts are frequently fired when the APIs are unresponsive.
Business Requirements
HipLocal's investors want to expand their footprint and support the increase in demand they are seeing. Their requirements are:
- Expand availability of the application to new regions.
- Support 10x as many concurrent users.
- Ensure a consistent experience for users when they travel to different regions.
- Obtain user activity metrics to better understand how to monetize their product.
- Ensure compliance with regulations in the new regions (for example, GDPR).
- Reduce infrastructure management time and cost.
- Adopt the Google-recommended practices for cloud computing.
Develop standardized workflows and processes around application lifecycle management.
Define service level indicators (SLIs) and service level objectives (SLOs).
Technical Requirements
- Provide secure communications between the on-premises data center and cloud-hosted applications and infrastructure.
- The application must provide usage metrics and monitoring.
- APIs require authentication and authorization.
- Implement faster and more accurate validation of new features.
- Logging and performance metrics must provide actionable information to be able to provide debugging information and alerts.
- Must scale to meet user demand.
For this question, refer to the HipLocal case study.
A recent security audit discovers that HipLocal's database credentials for their Compute Engine- hosted MySQL databases are stored in plain text on persistent disks. HipLocal needs to reduce the risk of these credentials being stolen. What should they do?