- Autoclass automatically moves objects between storage classes without impacting performance or availability, nor incurring retrieval costs. - It continuously optimizes storage costs based on access patterns without the need to set specific lifecycle management policies.

To design a data mesh architecture using Dataplex to eliminate bottlenecks caused by a central data platform team, consider the following:
Data Mesh Architecture:
Data mesh promotes a decentralized approach where domain teams manage their own data pipelines and assets, increasing agility and reducing bottlenecks.
Dataplex Lakes and Zones:
Lakes in Dataplex are logical containers for managing data at scale, and zones are subdivisions within lakes for organizing data based on domains, teams, or other criteria.
Domain and Team Management:
By creating a lake for each team and zones for each domain, each team can independently manage their data assets without relying on the central data platform team.
This setup aligns with the principles of data mesh, promoting ownership and reducing delays in data processing and insights.
Implementation Steps:
Create Lakes and Zones:
Create separate lakes in Dataplex for each team (analytics and data science).
Within each lake, create zones for the different domains (airlines, hotels, ride-hailing).
Attach BigQuery Datasets:
Attach the BigQuery datasets created by the respective teams as assets to their corresponding zones.
Decentralized Management:
Allow each domain to manage their own zone's data assets, providing them with the autonomy to update and maintain their pipelines without depending on the central team.
Reference:
Dataplex Documentation
BigQuery Documentation
Data Mesh Principles

https://cloud.google.com/dataflow/docs/guides/troubleshooting-your-pipeline#detect_an_exception_in_worker_code While your job is running, you might encounter errors or exceptions in your worker code. These errors generally mean that the DoFns in your pipeline code have generated unhandled exceptions, which result in failed tasks in your Dataflow job. Exceptions in user code (for example, your DoFn instances) are reported in the Dataflow monitoring interface.