All operations are atomic at the row level. For example, if you update two rows in a table, it's possible that one row will be updated successfully and the other update will fail. Avoid schema designs that require atomicity across rows.
Reference: https://cloud.google.com/bigtable/docs/schema-design#row-keys

Reference: https://cloud.google.com/solutions/data-lifecycle-cloud-platform

To optimize the performance of a complex Spark job on Dataproc that heavily relies on shuffling operations, and given the cost constraints of using preemptible VMs, switching from HDDs to SSDs and using HDFS as an intermediate storage layer can significantly improve performance. Here's why option C is the best choice:
Performance of SSDs:
SSDs provide much faster read and write speeds compared to HDDs, which is crucial for performance-intensive operations like shuffling in Spark jobs.
Using SSDs can reduce I/O bottlenecks during the shuffle phase of your Spark job, improving overall job performance.
Intermediate Storage with HDFS:
Copying data from Google Cloud Storage (GCS) to HDFS for intermediate storage can reduce latency compared to reading directly from GCS.
HDFS provides better locality and faster data access within the Dataproc cluster, which can significantly improve the efficiency of shuffling and other I/O operations.
Cost Considerations:
Although SSDs are more expensive than HDDs, the performance improvement for shuffle-heavy workloads can justify the cost, especially if the improved performance reduces the overall runtime and thereby the cost of using preemptible VMs.
Using preemptible VMs with SSDs for this workload balances the cost and performance trade-offs effectively.

Service accounts used with Cloud Dataproc must have Dataproc/Dataproc Worker role (or have all the permissions granted by Dataproc Worker role).
Reference: https://cloud.google.com/dataproc/docs/concepts/service-accounts#important_notes