The problem description indicates that VMs in VPC2 receive traffic but their replies don't reach VPC1, especially when a VPN gateway fails. This strongly suggests an asymmetric routing issue, where VPC2's routing table might not be aware of all necessary routes to send return traffic to VPC1, particularly in a multi- region setup with failover. By default, VPC networks are in regional dynamic routing mode, meaning they only learn routes from Cloud Routers in the same region. To ensure that routes learned from one region (where the active VPN tunnel might be) are available globally across the VPC, you need to enable global dynamic routing mode in the VPC that is experiencing the return traffic issue (VPC2 in this case). This allows VPC2 to learn and apply routes from Cloud Routers in all regions, ensuring that even if a VPN tunnel fails in one region, the routes learned from the active tunnel in another region are still available for return traffic.
Exact Extract:
"A VPC network's dynamic routing mode controls whether routes learned by Cloud Routers in one region are available to VMs in other regions. By default, VPC networks are in regional dynamic routing mode, which means Cloud Routers in a region only advertise routes to and learn routes from other Cloud Routers in the same region. This can lead to asymmetric routing issues in multi-region deployments."
"To ensure that routes learned from Cloud Routers are propagated to all regions within a VPC network, you must set the dynamic routing mode to global."Reference: Google Cloud VPC Documentation - Dynamic routing mode

When enabling Cloud CDN, for caching behavior to follow the application-defined caching headers, you need to configure the USE_ORIGIN_HEADERS caching mode. This setting ensures that the Cloud CDN respects the cache control headers specified by the backend, allowing the application-defined caching rules to dictate what content gets cached. This is often required when specific caching directives are already set by the application.
Reference: Google Cloud - Cloud CDN Caching Modes

Explanation: Creating a BGP route policy at the Cloud Router ensures that the subnets of the VPC spokes are properly advertised to the on-premises environment. This allows the on-premises router to receive and use those routes. Without the correct BGP policies, route advertisement may not happen as expected.
: Google Cloud BGP Route Configuration