Private NAT Gateway: Configuring a Private NAT gateway ensures that resources in different Google Cloud regions can use minimal IP address space by routing through region-specific NAT subnets (192.168.x.0/24). This approach conserves the limited routable IP space (192.168.0.0/20) while allowing cross-region communication.
VPC as a Spoke with Export Include Policy: Adding the VPC as a spoke to the Network Connectivity Center hub and using an export include policy ensures that only the NAT subnets (192.168.1.0/24, 192.168.2.0/24, 192.168.3.0/24) are advertised to the hub. This reduces the amount of IP address space consumed in routing.
Global Dynamic Routing: Enabling global dynamic routing allows resources in us-west1, europe- central1, and asia-southeast1 to access the on-premises environment via the Cloud Interconnect in us-east4. This minimizes latency and ensures connectivity across regions.

DNS queries are timing out because the traffic still must be allowed through the on-premises firewall on both UDP 53 and TCP 53 for the inbound forwarding path to work. Google Cloud documents that the effective firewall configuration must allow DNS traffic on both protocols, so permitting that traffic to the Cloud DNS inbound forwarding addresses resolves the timeout issue.
Reference:
https://docs.cloud.google.com/dns/docs/server-policies-overview

In this scenario, using cross-project binding is the simplest and most efficient way to allow each team to create and manage their own DNS private zones while ensuring that workloads across both service projects can resolve DNS records across the Shared VPC.
Cross-project binding: This feature allows private DNS zones created in the service projects (SP_ONE_PRJ and SP_TWO_PRJ) to be associated with the Shared VPC (SH_VPC) in the host project (SH_HOST_PRJ). It enables DNS queries for those zones to be resolved by workloads attached to the Shared VPC.
Centralized connectivity: Since all workloads in both service projects (SP_ONE_PRJ and SP_TWO_PRJ) use the Shared VPC (SH_VPC), no additional VPCs or VPC peering setups are required, minimizing setup effort.
Team independence: Each team manages their DNS private zones and records independently within their respective service projects, fulfilling the requirement for decentralization.
This design achieves the desired DNS resolution while requiring the least amount of setup effort compared to solutions involving new VPCs, DNS peering, or VPC Network Peering.

When Cloud DNS forwarding is used, the forwarded queries originate from Google-managed Cloud DNS resolver source IP ranges, not from the VM's IP. Your on-premises environment must be able to route return traffic to those resolver source ranges and allow them through firewalls on TCP/UDP 53. Once the Cloud DNS source range is reachable and permitted, VMs can resolve on-premises records via the default metadata DNS.

Network Connectivity Center with a Mesh Topology: Network Connectivity Center (NCC) provides a scalable and managed solution for interconnecting multiple VPCs. A mesh topology allows all VPCs to communicate with each other without requiring direct peering connections between every pair of VPCs, simplifying the configuration and management for a high-throughput environment.
Exclusion of Overlapping Subnets: Using an export exclude filter for 240.0.0.0/4, you ensure that overlapping subnets within the Class E address range (240.0.0.0/4) are not included in the routing table, which avoids routing conflicts between the two overlapping subnets.
Firewall Policy Rules: Adding appropriate Cloud network firewall rules ensures that only authorized traffic is allowed between the VPCs, enhancing security and maintaining compliance with organizational policies.