Problem connecting GKE cluster to remote network via VPN

[Anders Qvist]

I am trying to connect a GKE cluster to a remote network using a GCE VPN to a Cisco ASA 5510. Ping from GKE node 10.248.0.26 -> remote node 10.99.193.115 arrives at 10.99.193.115 and the ASA says that the echo reply goes back through the tunnel to GKE. However, tcpdump on 10.248.0.26 shows no replies coming in.

Firewall and routing as reported by Google Cloud Console:

    Name Source tag / IP range Allowed protocols / ports Target tags

    default-allow-icmp 0.0.0.0/0 icmp Apply to all targets

    default-allow-internal 10.240.0.0/16 tcp:1-65535; udp:1-65535; icmp Apply to all targets

    default-allow-ssh 0.0.0.0/0 tcp:22 Apply to all targets

    gke-zecluster-d6cc7a55-all 10.248.0.0/14 tcp; udp; icmp; Apply to all targets

    gke-zecluster-d6cc7a55-ssh <public_ip>/32 tcp:22 gke-zecluster-d6cc7a55-node

    gke-zecluster-d6cc7a55-vms 10.240.0.0/16 tcp:1-65535; udp:1-65535; icmp gke-zecluster-d6cc7a55-node

    k8s-fw-a1a92183fb18e11e5be3442010af0001 0.0.0.0/0 tcp:80,443 gke-zecluster-d6cc7a55-node

    k8s-fw-a1aa3fe95b18e11e5be3442010af0001 0.0.0.0/0 tcp:2003 gke-zecluster-d6cc7a55-node

    Name Destination IP ranges Priority Instance tags Next hop

    default-route-3eed071cad0670e8 0.0.0.0/0 1000 None Default internet gateway

    default-route-7a9ddc4457c714a0 10.240.0.0/16 1000 None Virtual network

    gke-zecluster-d6cc7a55-7b61213c-b187-11e5-be34-42010af00015 10.248.0.0/24 1000 None gke-zecluster-d6cc7a55-node-j4jx (Zone ze-zone-1)

    gke-zecluster-d6cc7a55-7ec5f7a9-b187-11e5-be34-42010af00015 10.248.1.0/24 1000 None gke-zecluster-d6cc7a55-node-rluf (Zone ze-zone-1)

    vpn-1-tunnel-1-route-1 10.99.0.0/16 1000 None

Is there some logging I can turn on to see what goes on? As far as I can see, the VPN says nothing pertinent about this traffic, only:

    15:24:51.058 sending DPD request

    15:24:51.058 generating INFORMATIONAL_V1 request 3069408857 [ HASH N(DPD) ]

    15:24:51.058 sending packet: from <gce-vpn-ip>[500] to <asa-ip>[500] (92 bytes)

    15:24:51.092 received packet: from <asa-ip>[500] to <gce-vpn-ip>[500] (92 bytes)

    15:24:51.092 parsed INFORMATIONAL_V1 request 146600869 [ HASH N(DPD_ACK) ]

If I modify the VPN tunnel (GCE VPN, ASA) to have the default net 10.240.0.0/16 at the GCE end traffic passes correctly in both directions.

I assume this is a routing issue, but what? Should not the route 10.248.0.0/24 send the traffic back to the GKE node? Or do I have to somehow declare the GKE network as a network?

[Kamran (Google Cloud Support)]

Hello Anders,

As this question is already posted on serverfault, I'll try to help you to resolve this issue via posting on the serverfault thread.

Sincerely,