Routing through ip forwarding gateway stopped working

[Flez Pirk]

Hi everyone,

I just hit a strange problem today with Compute and Networking - in my environment I have most of compute instances running with no external IP, and only one host with external IP and ip forwarding enabled to act as a gateway. All machines used to route to that gateway using a network route, with "instange tags" filter to select only non-internet-facing machines, and "Next hop" set to the gateway instance.

This set up was working with no issues for months, but today it suddenly stopped working for no apparent reason. Now when I enable a network route - it just doesn't route the traffic anymore, even though the gateway machine is operating just fine.

Has anyone else been affected by the same problem? Or can anyone help to debug it, as I'm at quite a loss for the moment what to do? I have given my machines ephemeral IPs and set up a proxy service on the gateway host to get by for the moment, but I'd really like to get a proper setup operational again.

Cheers.

[Carlos (Cloud Platform Support)]

Hi Flez,

The setup you mentioned is similar to the one described in the section ¨Configure a NAT gateway” of this document. It might be easier to start a setup from scratch than to troubleshoot the problem.  

Nevertheless I have some suggestions:

1. In order to understand if you have a problem with a route or the gateway per se, you can capture packets (tcpdump) in the gateway. If the packets are not received by the gateway, then you have a problem with a route in your Google Project otherwise the issue is related to the gateway. You can also use ¨mtr¨ or ¨traceroute¨ to confirm if the gateway is reached.

2. You can always look at the serial console of the gateway to check for error messages.

3. You can try to inspect the application and logs that are performing the NAT in the gateway. Make sure that this service is running.  i.e. if you are using iptables this can help.

4. Inspect the activity logs and operations logs of your console to see if your gateway had other issues or got recreated.

5. Make sure that the route defined on Google network to the gateway is still valid.  Check that the tags and the IP ranges are the right ones. If you have multiple routes, one might be chosen over the other due to lower priority.

Following the example provided in the document, this is how my network firewall rules and routes look. Only instances tagged as ¨no-ip-internet-route¨ will get redirected to the gateway.

gcloud compute routes  list | grep gce-network

default-route-d3ddfdd7c6acbb26 gce-network  0.0.0.0/0 default-internet-gateway 1000

default-route-e1ed0e8a5e5af73f gce-network  10.240.0.0/16                      1000 

no-ip-internet-route           gce-network  0.0.0.0/0  us-central1-a/instances/nat-gateway  800

gcloud compute firewall-rules  list | grep gce-network

gce-network-allow-internal               gce-network  10.240.0.0/16     tcp:1-65535,udp:1-65535,icmp

gce-network-allow-ssh                    gce-network  0.0.0.0/0         tcp:22

icmpall                                  gce-network  0.0.0.0/0         icmp

I can always confirm packets are being forwarded to the gateway by capturing them on it or by tracing the route in the source instance. 

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets

1 nat-gateway.c.project.internal (10.240.0.2) 1.196 ms 1.180 ms 1.164 ms

2 google-public-dns-a.google.com (8.8.8.8) 3.579 ms 1.585 ms 1.675 ms

There is also a critical detail that requires the gateway to have the IP forward flag during its creation and them enabled at the OS level  (sudo sysctl -w net.ipv4.ip_forward=1)

I hope this helps.

[Flez Pirk]

Hello Carlos,

after some debugging it turned out to be an iptables-caused issue, which was triggered by instances being restarted for some reason. Thank you for your suggestions, it really helped.