How to DNAT to GCE instance in subnetwork (port forwarding)?

[Anders Brown]

Dealing with a layer 4 network turned out harder than expected, because all the rules I learnt don't seem to apply. I want to run a service on a VM ("backend") that is protected against common attacks by not giving it a public IP. Only a single port shall be forwarded from the publicly visible head node ("head"). The service behind that port is safe against common attacks. 

The idea is that if an attacker manages to compromise "head", e.g. through a vulnerability with whatever service, or a temporary outage of the GCE firewall, there is some time left to respond until he makes his way to "backend". So, if an intrusion is detected on "head", the "backend" could be shut down automatically in an instant.

In short: 

VM "head" with a public IP forwards port XXXX to VM "backend". 

I managed to set this up with iptables and DNAT, but observed that source IPs of forwarded packets are replaced with the local IP of "head". If there was a way to preserve original source IPs, this would be the preferred perfect solution. How?

Setting this up with a load balancer seems like overkill for a single instance and didn't work for me yet, because there's no way to specify "head" as public entry point. I'm afraid that with another dedicated public IP for the load balancer, "backend" would be exposed to attacks from outside. I may be wrong, though.

Any idea how to go about with this scenario? 

Thanks in advance.

[Faizan (Google Cloud Support)]

Hello Anders,

In order to troubleshot this further can you provide me with the following information:

1. When creating your instance "head" have you enabled the IP forwarding flag? 

2. Have you also enabled IP forwarding inside your VM?

3. Can you share your DNAT and iptables config to reproduce the issue.

Looking forward to your response.

Faizan

[Anders Brown]

Thanks for your quick response, Faizan.

1. IP forwarding is on for the VM in console

2. IP forwarding is enabled in VM:

/etc/sysconfig/iptables-config  (IPTABLES_MODULES="iptable_nat ip_nat_ftp ip_conntrack ip_conntrack_ftp")

/etc/sysctl.conf (net.ipv4.ip_forward = 1)

3. iptables rules are:

iptables -t nat -A PREROUTING -p tcp -i eth0 -d <internal ip> --dport 8000 -j DNAT --to-destination <internal ip destination>

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

DNAT bascially works already, except for source IP replaced with <internal ip> of the head (bastion) when packets arrive at <internal ip destination>. Source IP should be preserved.

[Anders Brown]

Actually the title should read:

"Source IP lost after DNAT to instance in subnetwork"

[Anders Brown]

Maybe I should file a bug report now, as this is having a negative impact on a lot of things with us, like intrusion and abuse detection no longer working.

[Faizan (Google Cloud Support)]

Hello Anders,

Sorry for the delay in replying.

This seem to be an expected behavior when using MASQUERADE. You can refer to this post where similar question was answered.

With that said, I believe the easiest solution to configure security would be through GCE firewall rules. You can create the rules which can limit the incoming traffic to your desired port(s) only. It is also relatively easy to manage(create and delete)GCE network firewall rules. Another solution I can think of would be to configure a transparent proxy on "head" VM which will forward the packets to an other machine while keeping the source IP.

I hope that helps.

Faizan

[Anders Brown]

The MASQUERADE is for outgoing packets only. Changing that rule would not bring back the source IP of incoming packets. I'll try that anyway, just for the record.

I set up NAT exactly as suggested by the GCE documentation. This is also how NAT has always worked in all networks I remember. It's the expected way of doing this. If it doesn't work in GCE, the documentation should at least warn about that, or the bug should be addressed. The latter is much preferred. NAT without a valid source IP is pretty useless and can be considered broken.

[Anders Brown]

Found a fix:

iptables -t nat -A PREROUTING -p tcp -i eth0 -d 192.168.1.4 --dport 80 -j DNAT --to-destination 192.168.1.6:80
iptables -t nat -A POSTROUTING -p tcp --sport 80 -s 192.168.1.6/32 -o eth0 -j SNAT --to-source <public ip>
iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE

1st rule does DNAT the standard way for port forwarding.

2nd rule intercepts outgoing packets from the forwarded service exclusively (important: match IP and source port) to apply SNAT.

3rd rule does MASQUERADE all other outgoing traffic from the subnetwork, as suggested by Google documentation (w/o the packets catched by the previous rule).

The conclusion is that NAT using MASQUERADE as suggested by Google documentation conflicts with port forwarding, unless the outgoing packets of the forwarded service are specifically intercepted with an SNAT rule as detailed above. The SNAT rule does not work for other outgoing traffic, so the MASQUERADE rule is still required.

The documentation should be updated accordingly.

[Anders Brown]

Unfortunately, I'm still having difficulties with GCE networking. It's hard to understand where routing is done by instances or by outside hosts. With hardware hosts this was a lot easier. The proposed fix worked for the simple httpd test scenario, but not in a more complete infrastructure, where the SNAT rule seems to interfere with a VPN subnet (sigh). This however may be limited to my specific setup, so I'll continue looking for a fix.

Basically, pairing DNAT with SNAT does preserve the originating IP, which is a useful finding for now. It would be great if the documentation eventually included an example for port forwarding.

[Faizan (Google Cloud Support)]

Hello Anders,

Thanks for your detailed information. I'll forward this feedback to the document team to update our docs.

Faizan

[Anders Brown]

Faizan, please forget the previous suggestion. Here's a solution that was now tested and proven in a more complete production network.

The challenge with port forwarding in GCE is that instances only have a single network interface, so packets forwarded with DNAT get their source IP replaced by SNAT immediately when they run through the POSTROUTING chain. As this happens before they reach their destination instance, it looks like DNAT didn't work. The fix now is to protect forwarded packets after DNAT from immediate SNAT by skipping that rule in POSTROUTING. The second rule below does exactly that:

iptables -t nat -A PREROUTING -i eth0 -p tcp -d 192.168.1.4 --dport 80 -j DNAT --to-destination 192.168.1.6

iptables -t nat -A POSTROUTING -o eth0 -p tcp -d 192.168.1.6 --dport 80 -j RETURN

The above rules can be repeated for any number of ports to be forwarded. The final rule below ensures all instances in the local network have Internet access w/o them having a public IP (this rule replaces the MASQUERADE rule suggested in the GCE documentation):

iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source 192.168.1.4

Note that a GCE firewall rule must allow all forwarded ports from anywhere, on both the head node and the service node.

This solution enables port forwarding to instances w/o a public IP, while those instances also have full access to the Internet. The latter also requires IP Forwarding to be enabled for the head node, and a default route added to the subnetwork, as explained in the current documentation. Would be great if this example could be added to the documentation.

[Anders Brown]

Forgot to mention the current documentation:
https://cloud.google.com/compute/docs/networking 
Section "Configure a NAT gateway"