Can't send network traffic from network load balancer IP address

[Felicity Tarnell]

Hello,

I have the following configuration at GCE:

* Two instances with external IP addresses and canIpForward set to true;

* Some number of instances without external IP addresses;

* A default (0.0.0.0/0) network route via the external instances.

The purpose of this is to allow the instances without external IP addresses to access the Internet via NAT.  So far, this works fine.

Now I've configured a network load balancer with a target pool containing the two external instances, because I want to run haproxy on those instances to load balance some services that run on the instances that don't have external IP addresses.[0]  This works fine: incoming traffic is load balanced to the two external nodes, and haproxy on those nodes sends traffic to the internal instances.

However, there's a problem: connections from internal instances (those without external IP addresses) to the network load balancer's IP address do not work.  I've spent a while investigating this and from what I can see, the traffic never reaches the actual load balancer because google_ip_forwarding_daemon on the router/haproxy instances configures the external IP address as a local route, meaning the instances handle it as local traffic.  That should be fine: the traffic will go the local haproxy as if I'd connected to the instance IP address directly.

Unfortunately what actually happens is the connections time out.  The haproxy instances see the SYN packet and reply to it:

13:42:00.872208 IP 172.31.208.13.33318 > 35.189.89.28.22: Flags [S], seq 1900468373, win 28400, options [mss 1420,sackOK,TS val 18308920 ecr 0,nop,wscale 7], length 0

13:42:00.872246 IP 35.189.89.28.22 > 172.31.208.13.33318: Flags [S.], seq 1981388720, ack 1900468374, win 28160, options [mss 1420,sackOK,TS val 1616706 ecr 18308663,nop,wscale 7], length 0

But the client never sees the reply:

13:42:44.966876 IP 172.31.208.13.33322 > 35.189.89.28.22: Flags [S], seq 2828185934, win 28400, options [mss 1420,sackOK,TS val 18319943 ecr 0,nop,wscale 7], length 0

13:42:45.992172 IP 172.31.208.13.33322 > 35.189.89.28.22: Flags [S], seq 2828185934, win 28400, options [mss 1420,sackOK,TS val 18320200 ecr 0,nop,wscale 7], length 0

13:42:48.008117 IP 172.31.208.13.33322 > 35.189.89.28.22: Flags [S], seq 2828185934, win 28400, options [mss 1420,sackOK,TS val 18320704 ecr 0,nop,wscale 7], length 0

(172.31.208.13 is the client instance, 35.189.89.28 is the network load balancer address.)

I believe the problem here is that packets from the router/haproxy instance are being dropped by GCE when the source IP address is the load balancer address, even though the instance has canIpForward set.  I tested this using hping: packets from a random address like 1.1.1.1 are sent fine, and traffic from the address of _other_ load balancers is sent fine, but when the source IP address is this instance's load balancer address, the traffic never reaches the other instance.

So my question is: what's causing this, and is there a way to change it?

I can probably work around the problem by creating two more, separate instances to run haproxy, so the load balancer isn't configured on the router anymore, but that's additional cost and complexity that I'd like to avoid.

Thanks,

Liss.

[0] I can't use a network load balancer because I need to change the destination port (22 -> 30022); I can't use a TCP load balancer because port 22 isn't permitted; I can't use an HTTP load balancer because the traffic isn't HTTP; so while I'd rather avoid running haproxy myself, that seems to be the only option.

[Faizan (Google Cloud Support)]

Hello Liss,

Have you taken a look at Google Internal LB? I believe setting up external TCP/UDP LB along with Internal LB will fit your use case. You can refer to this link for more information on Internal LB.

Faizan