Load Balancer - intermittent timeouts and 502s

[Jarda Grešula]

Hi,

We use a global HTTP load balancer and are seeing intermittent 502 errors and timeouts. We believe that we tracked it down to some network issue that happens between our backend and the load balancer IP address 130.211.3.126. We don't see any network issues with other balancer IP addresses in the range 130.211.0.0/22.

For instance, the attached packet capture shows two HTTP POSTs to /convert/ on our backend. Our backend answers the first one (packet #4) with HTTP 408 (#6) but it takes the balancer almost two seconds to send an ACK (#9). The backend's response to the next /convert/ request (#12) on the same connection is not ACKed at all (no answer within 55s). We have several such packet captures and 130.211.3.126 is always involved.

Please advise. Let us know if you need more information about our project.

Regards,

Jarda Gresula

[Karthick (Cloud Platform Support)]

Hello Jarda,

A 502 error is a "bad gateway" response. What is health check status of the instance  at the time that the 502 error were occurring? It’s possible that your backend instance was unhealthy, causing failures.  Have you checked  the load balancing log entries of HTTP(S) traffic from log viewer?

What is the health parameter of your backend services, is it using the default, or have you customized it?

If you can provide me with your project ID, instance and load balancer name through a private message, I’ll try to investigate the issue further to find the root cause of the bad gateway response.

[Jarda Grešula]

Hi Karthick,

We see several 502s a day out of thousands requests that are processed by our backend. The balancer's backend is an instance group consisting of 2 instances (autoscaling is turned off). The health parameter is customized. The instances are healthy at the time the error occurs.

The backend service uses nginx that is configured with "keepalive_timeout 650" and "keepalive_requests 10000". We don't see any related errors in nginx logs.

All packet captures we have show that the problem occurs when 130.211.3.126 communicates with the backend. The netstat output on the backend instance constantly shows 50+ established connections between the backend and the balancer IP addresses from the 130.211.0.0/22 range but none of them is 130.211.3.126.

Most often we see "failed_to_connect_to_backend" and then "backend_connection_closed_before_data_sent_to_client" and "backend_timeout" errors in the log viewer.

Details follow in a private message. 

Thank you, Jarda.

[Karthick (Cloud Platform Support)]

Jarda,

Would you able to send me the sniffer file, along with the timestamp of the issue in a private message? I would also like to know more about the number of connections you are receiving through the load balancer, in particular, the number of connections that each instance of nginx is receiving. I will be taking this information to the engineering team so that they can investigate further. 

[Karthick (Cloud Platform Support)]

Hi Jarda,

I have forwarded your logs to engineering so that they may have it investigated. We presume you might be affected by an ongoing issue that is currently being worked on. And to confirm, can you enable debug logging and provide a sample from when a 502 occurs and along with the nginx configuration? This will help us see if the issue is related.  

[Jarda Grešula]

Hi Karthick,

Do you have any updates on the issue? 

Thank you,

Jarda

[Karthick (Cloud Platform Support)]

Hello Jarda,

I am currently working with the engineering team and unfortunately I am not able to provide an E.T.A on the update. I will get back to you as soon as I have any information. 

[Jarda Grešula]

Hi Karthick,

Could you at least confirm if it is a known LB issue? We are seeing a steep increase of 502s and timeouts in the past few days. 

Thank you, Jarda.

[Karthick (Cloud Platform Support)]

Hello Jarda,

Thank you for sending the pcap information today. Our further investigation could able to link this issue to another recently identified issue with cpu utilization of one of the components used by GCLB in us-central1-b.  

However I have forwarded your information to the internal engineering today, 4.5 seconds is the GFE's backend connection timeout. If the GFE doesn't successfully connect in that time, it'll generally retry once, so what you saw is consistent with the GFE never getting the SYN,ACK.

I will let you know of any update at once I have. 

Thank you for your patience. 

[Daniel Compton]

Hi Jarda

I've seen this issue too. There have been periods of much higher 502's being returned, along with a low-level (2-10) of 502's being returned every day. This is a real blocker for me, and I'm going to have to look at moving elsewhere if it can't be fixed.

-- Daniel.

[Evan Jones]

Hmm... My other reply to this got stuck in a moderator queue it seems:

This thread sounds extremely similar to an issue I've been seeing for months. I'm seeing a few 502 errors with statusDetails: "failed_to_connect_to_backend" each day, also in us-central1-b. Is this a known issue? Is there any configuration I can change to avoid it? It appears from my debugging that requests coming into my VM are occasionally suffering congestion and/or packet loss. I see this extremely rarely for traffic from other VMs on the same network, but frequently for traffic from GCLB. Could this be noisy neighbours or my VMs?

Details:

I'm running a Google Kubernetes Engine cluster with an Ingress using the Google Cloud Load Balancer, in us-central1-b. A few times a day we see a 502 logged with statusDetails: "failed_to_connect_to_backend". I finally have a "test" setup that seems to produce it once a day, so I've been trying to debug in further. My VMs are n1-standard-1, and are nearly completely idle (they are basically only running this "trivial" service).

So far, when I examine the logs and packet captures, I observe the following issues:

* Some requests through the load balancer take > 1 second. Usually when that happens, in the server side packet trace I see two SYN packets arrive within ~200 microseconds of each other. This suggests to me that something is timing out and retransmitting, but then the packets are getting queued and arriving in a burst. (See attached screenshot: The packet number 33796 and 33798 are duplicate SYNs).

In SOME cases I see a "correct" HTTP request, but it just arrives late enough to cause the high latency (e.g. maybe the initial SYN was dropped and the retransmit worked).

* Some requests through the load balancer take ~4.5 seconds, which I guess is the "retry" timeout. In these cases: I don't see the initial request AT ALL. I just see a 4.5 second gap in the request traces where the initial request should be.

* In the case of 502s, the request takes ~9 seconds, and again I don't see ANYTHING on the server side from this request.

* A smaller number of requests from other VMs take > 1 second. In these cases, on the client side I can see that the SYN is retransmitted, and it seems highly likely that this packet is getting dropped and retransmitted.

[Evan Jones]

This thread sounds extremely similar to an issue I've been seeing for months. I'm seeing a few 502 errors with statusDetails: "failed_to_connect_to_backend" each day, also in us-central1-b. Is this a known issue? Is there any configuration I can change to avoid it? It appears from my debugging that requests coming into my VM are occasionally suffering congestion and/or packet loss. I see this extremely rarely for traffic from other VMs on the same network, but frequently for traffic from GCLB. Could this be noisy neighbours or my VMs?

Details:

I'm running a Google Kubernetes Engine cluster with an Ingress using the Google Cloud Load Balancer, in us-central1-b. A few times a day we see a 502 logged with statusDetails: "failed_to_connect_to_backend". I finally have a "test" setup that seems to produce it once a day, so I've been trying to debug in further. My VMs are n1-standard-1, and are nearly completely idle (they are basically only running this "trivial" service).

So far, when I examine the logs and packet captures, I observe the following issues:

* Some requests through the load balancer take > 1 second. Usually when that happens, in the server side packet trace I see two SYN packets arrive within ~200 microseconds of each other. This suggests to me that something is timing out and retransmitting, but then the packets are getting queued and arriving in a burst.

In SOME cases I see a "correct" HTTP request, but it just arrives late enough to cause the high latency (e.g. maybe the initial SYN was dropped and the retransmit worked).

* Some requests through the load balancer take ~4.5 seconds, which I guess is the "retry" timeout. In these cases: I don't see the initial request AT ALL. I just see a 4.5 second gap in the request traces where the initial request should be.

* In the case of 502s, the request takes ~9 seconds, and again I don't see ANYTHING on the server side from this request.

* A smaller number of requests from other VMs take > 1 second. In these cases, on the client side I can see that the SYN is retransmitted, and it seems highly likely that this packet is getting dropped and retransmitted. It is suspicious to me that the retransmitted packets are ONLY ever SYNs.

[Pavel Madr]

Hi,

I see the very same problem with the same observation like Evan.

The problem appears also in other us-central1 zones, but not so often like in us-central-1b.

[Matthew Lenz]

I see this issue constantly in our load balancer logs.  It's increased over the last several months.  The instances are nowhere near capacity and fully healthy with regards to health checks.

[Daniel Compton]

I've setup a staging environment in us-east4 on the advice of some people who said this only affected us-central1. So far (24 hours) I haven't seen any 502's, where I have had ~10 in us-central1 over the last 24 hours. I'll update in a week on whether this continues to hold.

On Thu, Dec 7, 2017 at 5:03 AM Matthew Lenz <ml...@nocturnal.org> wrote:

I see this issue constantly in our load balancer logs.  It's increased over the last several months.  The instances are nowhere near capacity and fully healthy with regards to health checks.

--
© 2017 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to a topic in the Google Groups "gce-discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gce-discussion/7ETsl0YH1iQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/e7f897de-5f1f-462b-9424-ada2d14ecc77%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthew Arbesfeld]

Hi Daniel, have you been able to fix this issue in us-east?

[Daniel Compton]

Hi Matthew

I didn't see any more issues during my test for ~3 days in us-east4, (setting up mirror Stackdriver checks to prod) while I continued to see them in us-central1. I'm planning on migrating the production service to us-east4 but that is a bigger job...

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/5fadd620-d6bb-41a6-9052-0020aeeb7151%40googlegroups.com.

[Jono MacDougall]

Hi Jarda, 

We have also seen a very similar issue as you describe. 502s, failed_to_connect_to_backend, etc. We were able to catch it in the act with simulated traffic and noticed the latency was ~9 seconds when the issue happens. This lines up nicely with the 4.5s timeout and retry mentioned above. We are trying to capture a pcap of it in action now but it only happens once every 2-6 hours so it is taking some time.

The biggest difference here is we are running in eu-west1. Do we know if this is a load balancer issue or is this a configuration problem? It is becoming a big concern for us and our clients are starting to notice the errors. 

Thanks,

Jono

[Martin Runelöv]

We are also seeing intermittent 502s with "failed_to_connect_to_backend".

We have a "global" multi-cluster load balancer but we only have backends in asia-south1 so far.
However, I can see the North America and Europe frontends receiving a small amount of traffic. Would this be a possible cause of 502s, or should they be forwarded to our existing backends in Asia?

[Rohit Singh]

I can confirm that we're facing this in asia-south1 region, over the last 10 days without any changes being done at our backend, and the instances which are serving traffic are running at minimal capacity.

Has anyone found a solution to this yet ?

Regards,

Rohit S

[Evan Jones]

I should have followed up on my post: Based on a discussion in the Google Cloud Community slack #networking channel [1], it appears there is some Google Cloud issue with Load Balancers, at least in some regions. Specifically, another customer said they got the following comment on one of their support tickets:

"Our engineering team is currently aware of an issue in us-central1 that affects a very small subset of our load balancers and causes low-level packet loss on the load balancers. If the right set of packets are dropped, you would end up with the failed_to_connect_to_backend error you are seeing."

This doesn't explain if this could be the same issue in eu-west1 or asia-south1, but looking at the log there were complaints about us-east1. Another customer reported they tested us-east4 and did not observe issues over ~2 days, but I'm not aware of any of the details. I can also report that I'm still see about one of these every ~2 days, but our app is in us-central1. My only suggestion is to file a support ticket and see if they can help, and let's hope Google fixes the underlying root cause soon.

Evan

[1] GCP Community Slack: https://gcp-slack.appspot.com/

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to a topic in the Google Groups "gce-discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gce-discussion/7ETsl0YH1iQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/a4ca9e73-0c57-4388-a0ea-64947a74cc33%40googlegroups.com.

[Rohit Singh]

Thanks Evan. Ironically, the gcp-slack appspot link first gave the 502 timeout error, and loaded on consequent reload.

Regards

[Evan Goldenberg]

We're seeing this issue as well and it is quite severe. Up to 10% of requests are taking 4.5-5 seconds when that would typically take a couple hundred ms at most. That appears to be consistent with the 4.5s retry threshold mentioned above. 502s are not as common, but we are seeing a lot of those too.

[Matthew Lenz]

My personal opinion is that everyone should request billing credits every month until it is resolved.  Be sure to reference this discussion in the request.

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to a topic in the Google Groups "gce-discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gce-discussion/7ETsl0YH1iQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/a51a464d-b080-4df2-a81a-afa0b8e55f60%40googlegroups.com.

[Julius Žaromskis]

Hi all,

we're seeing this issue as well in europe-west-1. Intermittent

[Jono MacDougall]

We were able to capture some traffic from this when the issue occurred.  It looks like our server is getting traffic from the load balancer but traffic from our server is not making it back to the load balancer. The ACKs are being lost somehow so the load balancer keeps retrying.  At some point the load balancer seems to give up and send a FIN which is ACK'ed by nginx but this ACK is also lost and the load balancer continues to attempt to send traffic on this stream for almost 50 seconds after the FIN should have closed it. Again, this implies the FIN ACK from nginx is not reaching the load balancer. 

In this particular case, this was localised to a specific load balancer IP. Traffic proceeded as normal against other load balancer IPs.

This really does look like a problem with the google load balancer and not a client side issue. If so, this needs to be escalated and fixed. This is not acceptable for something as simple as a load balancer. 

[Evan Jones]

Hmm... This seems similar to the symptoms I observed, but not totally the same. When I examined this in December, I saw that in the case of failed_to_connect_to_backend errors, we didn't see ANYTHING on our server from the load balancer. However, we did see the issues you describe for requests that were unusually slow: E.g. we see some requests that take ~4.5 seconds, and in those cases we can see SYN retries from the load balancer.

I don't recall if in our case the issues were with a single load balancer IP or not. I guess I should possibly attempt to reproduce the issues again ...

Thanks for the report!

Evan

--

© 2017 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to a topic in the Google Groups "gce-discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gce-discussion/7ETsl0YH1iQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/5248f7bd-cabb-4f3a-90da-a063a5f2c38c%40googlegroups.com.

[Dave H]

We are seeing this in us-east-1 on a regular basis.  We are running a repository for large files there and these intermittent 502s are keeping users from being able to download any file > 2GB.

[Jono MacDougall]

Over the last few days I have not seen any 502 - `failed_to_connect_to_backend` errors in the load balancer. I believe a recent lb deploy has fixed the issue. Please chime in if you are still seeing the error but from our perspective this issue is fixed.

[Matthew Lenz]

Just before midnight CST last night is the last one I've had.  "2018-02-05T05:32:01.696881876Z" is the last time stamp

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043

 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to a topic in the Google Groups "gce-discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gce-discussion/7ETsl0YH1iQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/1e5329b5-cf81-4e87-ac2d-251dcb75431b%40googlegroups.com.

[Pavel Madr]

We have the latest at 2018-01-31 12:03:06.491 UTC. It seems to be fixed. Thanks Google.

[Julius Žaromskis]

I'm seeing a major improvement since 2018-02-02 in europe-west-1. Although some 502 errors still happen. Also seeing this new status occasionally

[Kevin Marsh]

Adding that we, too, are seeing these issues using a Google Load Balancer in us-central-1. The LB is a K8s Ingress in front of a Rails apps running Puma. Similar symptoms as mentioned: 502s, failed_to_connect_to_backend for a % of requests. I'm attaching a graph that shows % of requests affected by day. Our worst day was yesterday with ~1.3% of requests.

Also noticed backend_service_name is blank, but for all requests not just failed ones. I've observed less than 100% of instances being marked healthy by the load balancer, but saw the instances that were marked unhealthy were simply not being scheduled to run the pods for our app by K8s, so I'm not sure why they were even being checked. Also, the cluster seems well below capacity so I don't know why 1 or 2 backends being down would cause requests to fail and not just be sent to a "healthy" backend.

I've filed a case with support, but haven't gotten anywhere so far.

[Julius Žaromskis]

I am now getting occasional "backend_connection_closed_before_data_sent_to_client" errors. In fact it seems that all `failed_to_connect_to_backend` errors were replaced with aforementioned status. That does not make sense, because my server is configured with keep alive timeout of 620s as instructed here: https://cloud.google.com/compute/docs/load-balancing/http/

I have manually confirmed that idle connections are closed after 620s by telneting to port 80

What's more - I'm getting these on different services nginx, nodejs and multiple different VMs. That means it's not just one server misconfigured.

I know there's another "feature" where GCE would kill connections from internet after 600s. Could this be killing TCP sessions to load balancer? I'll experiment with TCP keepalive.

On Monday, October 23, 2017 at 11:00:12 PM UTC+3, Karthick (Cloud Platform Support) wrote:

Hello Jarda,

A 502 error is a "bad gateway" response. What is health check status of the instance  at the time that the 502 error were occurring? It’s possible that your backend instance was unhealthy, causing failures.  Have you checked  the load balancing log entries of HTTP(S) traffic from log viewer?

What is the health parameter of your backend services, is it using the default, or have you customized it?

If you can provide me with your project ID, instance and load balancer name through a private message, I’ll try to investigate the issue further to find the root cause of the bad gateway response.

[ra...@skimlinks.com]

Just FYI,

I had the same problem that everybody relates here.

- Intermittent 502,

- Average of 9 seconds in the failed requests

- Same message in the load balancer logs "failed_to_connect_to_backend"

- Request in the GKE node (not using the load balancer) has a 100% availability

In our case the problem was incorrect configuration of ports for the backends.

So we have two backends matching two GKE pools: the first one had the right port 31323 the second one was configured by mistake with the port 80.

Change the port solved the problems.

[Marc]

Since the last week, the problem seems to be active again. We experience a lot of 502 errors too on a GAE app (us-central)?

We have no choice to redeploy app more than once a day to temporally fix that.

[Alex G]

It seems we also have the same issue, but we are using App Engine Flexible, which I believe is using the same HTTP load balancer as GCE under the hood.

Is this something happening with all requests, or just POST ones? I guess this shouldn't matter if the issue is related to the TCP layer, though. I just found it interesting that a few people mentioned the issue happening with an HTTP POST request.

[Pavel Madr]

We are still getting a few 502 errors a day. It appears rarely for GET request too.

[Justin Reiners]

I am using 11 different balancers, of all types with a metric ton of traffic, We are not witnessing any 502 errors or timeouts.

I host in us-central1, and have had no issues at all. This is strange.

--

© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---

You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.

To post to this group, send email to gce-discussion@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/2e873ee6-27f2-4d9f-8652-f305c3f91a0c%40googlegroups.com.

[Dinesh (Google Platform Support)]

If you consider this is a bug, please raise this issue on Google public issue tracker(PIT)[1] platform. PIT meant for bugs and feature request tracking and their management. 

[1]: https://issuetracker.google.com 

[rich mudder]

Can't help you.

--

© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/78e81cd7-1380-4a82-9d41-27164b36f5c3%40googlegroups.com.