Load balancer returning 502; "backend_early_response_with_non_error_status"

[Dylan Scott]

I have a service running on GCP behind a load balancer, configured as described here. Largely the service seems to work fine, but I'm encountering an issue specifically with one endpoint, where the load balancer is definitely forwarding the request to the backend, but then returning a 502. If I look at the request in the Load Balancer section of the Stackdriver Logging page, I can see that some metadata has been attached under the "structPayload" key. In particular:

structPayload: {

  statusDetails: "backend_early_response_with_non_error_status"

  @type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"

}

I can't seem to find any documentation for this "backend_early_response_with_non_error_status" message. Looking at my service logs, the request definitely makes it there.

[Dylan Scott]

Actually it looks like this was my bad, not reading the page I linked closely enough. It notes that the load balancer does not support HTTP/1.1 100 Continue responses, which is what the backend is sending in this case.

[Dylan Scott]

Though actually, I would love some advice on what my options are if anyone has dealt with this issue before. My situation:

1. A client, who I do not control, sends requests to my endpoint with the Expect: 100-continue header.
2. The load balancer is configured to send requests to an nginx proxy, which ultimately forwards requests to the proper backend. It appears that nginx is the one immediately sending the 100 Continue, and from what I can gather there is no way for me to disable this behavior.

Am I out of luck here?

[Dylan Scott]

Apologies for the spam, but in case anyone stumbles on this thread I was able to work around this using the approach described in this bug. This required building a custom nginx image which includes the headers-more-nginx-module, but it seems to work. Hopefully the load balancer will be fixed soon.

[Ian Rose]

The timing of this is rather coincidental - I just concluded a thread with google support for the same reason.

Dylan - we are considering a similar solution (ignoring the Expect header on the server, although we are using a Go server, not nginx).  If you do this, you are counting on clients to give up and time out after not receiving a "100 Continue" response for some amount of time, and sending their request body anyways, right?  We are worried that we might break something by doing this.  Are you confident that all modern browsers have a timeout-fallback like this?  Do you have any idea what the timeouts usually are (i.e. how much additional latency will we be adding)?

Thanks!

 - Ian

[Dylan Scott]

According to the HTTP/1.1 RFC:

Because of the presence of older implementations, the protocol allows ambiguous situations in which a client may send "Expect: 100- continue" without receiving either a 417 (Expectation Failed) status or a 100 (Continue) status. Therefore, when a client sends this header field to an origin server (possibly via a proxy) from which it has never seen a 100 (Continue) status, the client SHOULD NOT wait for an indefinite period before sending the request body.

I don't actually know what most clients do in practice, but the one I'm having to deal with (a service sending a webhook) seems well behaved. I can definitely imagine that not all clients do this though, I just couldn't find any other alternative given that my service is behind Google's load balancer.

[Ian Rose]

FWIW I found that in my case the 100-continue thing was a total red herring.  The vast majority of my error were actually due to my server timing out and closing idle tcp connections opened by the load balancer.  If you still run into any 502s in your system, I encourage you to take a tcpdump on your server - very illuminating for me.