How to deal with "Expect: 100-continue" *without* sending a 100 status

[Ian Rose]

Howdy,

I'm currently running a group of Go web servers behind an HTTP(s) load balancer on Google Compute Engine.  Unfortunately I have learned that GCE load balancers do not support the "Expect: 100-continue" header [1].  From my experiments, it appears that it isn't actually the request header that causes the problem, but instead is the server's "100 Continue" response that the load balancer dies on.  Specifically, the load balancer responds with a 502 to the client.

Any suggestions on how to deal with this?  We don't control our clients (they are just "browsers across the internet") so solving things on that side isn't possible.  After digging through the net/http code a bit, my best thought is to hijack the connection, which (I think) will prevent a "100 Continue" status from being sent.  I'm concerned, however, that this won't work in all cases - for example http2 connections are not hijackable (https://github.com/golang/go/issues/15312).

Is there a better path forward?

Thanks,

Ian

[1] https://code.google.com/p/google-compute-engine/issues/detail?id=298   (also see "notes and restrictions" here: https://cloud.google.com/compute/docs/load-balancing/http/)

[David Anderson]

Tricky one. A couple of options spring to mind, none of them amazingly good:

• Use a GCE Network LB instead of HTTP LB. You can bring the TCP sessions straight to your web servers, with load-balancing done per-TCP-session rather than per-HTTP-request.
• Build your web server using a modified Go stdlib codebase that removes this conditional block: https://golang.org/src/net/http/server.go#L1562 . If you do this, I suggest also filing a bug against Go to evaluate whether "don't do automatic Continue support" should be added as a Server knob.
• Stick the Go web servers behind a non-Go proxy layer (e.g. nginx) that strips out the "Expect: 100-continue" header before forwarding to the Go server. Run one nginx per Go server, on the same machines (or in the same Kubernetes pods if using GKE), so that the system properties look the same from the POV of the upstream load-balancer (same number of backends, same arrangement...).
• Wait for GCE to support 100-Continue. Given that 100-Continue is almost non-existent on the web, personally I wouldn't hold my breath, I suspect it's a low-priority item.
• You say your clients can't be modified... Can't they? I've never heard of browsers using 100-Continue unprompted, so if it is just chrome/firefox/IE, what are you doing that's causing them to use 100-Continue? Or are they some other client software like Mercurial?

- Dave

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ian Rose]

Thanks Dave - all good suggestions!

> Use a GCE Network LB instead of HTTP LB.

I'm hoping to avoid using a network LB since we are operating over https so that means we'll have to terminate SSL ourselves (which means distributing our certs to each backend instance).  Using an http(s) LB means we only have to upload our cert to our google cloud project and our backends only have to speak http - very convenient.

> You say your clients can't be modified... Can't they? I've never heard of browsers using 100-Continue unprompted, so if it is just chrome/firefox/IE, what are you doing that's causing them to use 100-Continue? Or are they some other client software like Mercurial?

I'm curious about this as well.  I've tried to find some info online on when chrome (or other browsers) use 100-continue, but haven't found anything useful or current.  I've also tried playing around locally and haven't been able to get my local chrome to send that header.  Our app in this case is a javascript snippet that sends POST XHRs with fairly large json payloads.  I managed to correlated in our logs (via user agents) that one of the clients that was served 502s (and thus presumably send us an Expect header) was Chrome 52 and another was Edge.  So certainly nothing too esoteric.

If we could somehow figure out a way to change our javascript so that we don't trigger whatever conditions convince browsers to (sometimes?) use this header, that would be great.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[jrw...@gmail.com]

Sorry to resurrect a month old thread, but I just implemented bullet #2 with this patch.

I need to disable the automatic 100-continue behavior so that my handler can reply with a 40X status before the client read the 100-contine and starts sending data.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.