error message "upstream connect error or disconnect/reset before headers"

2,559 views
Skip to first unread message

ibra...@gmail.com

unread,
Feb 3, 2018, 7:57:41 AM2/3/18
to envoy-users
Hi everyone, 

I've been facing a strange issue the past couple of days...

I'm proxying a web server (nginx) behind envoy (heptio-contour on a k8s with ingresses).

On requesting the main page (index.html), I'm getting the following error message as a response: upstream connect error or disconnect/reset before headers

I'm sure the web server is reachable from the envoy container and can get 404s with no issues from the web server. I can even request the main page from the container with no issues and I get the full response from the web server.

After some fiddling around I found the issue. It turns out that one of the headers had some unicode characters in the header key: "X%u2010XSS%u2010Protection" (X-XSS-Protection). Changing the key to use ASCII dashes instead resolved the issue.

Is this expected behavior? 

Regards,
Ibrahim

Matt Klein

unread,
Feb 3, 2018, 4:25:45 PM2/3/18
to ibra...@gmail.com, envoy-users
HTTP header field names can only contain a subset of ASCII characters, and the parsers Envoy uses are strict. See:

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/b4f2321b-89c8-4420-b3ba-b6a103e0ad62%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

Ibrahim AshShohail

unread,
Feb 4, 2018, 12:39:57 PM2/4/18
to Matt Klein, envoy-users
Is it expected behavior to treat a backend with invalid headers as failing?

If so, I'd think it'll be better if this is logged somewhere. I wasted a few days trying to figure out what was wrong.

On Sun, Feb 4, 2018 at 12:25 AM Matt Klein <mkl...@lyft.com> wrote:
HTTP header field names can only contain a subset of ASCII characters, and the parsers Envoy uses are strict. See:

On Sat, Feb 3, 2018 at 4:57 AM, <ibra...@gmail.com> wrote:
Hi everyone, 

I've been facing a strange issue the past couple of days...

I'm proxying a web server (nginx) behind envoy (heptio-contour on a k8s with ingresses).

On requesting the main page (index.html), I'm getting the following error message as a response: upstream connect error or disconnect/reset before headers

I'm sure the web server is reachable from the envoy container and can get 404s with no issues from the web server. I can even request the main page from the container with no issues and I get the full response from the web server.

After some fiddling around I found the issue. It turns out that one of the headers had some unicode characters in the header key: "X%u2010XSS%u2010Protection" (X-XSS-Protection). Changing the key to use ASCII dashes instead resolved the issue.

Is this expected behavior? 

Regards,
Ibrahim

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.

Matt Klein

unread,
Feb 4, 2018, 4:52:48 PM2/4/18
to Ibrahim AshShohail, envoy-users
Is it expected behavior to treat a backend with invalid headers as failing?

Yes. If the upstream speaks a protocol we can't parse, there is nothing we can do other than return a 503.

 If so, I'd think it'll be better if this is logged somewhere.

If you turn on debug logging (-l debug) I would imagine it would be pretty obvious. Also the upstream protocol_error stat will be incremented.

On Sun, Feb 4, 2018 at 9:39 AM, Ibrahim AshShohail <ibra...@gmail.com> wrote:
Is it expected behavior to treat a backend with invalid headers as failing?

If so, I'd think it'll be better if this is logged somewhere. I wasted a few days trying to figure out what was wrong.
On Sun, Feb 4, 2018 at 12:25 AM Matt Klein <mkl...@lyft.com> wrote:
HTTP header field names can only contain a subset of ASCII characters, and the parsers Envoy uses are strict. See:

On Sat, Feb 3, 2018 at 4:57 AM, <ibra...@gmail.com> wrote:
Hi everyone, 

I've been facing a strange issue the past couple of days...

I'm proxying a web server (nginx) behind envoy (heptio-contour on a k8s with ingresses).

On requesting the main page (index.html), I'm getting the following error message as a response: upstream connect error or disconnect/reset before headers

I'm sure the web server is reachable from the envoy container and can get 404s with no issues from the web server. I can even request the main page from the container with no issues and I get the full response from the web server.

After some fiddling around I found the issue. It turns out that one of the headers had some unicode characters in the header key: "X%u2010XSS%u2010Protection" (X-XSS-Protection). Changing the key to use ASCII dashes instead resolved the issue.

Is this expected behavior? 

Regards,
Ibrahim

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.

To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/b4f2321b-89c8-4420-b3ba-b6a103e0ad62%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Reply all
Reply to author
Forward
0 new messages