502 Gateway Error + squid TCP_MISS_ABORTED

[Spike]

Hi,

I'm having problems with some site where browsers get back a 502. Looking at the logs, squid shows a TCP_MISS_ABORTED/200, but nothing else logged in e2g.

anybody knows what's going on? it all happens *very* quickly, a few secs, so it doesn't seem to be a timeout to me.

also I tested on the server where e2g + squid are running. Doing a curl going through squid works just fine as it does a curl without being proxied at all. But if I curl -x localhost:8080 I get the bad gateway so it's definitely something connected to e2g, not the site or squid altho obviously the combination of the above is a problem.

any ideas?

Spike

[FredB Numsys]

Do you have something in syslog ?

--
Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser ma brièveté.

[FredB]

> also I tested on the server where e2g + squid are running. Doing a
> curl going through squid works just fine as it does a curl without
> being proxied at all. But if I curl -x localhost:8080 I get the bad
> gateway so it's definitely something connected to e2g, not the site
> or squid altho obviously the combination of the above is a problem.
>
>

This is related with some specific websites or no matter ? Always the same ?
Through Squid but the same nic ? curl -x localhost:3128 ?

[Philip Pearce]

502 Gateway error indicates a network error in either connecting to squid or a connection that has been aborted either by squid or by something in the network path connecting e2g to squid. (check firewalls on e2g box and on squid box, network stack errors, errors on switches, port settings on switches, etc).

More information as to whether the problem is on connection or an abort once connected is in the body of the 502 response sent to the browser and also put in the syslog.

If a timeout is detected then you would get a 504 Gateway Time-out response.

Regards

Philip

---

--
You received this message because you are subscribed to the Google Groups "e2guardian" group.
To unsubscribe from this group and stop receiving emails from it, send an email to e2guardian+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Spike]

Thanks for all the input.

Responding to various questions/comments so far:

- this is only happening with one specific website.

- If I connect with -x localhost:3128, ie I connect directly bypassing e2g, the problem does not manifest.

- If I curl -x localhost:8080, ie go through e2g, I get the error. 

- I've used ---interface with both outgoing uplink to make sure, same behavior on both

- the error is immediate, ie not a timeout, unless the timeout somehow is in the order of ms because I get the error as soon as I hit enter on the curl command, no delay whatsover

- given that all other websites work without problems it seems unlikely this is a network error

- like I said squid in the logs says "TCP_MISS_ABORTED", vs the more normal TCP_MISS I'm used to see. I don't know if that's somehow causing problems to E2G but it's the only idea I have right now, altho I don't know why that would even be a problem. And in fact, I've just retested, and I guess because it was curl'ed successfully it's now in squid's cache so new hits in the logs are now showing as TCP_MEM_HIT. Nonetheless even if now I go through e2g I still get the same error.

- I have nothing in syslog.

thanks for any input,

Spike

[FredB]

>
> - this is only happening with one specific website.

It's a confidential url or I can make a test ?

Fred

[Spike]

some users now told me the issue is happening with others sites, but got no urls yet, will share as those come in.

that said, I finally got the time to run e2g in debug mode and found something interesting. This is at the ever end of the log:

header:in after getLine - firsttime : 0 Line: 2107 Function: in

header:size too big =  41 Lines: 2120 Function: in

58156 -Attempting graceful connection close Line: 3201 Function: handleConnection

handle_peer returned: 0

Compiling ,*[a-z|A-Z].*

...with PCRE 

 in handle connection

 waiting connection on http_worker_Q 140066517513984

busychildren:0

worker Q size:0

So that seems the reason for the 502. Indeed when I look at the previous lines this particular IIS server seems to return a *ton* of headers, including a lot of stuff from a varnish instance I'm assuming they have in front of their IIS or something.

hope that provides a clue.

thanks for any input.

spike

[FredB]

Hi,

Can you make a try with 100 (just for testing)

# Limit number of http header lines in a request/response

# (to guard against attacks)

# Minimum 10 max 2000

# default 40

# maxheaderlines = 40

[Spike]

indeed that was it. I don't think I've ever even seen that option, thanks for pointing it out to me. I guess I'll keep it to 100 and see if people complain about other sites, there were a few more reports coming in, but more than 100 headers really seems nuts.

I've added a small section to the docs to document this behavior as I may not be the only one running into it, and it's not immediate to debug it because normally there's no message printed. This may actually be a valid feature request, I'll create an issue for it for consideration. In the meantime the docs are here:

https://github.com/e2guardian/e2guardian/wiki/Configuration#misc

thanks again for all your help. 

Spike