[Cherokee] Bad Gateway with Cherokee .99.22 as Reverse Proxy

[Yo'av Moshe]

Hey,
I updated to Cherokee 0.99.21 and then to .22 as soon as it was out, and I love the new status page and the graphs. Congratulations!

It seems that ever since the upgrade, I'm having various problems with Cherokee as a Reverse Proxy. I'm using Cherokee to serve a TurboGears application, and ever since 0.99.21 I get "502 Bad Gateway" all the time on certain pages. Most of these pages are where my TurboGears application is trying to redirect the browser, but I also have redirects that works.

I'm not sure what other information to provide, but I'm also having troubles to downgrade to 0.99.20 because my config file has 0.99.21+ specific keys now ("ERROR: Server parser: Unknown key "collector""), so basically my application isn't running too-good now... :(

Please help!

P.S.
Needless to to say, all these pages that I get "502 Bad Gateway" for are working perfectly when not connecting through the reverse proxy.
Also, my code hasn't changed, so these 2 things make me believe something got wrong with Cherokee... :-s

Thanks!

--
Yo'av Moshe

[Yo'av Moshe]

Hey,
Anyone seen this behavior? Any ideas on what to check?
As I said, I'm pretty sure something has changed with Cherokee between 0.99.20 to 0.99.21 that causes these "502 Bad Gateway" errors.

My server is a production server and I'm right now it's not quite working as it should... :-p

Yo'av

2009/8/12 Yo'av Moshe <bje...@gmail.com>

--
Yo'av Moshe

[Miguel Angel]

I don't have any idea, but may be that could be related with the php-cgi behaviour alex is suffering.

2009/8/13 Yo'av Moshe <bje...@gmail.com>

Hey,
Anyone seen this behavior? Any ideas on what to check?
As I said, I'm pretty sure something has changed with Cherokee between 0.99.20 to 0.99.21 that causes these "502 Bad Gateway" errors.

My server is a production server and I'm right now it's not quite working as it should... :-p

Yo'av

2009/8/12 Yo'av Moshe <bje...@gmail.com>

Hey,
I updated to Cherokee 0.99.21 and then to .22 as soon as it was out, and I love the new status page and the graphs. Congratulations!

It seems that ever since the upgrade, I'm having various problems with Cherokee as a Reverse Proxy. I'm using Cherokee to serve a TurboGears application, and ever since 0.99.21 I get "502 Bad Gateway" all the time on certain pages. Most of these pages are where my TurboGears application is trying to redirect the browser, but I also have redirects that works.

I'm not sure what other information to provide, but I'm also having troubles to downgrade to 0.99.20 because my config file has 0.99.21+ specific keys now ("ERROR: Server parser: Unknown key "collector""), so basically my application isn't running too-good now... :(

Please help!

P.S.
Needless to to say, all these pages that I get "502 Bad Gateway" for are working perfectly when not connecting through the reverse proxy.
Also, my code hasn't changed, so these 2 things make me believe something got wrong with Cherokee... :-s

Thanks!

--
Yo'av Moshe

--
Yo'av Moshe

_______________________________________________
Cherokee mailing list
Cher...@lists.octality.com
http://lists.octality.com/listinfo/cherokee

--
Miguel Angel Ajo Pelayo
http://www.nbee.es
+34 91 120 1798
+34 636 52 25 69
skype: ajoajoajo

[Alvaro Lopez Ortega]

Hello there,

On 13-ago-09, at 15:13, Miguel Angel wrote:

> I don't have any idea, but may be that could be related with the php-
> cgi behaviour alex is suffering.

I'll investigate it as soon as I finish some really urgent stuff I
must finish today.

Meanwhile, could you please check whether reverting the change-set
3488 helps?

--
Greetings, alo
http://www.alobbs.com/

[LinuxInsight]

Yo'av Moshe wrote:
> Hey,
> Anyone seen this behavior? Any ideas on what to check?
> As I said, I'm pretty sure something has changed with Cherokee between
> 0.99.20 to 0.99.21 that causes these "502 Bad Gateway" errors.
>
> My server is a production server and I'm right now it's not quite
> working as it should... :-p
>

Yo'av, don't know enough about your setup, sorry, joined list just today.

But, from some extensive testing I did few days ago, I know one thing.
Using UNIX sockets for communication between cherokee and php-cgi farm
is very fragile. At some point in time, under high load, php-cgi's
become unresponsive and cherokee starts retunring 504 Gateway timeout.
To make matters worse, the problem won't remedy itself, full restart
(cherokee + php-cgi) is required to fix it. Now, I don't know the cause
of the problem, it might be even in kernel (some race condition in
socket code under high load), php-cgi mechanism or the way cherokee
communicates with it...

What I do know is that using TCP/IP sockets for the communication (e.g.
127.0.0.1:9000, instead of /tmp/cherokee-php.socket) makes it much more
robust. And before anyone says that using TCP/IP is slower, I don't
think so, couldn't find any difference in speed, although I didn't
measure it extensively. It seems that localhost TCP/IP transport on
Linux is heavily optimized. But, my primary concern at this time is
robustness.

Anybody having gateway problems, try switching to TCP/IP communication,
you might be pleasantly surprised.
--
http://www.linuxinsight.com/

[Yo'av Moshe]

Hey everyone!

I'm *not* using PHP anywhere...
My application is a TurboGears application, which is running it's on server and Cherokee is standing infront of it as a Reverse Proxy. For what it worth - it's a Python application.

Alvaro, I'll try to test r3488 and report here.
Meanwhile I did this test:
I reverted only handler_proxy.c to r3165, which seems to me like the first change since 0.99.20, and it didn't help.
I checked the same TurboGears server - when connecting directly it all works, and through Cherokee 0.99.22 I get '502 Bad Gateway'. I noticed that Cherokee says "NOTICE: Taking source='localhost:8008' off-line" when I try to get the page, and after a while it takes the source 'back on-line'. It happens for all my sources (not just the one on port 8008), of course, on the same pages.

Thank you.
Yo'av

2009/8/13 LinuxInsight <linp...@gmail.com>

--
Yo'av Moshe

[LinuxInsight]

Yo'av Moshe wrote:
> Hey everyone!
>
> I'm *not* using PHP anywhere...
> My application is a TurboGears application, which is running it's on
> server and Cherokee is standing infront of it as a Reverse Proxy. For
> what it worth - it's a Python application.

What is your backend web server (the one with python application) and
how are you connected to python interpreter? Don't know how Turbogears
operates, does it have built in HTTP server?

Also, when you get those gateway errors, is your backend server
functioning properly?

[Yo'av Moshe]

TurboGears uses an HTTP server called CherryPy.
I'm runinng it on ports 8000 to 8010, and Cherokee is proxying it.

It is working properly.

Thanks,
Yo'av.

2009/8/13 LinuxInsight <linp...@gmail.com>

Yo'av Moshe wrote:

Hey everyone!

I'm *not* using PHP anywhere...
My application is a TurboGears application, which is running it's on server and Cherokee is standing infront of it as a Reverse Proxy. For what it worth - it's a Python application.

What is your backend web server (the one with python application) and how are you connected to python interpreter? Don't know how Turbogears operates, does it have built in HTTP server?

Also, when you get those gateway errors, is your backend server functioning properly?
--
http://www.linuxinsight.com/

--
Yo'av Moshe

[Yo'av Moshe]

Alvaro,
I'm sorry, but I can't seem to check anything from the SVN server.

~/% svn co svn://svn.cherokee-project.com/cherokee/trunk@3488 cherokee-3488
svn: Can't connect to host 'svn.cherokee-project.com': Connection timed out

I tried both on my server and locally...

Is there any other way to get that revision? a tarball somewhere?

Thank you, really!
Yo'av.

2009/8/13 Alvaro Lopez Ortega <alv...@alobbs.com>

Hello there,

On 13-ago-09, at 15:13, Miguel Angel wrote:

I don't have any idea, but may be that could be related with the php-cgi behaviour alex is suffering.

I'll investigate it as soon as I finish some really urgent stuff I must finish today.

Meanwhile, could you please check whether reverting the change-set 3488 helps?

--
Greetings, alo
http://www.alobbs.com/

--
Yo'av Moshe

[Alvaro Lopez Ortega]

On 13-ago-09, at 15:55, Yo'av Moshe wrote:

> ~/% svn co svn://svn.cherokee-project.com/cherokee/trunk@3488
> cherokee-3488
> svn: Can't connect to host 'svn.cherokee-project.com': Connection
> timed out
>
> I tried both on my server and locally...
> Is there any other way to get that revision? a tarball somewhere?

Are you DNS working?

$ svn co svn://svn.cherokee-project.com/cherokee/trunk@3488
cherokee-3488
A cherokee-3488/m4
....

--
Greetings, alo
http://www.alobbs.com/

_______________________________________________

[Yo'av Moshe]

No, DNS is working.

'ping svn.cherokee-project.com' works, and besides, every other thing on the internet works too. Maybe it's a firewall thing.
I can't seem to find a way to check that specific revision.

I tried to latest SVN checkout for the tarball and it didn't help.

Any other idea?

Yo'av

2009/8/13 Alvaro Lopez Ortega <alv...@alobbs.com>

On 13-ago-09, at 15:55, Yo'av Moshe wrote:

~/% svn co svn://svn.cherokee-project.com/cherokee/trunk@3488 cherokee-3488
svn: Can't connect to host 'svn.cherokee-project.com': Connection timed out

I tried both on my server and locally...
Is there any other way to get that revision? a tarball somewhere?

Are you DNS working?

$ svn co svn://svn.cherokee-project.com/cherokee/trunk@3488 cherokee-3488

A    cherokee-3488/m4
....

--
Greetings, alo
http://www.alobbs.com/

--
Yo'av Moshe

[Alvaro Lopez Ortega]

On 17-ago-09, at 22:54, Yo'av Moshe wrote:

> No, DNS is working.
>
> 'ping svn.cherokee-project.com' works, and besides, every other
> thing on the internet works too. Maybe it's a firewall thing.
> I can't seem to find a way to check that specific revision.
>
> I tried to latest SVN checkout for the tarball and it didn't help.
> Any other idea?

Yeah, then I suppose the (outgoing) TCP port 3690 is blocked in your
network. It could be either your local firewall, your router/firewall
and any of its uplinks.

There have been a number of people using the SVN during the last days,
so the problem must be at your side. I wish I could help more..

http://lists.octality.com/pipermail/cherokee-commits/2009-August/thread.html

--
Greetings, alo
http://www.alobbs.com/

_______________________________________________

[Yo'av Moshe]

Hey...
I managed to get revision 3488 from another computer, and still, it doesn't work. I get a Bad Gateway error of Cherokee 0.99.21.
I had to fix something in common-internal.h:88 and the avl.h thing to make it install, but that's all I changed.

I'll try to revert my config file so I can install 0.99.20 again - I have a showcase of the website tomorrow :(

I'm holding back Cherokee's updates for now, and will stay tuned for the changelog.

Yo'av

2009/8/18 Alvaro Lopez Ortega <alv...@alobbs.com>

On 17-ago-09, at 22:54, Yo'av Moshe wrote:

No, DNS is working.

'ping svn.cherokee-project.com' works, and besides, every other thing on the internet works too. Maybe it's a firewall thing.
I can't seem to find a way to check that specific revision.

I tried to latest SVN checkout for the tarball and it didn't help.
Any other idea?

Yeah, then I suppose the (outgoing) TCP port 3690 is blocked in your network. It could be either your local firewall, your router/firewall and any of its uplinks.

There have been a number of people using the SVN during the last days, so the problem must be at your side. I wish I could help more..

 http://lists.octality.com/pipermail/cherokee-commits/2009-August/thread.html

--
Greetings, alo
http://www.alobbs.com/

--
Yo'av Moshe

[Alvaro Lopez Ortega]

Hello Yo'av,

On 19-ago-09, at 15:27, Yo'av Moshe wrote:

> I managed to get revision 3488 from another computer, and still, it
> doesn't work. I get a Bad Gateway error of Cherokee 0.99.21.

We are currently at revision 3573 - almost a hundled changes ahead.
Could you please test the latest trunk?

http://www.cherokee-project.com/download/trunk/

--
Greetings, alo
http://www.alobbs.com/

_______________________________________________

[Yo'av Moshe]

Hey, I had no problem to get the latest trunk from the tarball, I didn't need to use SVN for that...
I thought you wanted me to try that specific version... :-s

I'll try it in an hour and will let you know.

Yo'av.

2009/8/19 Alvaro Lopez Ortega <alv...@alobbs.com>

Hello Yo'av,

On 19-ago-09, at 15:27, Yo'av Moshe wrote:

I managed to get revision 3488 from another computer, and still, it doesn't work. I get a Bad Gateway error of Cherokee 0.99.21.

We are currently at revision 3573 - almost a hundled changes ahead. Could you please test the latest trunk?

 http://www.cherokee-project.com/download/trunk/

--
Greetings, alo
http://www.alobbs.com/

--
Yo'av Moshe

[Yo'av Moshe]

Alvaro,
I got the latest SVN snapshot (3574) from the tarball, did ./configure
&& make && su && make install, and got the same "502 Bad Gateway"
error, just versioned as Cherokee 0.99.23 this time...

Yo'av

2009/8/19, Yo'av Moshe <bje...@gmail.com>:

--
Yo'av Moshe

[Yo'av Moshe]

Last update from me:
I installed Cherokee 0.99.20 again, run 'cat cherokee.conf.new | grep -v collector > cherokee.conf' and my website is working again. No new status page, but still... it is working with Cherokee 0.99.20.

Yo'av

2009/8/19 Yo'av Moshe <bje...@gmail.com>

--
Yo'av Moshe

[Antonio Pérez]

Hi!

On Wed, Aug 19, 2009 at 4:13 PM, Yo'av Moshe<bje...@gmail.com> wrote:

> Hey, I had no problem to get the latest trunk from the tarball, I didn't
> need to use SVN for that...
> I thought you wanted me to try that specific version... :-s

I think Álvaro says you that try trunk REVERTING changeset 3488 (3488
and 3489 actually). Here's how you can do it:

$ svn co svn://svn.cherokee-project.com/cherokee/trunk cherokee
$ cd cherokee/
$ svn merge -r 3489:3487 .

Or downloading the latest trunk from the tarball and applying the
attached patch.

Please, it'd be useful if you could give it a try.

>
> I'll try it in an hour and will let you know.
>
> Yo'av.
>
> 2009/8/19 Alvaro Lopez Ortega <alv...@alobbs.com>
>>
>> Hello Yo'av,
>>
>> On 19-ago-09, at 15:27, Yo'av Moshe wrote:
>>
>>> I managed to get revision 3488 from another computer, and still, it
>>> doesn't work. I get a Bad Gateway error of Cherokee 0.99.21.
>>
>>
>> We are currently at revision 3573 - almost a hundled changes ahead. Could
>> you please test the latest trunk?
>>
>>  http://www.cherokee-project.com/download/trunk/
>>
>> --
>> Greetings, alo
>> http://www.alobbs.com/
>>
>
>
> --
> Yo'av Moshe
>

> _______________________________________________
> Cherokee mailing list
> Cher...@lists.octality.com
> http://lists.octality.com/listinfo/cherokee
>
>

--
Saludos:
Antonio Pérez

[Yo'av Moshe]

Hey,
Sorry it took me a lot of time, my website was under a lot of pressure and I couldn't find time for the tests.

I applied the patch on Cherokee 0.99.24 and the problem still arises, just the same.

I'm not sure what to do. My website is working fine under Cherokee 0.99.20, but I don't want to use a software when I know that I can never update it. Besides, I'd like to use the new graphs :p

Any ideas about what else can I check?

Thank you!
Yo'av

2009/8/20 Antonio Pérez <ape...@skarcha.com>

--
Yo'av Moshe

[Yo'av Moshe]

Hey,
No other idea? I can't really keep using cherokee if i'm stuck with an
old version i can't upgrade.

This is obviously a cherokee problem, and it started at 0.99.21.

I would love to supply any additional information about my setup.

Please help!
Yo'av.

--
Sent from Gmail for mobile | mobile.google.com

[Alvaro Lopez Ortega]

On 14/10/2009, at 19:19, Yo'av Moshe wrote:

> I would love to supply any additional information about my setup.

Ok, let's try to refocus this whole issue.
Do you know how could I reproduce the issue locally? That'd be enough
for me to debug the problem.

--
Octality
http://www.octality.com/

[Yo'av Moshe]

Not really, since I'm can't understand why some pages work and other don't.

I can setup one of my backend servers to run on a public port - this way maybe you could try to reverse proxy it. I'll tell what pages break.

Is it okay?
I can also generate all sorts of logs if it will help, just say what to...

Yo'av

2009/10/14 Alvaro Lopez Ortega <alv...@octality.com>

On 14/10/2009, at 19:19, Yo'av Moshe wrote:

I would love to supply any additional information about my setup.

Ok, let's try to refocus this whole issue.
Do you know how could I reproduce the issue locally? That'd be enough for me to debug the problem.

--
Octality
http://www.octality.com/

--
Yo'av Moshe

[Yo'av Moshe]

I'm sorry, but I just have to get this resolved...

Just tried 0.99.31 and still the same.

So now I have compiled cherokee with trace support, and I'm attaching the relevant portion of the log.

Scenario is this:

1. Cherokee is up (port 80), backend server is up (port 8100).
2. I pointed my browser to some.url.org/login and logged in successfully (POST)
   
3. Moved to some.url.org/panel, and get the admin panel as I should (GET).
4. Clicked on the "Photos Panel", which is at /panel/photos/9, and got 502 error (GET).

Same scenario is working just fine when connecting directly to backend system.

Please, help!
If there's anything else I can do other from this log I'll be more then glad to create it, just tell me what log you need...

Let's get this issue fixed!

Thank you.
Yo'av

2009/10/19 Yo'av Moshe <bje...@gmail.com>

--
Yo'av Moshe

[Yo'av Moshe]

Okay, since there was no response I'm taking it a bit further.

This should be more then enough.

I opened up my source @ maagal.dontexist.org:40 . You can click the link and see that the website is up.

This is how to reproduce the bug:

• Create an information source for my server (remote host).
• Create a new virtual server with an "HTTP Reverse Proxy" handler for my server.
• Make sure you check "Preserve Host Header".
• Save and gracefully restart Cherokee.
• Point your browser to your virtual server and follow these steps:
1. Go to /admin/
2. It will ask you to log in, use "admin" and "1234".
3. Go to /admin/articles_manager/9.
4. You'll get 502 error from cherokee.
5. Try it on my server, and see that it's working.
• If you want, check the same with Cherokee 0.99.20 and see that it's working.

Please, Alvaro, take a look at it when you can. I really can't think of any other tools I can give you to diagnose this bug.

Thank you.

Yo'av

2009/12/9 Yo'av Moshe <bje...@gmail.com>

--
Yo'av Moshe

[Yo'av Moshe]

Sorry for the double posting, but I found out that the link - /admin/articles_manager/9, while working on my server and not when proxied by Cherokee, actually redirects to articles/new_embed/9 which does working when proxied by Cherokee.

Maybe Cherokee gets confused because of the redirect?

Yo'av

2009/12/11 Yo'av Moshe <bje...@gmail.com>

--
Yo'av Moshe