Getting a bad gateway error (502) using nginx

[jeromegn]

Hey,

I've been getting the 502 error a lot on certain pages of my rails
application running on Ubuntu 8.10 + Passenger 2.2.4 + nginx 0.7.61 +
Rails 2.3.2

It happens when I want to upload a photo... but also when I simply
want to access a random page with a GET request. So I'm not sure how
they could be related.

Could it have anything to do with permissions? On that tmp directory?

My Nginx runs as www-data and my rails application is all chowned www-
data. I'm also running MySQL (I know people had that error with
sqlite) from the socket in /var/run/mysqld/mysql.sock

The site works flawlessly on the very similar setup of my dev
machine... I'm using the same nginx, passenger, rails versions, but
with sqlite.

Here's the error:

Error during failsafe response: closed stream
2009/07/02 21:16:29 [error] 12604#0: *29 upstream prematurely closed
connection while reading response header from upstream, client:
206.248.138.74, server: webby2059, request: "POST /admin/products/
produit-de-test HTTP/1.1", upstream: "passenger://unix:/tmp/passenger.
$
*** Exception NoMethodError in application (undefined method `each'
for nil:NilClass) (process 12797):
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/rack/request_handler.rb:95:in
`process_request'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/abstract_request_handler.rb:
206:in `main_loop'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/railz/application_spawner.rb:
376:in `start_request_handler'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/railz/application_spawner.rb:
334:in `handle_spawn_application'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/utils.rb:182:in `safe_fork'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/railz/application_spawner.rb:
332:in `handle_spawn_application'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:351:in
`__send__'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:351:in
`main_loop'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:195:in
`start_synchronously'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:162:in
`start'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/railz/application_spawner.rb:
213:in `start'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:261:in
`spawn_rails_application'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/abstract_server_collection.rb:
126:in `lookup_or_add'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:255:in
`spawn_rails_application'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/abstract_server_collection.rb:
80:in `synchronize'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/abstract_server_collection.rb:
79:in `synchronize'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:254:in
`spawn_rails_application'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:153:in
`spawn_application'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:286:in
`handle_spawn_application'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:351:in
`__send__'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:351:in
`main_loop'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:195:in
`start_synchronously'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/
passenger-2.2.4/bin/passenger-spawn-server:61

Any clue?

[Hongli Lai]

On Fri, Jul 3, 2009 at 4:05 PM, jeromegn<jero...@gmail.com> wrote:
>
> Hey,
>
> I've been getting the 502 error a lot on certain pages of my rails
> application running on Ubuntu 8.10 + Passenger 2.2.4 + nginx 0.7.61 +
> Rails 2.3.2
>
> It happens when I want to upload a photo... but also when I simply
> want to access a random page with a GET request. So I'm not sure how
> they could be related.
>
> Could it have anything to do with permissions? On that tmp directory?
>
> My Nginx runs as www-data and my rails application is all chowned www-
> data. I'm also running MySQL (I know people had that error with
> sqlite) from the socket in /var/run/mysqld/mysql.sock
>
> The site works flawlessly on the very similar setup of my dev
> machine... I'm using the same nginx, passenger, rails versions, but
> with sqlite.
>
> Here's the error:
>
> Error during failsafe response: closed stream
> 2009/07/02 21:16:29 [error] 12604#0: *29 upstream prematurely closed
> connection while reading response header from upstream, client:
> 206.248.138.74, server: webby2059, request: "POST /admin/products/
> produit-de-test HTTP/1.1", upstream: "passenger://unix:/tmp/passenger.
> $
> *** Exception NoMethodError in application (undefined method `each'
> for nil:NilClass) (process 12797):

This means that Rails, for some reason, did not return a proper Rack
response, causing Phusion Passenger's request handler to crash.

Are there any errors in the Rails error log?

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

[Jérôme Gravel-Niquet]

Thanks for the quick reply!

No, it seems like it can't even write to the log file: Rails Error:
Unable to access log file. Please ensure that /var/rails/ekovia/log/
production.log exists and is chmod 0666. The log level has been raised
to WARN and the output directed to STDERR until the problem is fixed.

Even though there is a log file right there, chowned www-data, chmod
666...

Interesting... how can that happen when it works perfectly with the
same cloned repository on my dev machine?

Thanks a lot.

On 3 juil, 10:09, Hongli Lai <hon...@phusion.nl> wrote:

> E-mail: i...@phusion.nl

[Jérôme Gravel-Niquet]

Ok, I fixed the log problem, but it remains blank and I still am
getting 502 error :(

Jerome.

[Hongli Lai]

On Fri, Jul 3, 2009 at 5:09 PM, Jérôme Gravel-Niquet<jero...@gmail.com> wrote:
>
> Ok, I fixed the log problem, but it remains blank and I still am
> getting 502 error :(

I believe it's this bug which I just confirmed:
http://code.google.com/p/phusion-passenger/issues/detail?id=332

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Jérôme Gravel-Niquet]

Might be it, seems like the production.log file not being accessible
is not fixed afterall... Not sure why at all honestly, here's what ls -
l gives me in log/:
drw-rw-rw- 1 www-data www-data 0 Jul 3 15:08 production.log

Nginx runs as www-data and drw-rw-rw- means chmod 0666...

I'm still getting the unwritable log file error and some other weird
one, as if I pressed stop while executing the request... but I didn't.

Rails Error: Unable to access log file. Please ensure that /var/rails/

mysite/log/production.log exists and is chmod 0666. The log level has

been raised to WARN and the output directed to STDERR until the
problem is fixed.

NOTICE: Mail not enabled
[ pid=3813 file=ext/nginx/HelperServer.cpp:478 time=2009-07-03
19:20:52.372 ]:
Couldn't forward the HTTP response back to the HTTP client: It seems
the user clicked on the 'Stop' button in his browser.

Error during failsafe response: closed stream

Dang, not out of the woods yet!

On 3 juil, 14:24, Hongli Lai <hon...@phusion.nl> wrote:

> On Fri, Jul 3, 2009 at 5:09 PM, Jérôme Gravel-Niquet<jerom...@gmail.com> wrote:
>
> > Ok, I fixed the log problem, but it remains blank and I still am
> > getting 502 error :(
>
> I believe it's this bug which I just confirmed:http://code.google.com/p/phusion-passenger/issues/detail?id=332
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Scott Smith]

Jérôme Gravel-Niquet wrote:
> Might be it, seems like the production.log file not being accessible
> is not fixed afterall... Not sure why at all honestly, here's what ls -
> l gives me in log/:
> drw-rw-rw- 1 www-data www-data 0 Jul 3 15:08 production.log
>
> Nginx runs as www-data and drw-rw-rw- means chmod 0666...
>

Your log "file" is actually a directory with no execution bit set.

-scott

[Jérôme Gravel-Niquet]

Maybe, I had created it using nano production.log and saved it empty.
Now I tried it with a touch production.log, applied the chmod and the
chown, no cigar.

I'm at a loss here, not getting the Rails error anymore, but still the
502 bad gateway...

Error during failsafe response: closed stream

2009/07/04 00:33:13 [error] 5319#0: *5 upstream prematurely closed

connection while reading response header from upstream, client:

206.248.138.74, server: marmotte, request: "POST /admin/images/2 HTTP/
1.1", upstream: "passenger://unix:/tmp/passenger.5270/master/helper_$

Thanks for everything.

[Scott Smith]

Jérôme Gravel-Niquet wrote:
> Maybe, I had created it using nano production.log and saved it empty.
> Now I tried it with a touch production.log, applied the chmod and the
> chown, no cigar.
>
> I'm at a loss here, not getting the Rails error anymore, but still the
> 502 bad gateway...

Check the permissions of every parent directory, up to /

-scott

[Jérôme Gravel-Niquet]

Ok, well I posted in the Webbynode forums, since it was probably more
related to them.

The staff there found the solution, it seems like I had to specify
passenger_default_user www-data; in my nginx configuration file in the
http {} section!

Thanks for the help everyone, hopefully this will make it easier for
others. But would you consider making this the default? I suspect it's
the most common choice of user to run a web server. Or should it
default to the nginx-equivalent setting? I let you decide.

Jerome.

[Scott Smith]

Jérôme Gravel-Niquet wrote:
> Thanks for the help everyone, hopefully this will make it easier for
> others. But would you consider making this the default? I suspect it's
> the most common choice of user to run a web server. Or should it
> default to the nginx-equivalent setting? I let you decide.
>

http://www.modrails.com/documentation/Users%20guide%20Nginx.html#PassengerDefaultUser

-scott

[Hongli Lai]

On Fri, Jul 3, 2009 at 9:23 PM, Jérôme Gravel-Niquet<jero...@gmail.com> wrote:
>
> Might be it, seems like the production.log file not being accessible
> is not fixed afterall... Not sure why at all honestly, here's what ls -
> l gives me in log/:
> drw-rw-rw- 1 www-data www-data 0 Jul  3 15:08 production.log
>
> Nginx runs as www-data and drw-rw-rw- means chmod 0666...
>
> I'm still getting the unwritable log file error and some other weird
> one, as if I pressed stop while executing the request... but I didn't.
>
> Rails Error: Unable to access log file. Please ensure that /var/rails/
> mysite/log/production.log exists and is chmod 0666. The log level has
> been raised to WARN and the output directed to STDERR until the
> problem is fixed.
> NOTICE: Mail not enabled
> [ pid=3813 file=ext/nginx/HelperServer.cpp:478 time=2009-07-03
> 19:20:52.372 ]:
>  Couldn't forward the HTTP response back to the HTTP client: It seems
> the user clicked on the 'Stop' button in his browser.
> Error during failsafe response: closed stream
>
> Dang, not out of the woods yet!
>
> On 3 juil, 14:24, Hongli Lai <hon...@phusion.nl> wrote:
>> On Fri, Jul 3, 2009 at 5:09 PM, Jérôme Gravel-Niquet<jerom...@gmail.com> wrote:
>>
>> > Ok, I fixed the log problem, but it remains blank and I still am
>> > getting 502 error :(
>>
>> I believe it's this bug which I just confirmed:http://code.google.com/p/phusion-passenger/issues/detail?id=332

I've fixed the logging bug today, in commit 72416a5. You might want to
give that a try.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl