Errno::EPIPE in Passenger

[Evil Trout]

I am getting many errors in my apache 2 error log that look like this:

*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe)
(process 12308):
from /opt/ruby-enterprise-1.8.6-20090201/lib/ruby/gems/1.8/gems/
passenger-2.1.2/lib/phusion_passenger/rack/request_handler.rb:82:in
`write'
from /opt/ruby-enterprise-1.8.6-20090201/lib/ruby/gems/1.8/gems/
passenger-2.1.2/lib/phusion_passenger/rack/request_handler.rb:82:in
`process_request'
from /var/www/forumwarz/vendor/rails/actionpack/lib/action_controller/
response.rb:155:in `each_line'

I am using the latest versions of Rails, Phusion Passenger and Ruby
Enterprise Edition.

It is behind haproxy if that makes a difference, to spread load across
two servers.

There doesn't seem to be any errors on the front end. Any idea what's
causing all these errors?

[Evil Trout]

I should also mention I am using conservative spawning.

[Tim Underwood]

Did you figure out what the problem is?

I'm also seeing the same thing running HAProxy and it appears to
correspond with the HAProxy http check. If I change the check to a
static page (e.g. /ping.html), which bypasses the rails app, then the
errors go away so it doesn't look like real web traffic is causing
it. I think it started after I deployed Passenger 2.1.3 and Rails
2.3.2.

-Tim

[Greg]

Today I also noticed similar errors at my installation. I have

Passenger 2.1.3 and Rails 2.3.2.

After analysis I must say that logs look very strange:
- The failed "save" post request doesn't have any post parameters!
That's why Rails throws InvalidAuthenticityToken exception.
- The last entry from access.log is surrounded by entries which
happened at about 12:05:40. So below errors from 3 log files seem to
related.
- A user to post the "save" action, have to get "edit" page first. But
there is no http get to "edit" page before that failed "save" action.
So how the user could post (click) the save? BTW hopefully the user
didn't reported any problems with application.
- There was successful "save" action at 12:00:51 and that's all
requests / actions related to the record # 2130207535 which was
edited.

So that failed "save" action looks like a phantom of earlier
successful save ;-)

Does this have something common with IE6, SSL and ssl-unclean-shutdown
option I had to enable for the IE6 & 7 to make them work with SSL?

Greg

This is Rails log:
Processing XyzController#save (for 11.11.11.11 at 2009-04-09 12:05:40)
[POST]
Parameters: {"action"=>"save", "id"=>"2130207535",
"controller"=>"xyz"}

ActionController::InvalidAuthenticityToken
(ActionController::InvalidAuthenticityToken):
/usr/local/src/passenger-2.1.3/lib/phusion_passenger/rack/
request_handler.rb:65:in `process_request'
/usr/local/src/passenger-2.1.3/lib/phusion_passenger/
abstract_request_handler.rb:197:in `main_loop'
...

This is Apache error.log:
[ pid=29676 file=Hooks.cpp:529 time=2009-04-09 12:05:40.765 ]:
Unexpected error in mod_passenger: An error occurred while receiving
HTTP upload data.
Backtrace:
in 'int Hooks::handleRequest(request_rec*)' (Hooks.cpp:458)

*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe)

(process 30398):
from /usr/local/src/passenger-2.1.3/lib/phusion_passenger/rack/
request_handler.rb:67:in `write'
from /usr/local/src/passenger-2.1.3/lib/phusion_passenger/rack/
request_handler.rb:67:in `process_request'
from /usr/local/src/passenger-2.1.3/lib/phusion_passenger/
abstract_request_handler.rb:197:in `main_loop'
...

And this is Apache access.log:
11.11.11.11 - - [09/Apr/2009:12:00:22 +0200] "GET /xyz/edit/2130207535
HTTP/1.1" 200 1723 "https://www.example.com/xyz/month" "Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET
CLR 2.0.50727)"
11.11.11.11 - - [09/Apr/2009:12:00:51 +0200] "POST /xyz/save/
2130207535 HTTP/1.1" 200 90 "https://www.example.com/xyz/edit/
2130207535" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;
SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
...
11.11.11.11 - - [09/Apr/2009:12:00:40 +0200] "POST /xyz/save/
2130207535 HTTP/1.1" 500 467 "https://www.example.com/xyz/edit/
2130207535" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;
SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
...

[mikehale]

I'm seeing the same problem in passenger 2.2.2 using MRI running a
merb slice. Here are my logs: http://gist.github.com/107165. Any
suggestions would be awesome.

On Mar 27, 11:55 am, Evil Trout <robin.w...@gmail.com> wrote:
> I am getting many errors in my apache 2 error log that look like this:
>

> *** Exception Errno::EPIPEin Passenger RequestHandler (Broken pipe)

[Chris]

I'm seeing similar behavior on passenger 2.2.0 with Apache. Like
another person pointed out, it only seems to happen for us with IE6 or
IE7 on POSTs. I read online there can be some issues with IE and
Apache, so I tried adding the following for our https vhost:
BrowserMatch ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0
force-response-1.0

This didn't make any difference. I still saw the error occurring.

Also if you check the rails production log, the request simply shows
as a normal 200 response, yet the apache access log shows a 500 error.
It also doesn't seem to make a difference what rails action it is. It
seems to occur on any action that can be POSTed to. It also doesn't
happen very often for us, maybe 1 out of 1000 POST requests? It also
doesn't seem to be very repeatable, since it appears the client
retries their POST and it always goes through the second time.

Also, I'm pretty sure it happens on http in addition to https, but not
100% sure since I only recently turned on debug level logs. But I had
very similar 500 errors in my access logs for http with the same
symptoms (IE browser, POST, rails log showed 200 response) so my guess
is that it affects both.

Anyways, here's our trace:
[Mon May 04 06:43:45 2009] [info] [client 143.111.22.26] (32)Broken
pipe: core_output_filter: writing data to the network

*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe)

(process 5691):
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
rack/request_handler.rb:76:in `write'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
rack/request_handler.rb:76:in `process_request'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
abstract_request_handler.rb:203:in `main_loop'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
railz/application_spawner.rb:340:in `start_request_handler'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
railz/application_spawner.rb:298:in `handle_spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
utils.rb:181:in `safe_fork'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
railz/application_spawner.rb:296:in `handle_spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
abstract_server.rb:337:in `__send__'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
abstract_server.rb:337:in `main_loop'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
abstract_server.rb:187:in `start_synchronously'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
abstract_server.rb:154:in `start'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
railz/application_spawner.rb:192:in `start'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
spawn_manager.rb:257:in `spawn_rails_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
abstract_server_collection.rb:126:in `lookup_or_add'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
spawn_manager.rb:251:in `spawn_rails_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
abstract_server_collection.rb:80:in `synchronize'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
abstract_server_collection.rb:79:in `synchronize'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
spawn_manager.rb:250:in `spawn_rails_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
spawn_manager.rb:153:in `spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
spawn_manager.rb:282:in `handle_spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
abstract_server.rb:337:in `__send__'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
abstract_server.rb:337:in `main_loop'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/lib/phusion_passenger/
abstract_server.rb:187:in `start_synchronously'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.0/bin/passenger-spawn-
server:55

-Chris

[Chris Bailey]

Anyone figured this out? We recently switched to Nginx + Passenger
2.2.2, with Rails 2.3.2. I am seeing this a fair bit in our logs.
Here's our trace:

[ pid=31486 file=ext/nginx/HelperServer.cpp:460 time=2009-05-18
09:51:37.809 ]:
Uncaught exception in PassengerServer client thread:
exception: write() failed: Broken pipe (32)
backtrace:
in 'void Client::forwardResponse
(boost::shared_ptr<Passenger::Application::Session>&,
FileDescriptor&)' (HelperServer.cpp:344)
in 'bool Client::handleRequest
(FileDescriptor&)' (HelperServer.cpp:438)
in 'void Client::threadMain()' (HelperServer.cpp:484)

*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe)

(process 5546):
from /usr/lib/passenger/lib/phusion_passenger/rack/
request_handler.rb:98:in `write'
from /usr/lib/passenger/lib/phusion_passenger/rack/
request_handler.rb:98:in `process_request'
from /usr/lib64/ruby/gems/1.8/gems/actionpack-2.3.2/lib/
action_controller/response.rb:155:in `each_line'
from /usr/lib64/ruby/gems/1.8/gems/actionpack-2.3.2/lib/
action_controller/response.rb:155:in `each'
from /usr/lib/passenger/lib/phusion_passenger/rack/
request_handler.rb:97:in `process_request'
from /usr/lib/passenger/lib/phusion_passenger/
abstract_request_handler.rb:203:in `main_loop'
from /usr/lib/passenger/lib/phusion_passenger/railz/
application_spawner.rb:340:in `start_request_handler'
from /usr/lib/passenger/lib/phusion_passenger/railz/
application_spawner.rb:298:in `handle_spawn_application'
from /usr/lib/passenger/lib/phusion_passenger/utils.rb:181:in
`safe_fork'
from /usr/lib/passenger/lib/phusion_passenger/railz/
application_spawner.rb:296:in `handle_spawn_application'
from /usr/lib/passenger/lib/phusion_passenger/
abstract_server.rb:337:in `__send__'
from /usr/lib/passenger/lib/phusion_passenger/
abstract_server.rb:337:in `main_loop'
from /usr/lib/passenger/lib/phusion_passenger/
abstract_server.rb:187:in `start_synchronously'
from /usr/lib/passenger/lib/phusion_passenger/
abstract_server.rb:154:in `start'
from /usr/lib/passenger/lib/phusion_passenger/railz/
application_spawner.rb:192:in `start'
from /usr/lib/passenger/lib/phusion_passenger/spawn_manager.rb:
257:in `spawn_rails_application'
from /usr/lib/passenger/lib/phusion_passenger/
abstract_server_collection.rb:126:in `lookup_or_add'
from /usr/lib/passenger/lib/phusion_passenger/spawn_manager.rb:
251:in `spawn_rails_application'
from /usr/lib/passenger/lib/phusion_passenger/
abstract_server_collection.rb:80:in `synchronize'
from /usr/lib/passenger/lib/phusion_passenger/
abstract_server_collection.rb:79:in `synchronize'
from /usr/lib/passenger/lib/phusion_passenger/spawn_manager.rb:
250:in `spawn_rails_application'
from /usr/lib/passenger/lib/phusion_passenger/spawn_manager.rb:
153:in `spawn_application'
from /usr/lib/passenger/lib/phusion_passenger/spawn_manager.rb:
282:in `handle_spawn_application'
from /usr/lib/passenger/lib/phusion_passenger/
abstract_server.rb:337:in `__send__'
from /usr/lib/passenger/lib/phusion_passenger/
abstract_server.rb:337:in `main_loop'
from /usr/lib/passenger/lib/phusion_passenger/
abstract_server.rb:187:in `start_synchronously'
from /usr/lib/passenger/bin/passenger-spawn-server:61

[rrn]

I get this as well. It seem the error seems to happen in chunks. All
of the sudden 10-20 of these exceptions get written to the logs. After
this, I do not see the error for some time, and then another big chunk
of EPIPE exceptions appears.

Still no solution?

Here's my trace:

*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe)

(process 2609):
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/rack/request_handler.rb:83:in `write'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/rack/request_handler.rb:83:in `process_request'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/abstract_request_handler.rb:203:in `main_loop'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/railz/application_spawner.rb:340:in
`start_request_handler'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/railz/application_spawner.rb:298:in
`handle_spawn_application'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/utils.rb:181:in `safe_fork'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/railz/application_spawner.rb:296:in
`handle_spawn_application'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/abstract_server.rb:337:in `__send__'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/abstract_server.rb:337:in `main_loop'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/abstract_server.rb:187:in `start_synchronously'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/abstract_server.rb:154:in `start'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/railz/application_spawner.rb:192:in `start'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/spawn_manager.rb:257:in `spawn_rails_application'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/abstract_server_collection.rb:126:in `lookup_or_add'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/spawn_manager.rb:251:in `spawn_rails_application'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/abstract_server_collection.rb:80:in `synchronize'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/spawn_manager.rb:250:in `spawn_rails_application'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/spawn_manager.rb:153:in `spawn_application'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/spawn_manager.rb:282:in `handle_spawn_application'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/abstract_server.rb:337:in `__send__'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/abstract_server.rb:337:in `main_loop'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/
phusion_passenger/abstract_server.rb:187:in `start_synchronously'
from /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.2/bin/passenger-
spawn-server:61

I'm on Freebsd 6.4 (amd64), apache2.2.11, passenger 2.2.2, rails
2.3.2. Thanks in advance.

[Hongli Lai]

On Mon, May 25, 2009 at 8:30 AM, rrn <rasmusr...@gmail.com> wrote:
>
> I get this as well. It seem the error seems to happen in chunks. All
> of the sudden 10-20 of these exceptions get written to the logs. After
> this, I do not see the error for some time, and then another big chunk
> of EPIPE exceptions appears.
>
> Still no solution?

Can you try the 'fix_broken_pipes' branch from Git?

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

[Andrew Lippert]

Hongli,

Any other potentially unstable changes to the fix_broken_pipes branch?
I have a system in production exhibiting the broken pipe issue. I'm
willing to deploy the branch build to see if things stabilize if the
branch is safe for deployment. Thoughts?

Thanks,

Andrew Lippert
CTO - Biznik


On May 25, 12:06 am, Hongli Lai <hon...@phusion.nl> wrote:

> On Mon, May 25, 2009 at 8:30 AM, rrn <rasmusrniel...@gmail.com> wrote:
>
> > I get this as well. It seem the error seems to happen in chunks. All
> > of the sudden 10-20 of these exceptions get written to the logs. After
> > this, I do not see the error for some time, and then another big chunk
> > of EPIPE exceptions appears.
>
> > Still no solution?
>
> Can you try the 'fix_broken_pipes' branch from Git?
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Hongli Lai]

On Wed, May 27, 2009 at 1:18 AM, Andrew Lippert <acli...@gmail.com> wrote:
>
> Hongli,
>
> Any other potentially unstable changes to the fix_broken_pipes branch?
> I have a system in production exhibiting the broken pipe issue. I'm
> willing to deploy the branch build to see if things stabilize if the
> branch is safe for deployment. Thoughts?

Hi Andrew.

It should be safe for deployment in production systems. All automated
tests pass and we've tested it with a number of people.

Regards,
Hongli Li

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Andrew Lippert]

Thanks!

A

[Tom Copeland]

On May 26, 2009, at 8:11 PM, Hongli Lai wrote:

>
> On Wed, May 27, 2009 at 1:18 AM, Andrew Lippert
> <acli...@gmail.com> wrote:
>>
>> Hongli,
>>
>> Any other potentially unstable changes to the fix_broken_pipes
>> branch?
>> I have a system in production exhibiting the broken pipe issue. I'm
>> willing to deploy the branch build to see if things stabilize if the
>> branch is safe for deployment. Thoughts?
>
> Hi Andrew.
>
> It should be safe for deployment in production systems. All automated
> tests pass and we've tested it with a number of people.

FWIW, I'm using this branch with REE 20090520 and Rails 2.3.2 and it
appears to have fixed the EPIPE problem which I was seeing with
Passenger 2.2.2. Thanks guys!

Yours,

tom

[findchris]

We are seeing similar errors (Errno::ECONNRESET in Passenger
RequestHandler and Errno::EPIPE in Passenger RequestHandler).

I don't see the fix_broken_pipes branch, but I do see this one (is it
the same??):
http://github.com/FooBarWidget/passenger/tree/request_handler_with_pipes

Thanks.

On May 25, 12:06 am, Hongli Lai <hon...@phusion.nl> wrote:

> On Mon, May 25, 2009 at 8:30 AM, rrn <rasmusrniel...@gmail.com> wrote:
>
> > I get this as well. It seem the error seems to happen in chunks. All
> > of the sudden 10-20 of these exceptions get written to the logs. After
> > this, I do not see the error for some time, and then another big chunk
> > of EPIPE exceptions appears.
>
> > Still no solution?
>
> Can you try the 'fix_broken_pipes' branch from Git?
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Hongli Lai]

On Tue, Jun 9, 2009 at 8:08 PM, findchris<find...@gmail.com> wrote:
>
> We are seeing similar errors (Errno::ECONNRESET in Passenger
> RequestHandler and Errno::EPIPE in Passenger RequestHandler).
>
> I don't see the fix_broken_pipes branch

It was been merged into master last week.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[millisami]

Hi Guys,
Upgraded to Rails 2.3.2 and Passenger 2.2.4 on Ubuntu hardy slice at
slicehost with Apache2

I'm getting this same above discussed error in my Apache error.log of
system /var/logs/apache2/
<pre>
[ pid=4249 file=ext/apache2/Hooks.cpp:638 time=2009-07-04
11:47:32.752 ]:
No data received from the backend application (process 4383) within
45000 msec. Either the backend application is frozen, or your TimeOut
value of 45 seconds is too low. Please check whether your application
is frozen, or increase the value of the TimeOut configuration
directive.

*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe)

(process 4391):
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/rack/request_handler.rb:93:in `write'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/rack/request_handler.rb:93:in `process_request'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_request_handler.rb:206:in `main_loop'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/railz/application_spawner.rb:376:in
`start_request_handler'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/railz/application_spawner.rb:334:in
`handle_spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/utils.rb:182:in `safe_fork'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/railz/application_spawner.rb:332:in
`handle_spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:351:in `__send__'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:351:in `main_loop'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:195:in `start_synchronously'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:162:in `start'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/railz/application_spawner.rb:213:in `start'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/spawn_manager.rb:261:in `spawn_rails_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server_collection.rb:126:in `lookup_or_add'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/spawn_manager.rb:255:in `spawn_rails_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server_collection.rb:80:in `synchronize'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/spawn_manager.rb:254:in `spawn_rails_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/spawn_manager.rb:153:in `spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/spawn_manager.rb:286:in `handle_spawn_application'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:351:in `__send__'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:351:in `main_loop'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:195:in `start_synchronously'
from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/bin/passenger-spawn-
server:61

*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe)

(process 4383):
</pre>
and these too.
<pre>
pid=4362 file=ext/apache2/Hooks.cpp:638 time=2009-07-04
11:55:19.251 ]:
No data received from the backend application (process 4383) within
45000 msec. Either the backend application is frozen, or your TimeOut
value of 45 seconds is too low. Please check whether your application
is frozen, or increase the value of the TimeOut configuration
directive.
[ pid=4298 file=ext/apache2/Hooks.cpp:638 time=2009-07-04
11:55:19.255 ]:
No data received from the backend application (process 4252) within
45000 msec. Either the backend application is frozen, or your TimeOut
value of 45 seconds is too low. Please check whether your application
is frozen, or increase the value of the TimeOut configuration
directive.
[Sat Jul 04 11:55:19 2009] [error] [client 86.96.226.13] Premature end
of script headers: 41, referer: http://domain.com/
[ pid=4373 file=ext/apache2/Hooks.cpp:638 time=2009-07-04
11:55:19.559 ]:
</pre>

Its getting me mad and on the browser, sometimes its show and when
refreshed, Application Error 500 shows up in frequent basis.
any directions??


On Jun 10, 2:50 pm, Hongli Lai <hon...@phusion.nl> wrote:

> On Tue, Jun 9, 2009 at 8:08 PM, findchris<findch...@gmail.com> wrote:
>
> > We are seeing similar errors (Errno::ECONNRESET in Passenger
> > RequestHandler and Errno::EPIPE in Passenger RequestHandler).
>
> > I don't see the fix_broken_pipes branch
>
> It was been merged into master last week.
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Jonn Beames]

I am also seeing "Errno::EPIPE in Passenger" with:

Passenger 2.2.4
REE 1.8.6-20090610
Rails 2.3.2
Apache 2.2.11

The errors show up frequently in the apache error_log (as opposed to
the apache-error_log files defined in the VirtualHosts) and look like
this:

*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe)

(process 17635):
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/rack/request_handler.rb:
108:in `write'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/rack/request_handler.rb:
108:in `process_request'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/actionpack-2.3.2/lib/action_controller/response.rb:155:in
`each_line'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/actionpack-2.3.2/lib/action_controller/response.rb:155:in
`each'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/rack/request_handler.rb:
107:in `process_request'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/
abstract_request_handler.rb:206:in `main_loop'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/railz/
application_spawner.rb:376:in `start_request_h
andler'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/railz/
application_spawner.rb:334:in `handle_spawn_ap
plication'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/utils.rb:182:in
`safe_fork'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/railz/
application_spawner.rb:332:in `handle_spawn_ap
plication'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:
351:in `__send__'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:
351:in `main_loop'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:
195:in `start_synchronously'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:
162:in `start'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/railz/
application_spawner.rb:213:in `start'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:261:in
`spawn_rails_application'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/
abstract_server_collection.rb:126:in `lookup_or_add'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:255:in
`spawn_rails_application'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/
abstract_server_collection.rb:80:in `synchronize'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/
abstract_server_collection.rb:79:in `synchronize'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:254:in
`spawn_rails_application'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:153:in
`spawn_application'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/spawn_manager.rb:286:in
`handle_spawn_application'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:
351:in `__send__'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:
351:in `main_loop'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/lib/phusion_passenger/abstract_server.rb:
195:in `start_synchronously'
from /opt/local/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/
1.8/gems/passenger-2.2.4/bin/passenger-spawn-server:61

This entry is often, but not always, proceeded or proceeds an entry
that looks like this:

[ pid=8401 file=ext/apache2/Hooks.cpp:621 time=2009-07-15
12:34:57.287 ]:
Apache stopped forwarding the backend's response, even though the
HTTP client did not close the connection. Is this an Apache bug?

And it appears that the latter error always corresponds to a file
being sent to the client by mod_xsendfile after being located within a
rails controller (using the x_sendfile option of the send_file
method).

From what I have seen so far, there is no adverse behavior experienced
by the client corresponding to these error log entries. The files are
always sent as expected, no rails related UX problems have been
reported, and since switching from Apache 2.0.52 to Apache 2.2.11 the
expected http status codes sent by mod_xsendfile are being received by
the clients.

The only real problem noticed so far is really big error_log files
with a low signal to noise ratio.

- Jonn

On Jul 4, 4:58 am, millisami <millis...@gmail.com> wrote:
> Hi Guys,
> Upgraded to Rails 2.3.2 and Passenger 2.2.4 on Ubuntu hardy slice at
> slicehost with Apache2
>
> I'm getting this same above discussed error in my Apache error.log of
> system /var/logs/apache2/

> ...

[millisami]

Is this thread abandoned??
Or is this work in progress or resolved???
Any directions????

[Hongli Lai]

On Thu, Jul 16, 2009 at 10:51 PM, millisami<mill...@gmail.com> wrote:
>
> Is this thread abandoned??
> Or is this work in progress or resolved???
> Any directions????

It's not abandoned, but I don't know what's causing your problems,
which is why I didn't respond.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Pooya Woodcock]

I think I am seeing the same error running passenger 2.2.4, apache
2.2.11, and using x_sendfile. I downgraded to our last working version
of 2.0.3 and the error no longer pops up. Here's a snapshot of what I
was seeing with PassengerLogLevel 2

[ pid=7090 file=ext/apache2/Hooks.cpp:621 time=2009-07-20
13:19:37.315 ]:

Apache stopped forwarding the backend's response, even though the
HTTP client did not close the connection. Is this an Apache bug?

[ pid=7327 file=ext/apache2/Hooks.cpp:621 time=2009-07-20
13:19:37.324 ]:

Apache stopped forwarding the backend's response, even though the
HTTP client did not close the connection. Is this an Apache bug?

[ pid=7091 file=ext/apache2/Hooks.cpp:621 time=2009-07-20
13:19:37.367 ]:

Apache stopped forwarding the backend's response, even though the
HTTP client did not close the connection. Is this an Apache bug?

[ pid=7087 file=ext/apache2/Hooks.cpp:621 time=2009-07-20
13:19:37.493 ]:

Thanks,
Pooya

[Benjamin Curtis]

I'm seeing the same error, also with Passenger 2.2.4, using
x_sendfile.

[Angelim]

I'm having the same problem running passenger 2.2.4 with Nginx 0.6.37
and using X-accell redirect.
I'm also getting Exception Errno::EPIPE in Passenger RequestHandler
(Broken pipe) (process 18445): lots of time.
I'd appreciate any tips.

Thanks,

Angelim

[Katherine]

I get a lot of this on my error logs:
[ pid=22328 file=ext/nginx/HelperServer.cpp:460 time=2009-07-27
01:21:00.781 ]:

Uncaught exception in PassengerServer client thread:
exception: write() failed: Broken pipe (32)
backtrace:
in 'void Client::forwardResponse
(boost::shared_ptr<Passenger::Application::Session>&,
FileDescriptor&)' (HelperServer.cpp:344)
in 'bool Client::handleRequest
(FileDescriptor&)' (HelperServer.cpp:438)
in 'void Client::threadMain()' (HelperServer.cpp:484)

I'm using nginx.

[Katherine]

More information: reviewing nginx error log, I only got this error for
Rails 2.3.3 so this could be a Rails-specific bug. I am now using
Rails 2.3.4

I will be using to Rails 2.3.2 for now.
Hoping this would solve the issue.

[Katherine]

No it did not solve the issue.
I was using REE and Passenger 2.2.4

[Claudio Poli]

passenger 2.2.5, using rack/sinatra, snow leopard server default ruby
1.8.7/apache

on send file for download, happens often:

*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe)
(process 8264):
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/rack/request_handler.rb:112:in `write'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/rack/request_handler.rb:112:in `process_request'
from /Library/Ruby/Gems/1.8/gems/sinatra-0.9.4/lib/sinatra/base.rb:
164:in `each'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/rack/request_handler.rb:111:in `process_request'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/abstract_request_handler.rb:207:in `main_loop'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/rack/application_spawner.rb:118:in `run'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/rack/application_spawner.rb:69:in
`spawn_application'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/utils.rb:183:in `safe_fork'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/rack/application_spawner.rb:62:in
`spawn_application'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/rack/application_spawner.rb:45:in
`spawn_application'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/spawn_manager.rb:159:in `spawn_application'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/spawn_manager.rb:287:in `handle_spawn_application'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/abstract_server.rb:352:in `__send__'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/abstract_server.rb:352:in `main_loop'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/lib/
phusion_passenger/abstract_server.rb:196:in `start_synchronously'
from /Library/Ruby/Gems/1.8/gems/passenger-2.2.5/bin/passenger-spawn-
server:61