process hangs - (Errno::ESPIPE) "Illegal seek"

[wbharding]

I am having the dreaded illegal seek / 100% CPU usage error, reported
by many, since moving to Passenger a couple weeks ago. The problem is
that even the ep patch (http://ep.blogware.com/blog/_archives/
2008/10/14/3930392.html) ain't fixing the problem for me. The patch
does successfully shut up the Exception Notifier emails I'd gotten
before applying the patch, but when I go check our processes, usually
at least once or twice per day I'll find one or more stuck eating 100%
CPU. Yesterday evening I checked and about 10 of our processes were
using 100% CPU. Surprised it didn't crash the site.

Invariably, when I check the Rails log, I'll find that that process
stuck at 100% CPU had experienced the illegal seek error earlier in
the day.

I'm now experimenting with some very hacky tricks to work around this
(output PID of process into a text file, use cron to kill -9 any
processes in that file), but it would sure be grand if this problem
got resolved (or if there is some new and great way to fix it that I'm
not aware of).

I'm on:

ruby 1.8.6 (2008-08-08 patchlevel 286) [x86_64-linux]
passenger 2.0.6

Thanks,
Bill

[Brian Johnson]

I am also seeing the problem even after the patch. The only difference
is now the process just gets stuck, but still uses 100% CPU.

[Hongli Lai]

I'm current debugging this problem. The debugger seems to tell me that
this is either a Ruby bug or a glibc bug, but either is very unlikely so
I might be wrong.

Which Rails version are you using? I'm investigating whether the Rails
version might have anything to do with it.

How often does this problem occur?

Regards,
Hongli Lai
--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

[Hongli Lai]

I *think* I've managed to implement a workaround for this problem in
commit 1ae59609e1375. Can anyone verify?

[Brian Johnson]

I'm using REE, not MRI

On Mar 11, 3:38 pm, Hongli Lai <hon...@phusion.nl> wrote:
> I *think* I've managed to implement a workaround for this problem in
> commit 1ae59609e1375. Can anyone verify?
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Hongli Lai]

Brian Johnson wrote:
> I'm using REE, not MRI

REE is based on MRI so this workaround applies to REE as well.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Brian Johnson]

Ok, I'll try it out. You listed MRI specifically in the patch so I
thought it was exclusive to standard MRI and not REE.

On Mar 11, 4:59 pm, Hongli Lai <hon...@phusion.nl> wrote:
> Brian Johnson wrote:
> > I'm using REE, not MRI
>
> REE is based on MRI so this workaround applies to REE as well.
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Brian Johnson]

Under 2.1.2 the problem is actually worse. The process still goes to
100% CPU usage, but now it hangs passenger completely. No vhosts
respond and passenger-status never returns. If I hit ctrl-c from
passenger status I get the following trace:
/usr/lib64/ruby/gems/1.8/gems/passenger-2.1.2/lib/phusion_passenger/
message_channel.rb:131:in `readpartial': Interrupt
from /usr/lib64/ruby/gems/1.8/gems/passenger-2.1.2/lib/
phusion_passenger/message_channel.rb:131:in `read_scalar'
from /usr/lib64/ruby/gems/1.8/gems/passenger-2.1.2/lib/
phusion_passenger/admin_tools/control_process.rb:75:in `reload'
from /usr/lib64/ruby/gems/1.8/gems/passenger-2.1.2/lib/
phusion_passenger/admin_tools/control_process.rb:73:in `open'
from /usr/lib64/ruby/gems/1.8/gems/passenger-2.1.2/lib/
phusion_passenger/admin_tools/control_process.rb:73:in `reload'
from /usr/lib64/ruby/gems/1.8/gems/passenger-2.1.2/lib/
phusion_passenger/admin_tools/control_process.rb:50:in `status'
from /usr/lib64/ruby/gems/1.8/gems/passenger-2.1.2/bin/passenger-
status:34:in `show_status'
from /usr/lib64/ruby/gems/1.8/gems/passenger-2.1.2/bin/passenger-
status:53:in `start'
from /usr/lib64/ruby/gems/1.8/gems/passenger-2.1.2/bin/passenger-
status:71
from /usr/bin/passenger-status:19:in `load'
from /usr/bin/passenger-status:19
I had a similar problem in the past with global queue enabled where
the 100% process would hang everything.

[RichardBronosky]

We are bucket testing Phusion Passenger in our production environment
and have encountered this problem a few times. I would love to give
you some reliable feedback from a reasonably high traffic site. Can
you give me some instructions on how implement your fix? I'd prefer
installing a gem rather than patching passenger.

For the sake of context, you are not speaking to a Ruby/Rails
developer. I manage databases and systems administrators. So a lot of
the lingo that the cool kids know will be wasted on me.

Please advise.

On Mar 11, 6:38 pm, Hongli Lai <hon...@phusion.nl> wrote:
> I *think* I've managed to implement a workaround for this problem in
> commit 1ae59609e1375. Can anyone verify?
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Hongli Lai]

On Thu, Apr 2, 2009 at 3:06 PM, RichardBronosky
<Richard...@gmail.com> wrote:
>
> We are bucket testing Phusion Passenger in our production environment
> and have encountered this problem a few times. I would love to give
> you some reliable feedback from a reasonably high traffic site. Can
> you give me some instructions on how implement your fix? I'd prefer
> installing a gem rather than patching passenger.
>
> For the sake of context, you are not speaking to a Ruby/Rails
> developer. I manage databases and systems administrators. So a lot of
> the lingo that the cool kids know will be wasted on me.
>
> Please advise.

The patch is already included in Phusion Passenger 2.1.3. The
following commit shows the change:
http://github.com/FooBarWidget/passenger/commit/1ae59609e137537a3f91ddd09c6a1cdfc0c71a2a
It contains a big comment block that explains what the change does.
The language is very technical but because of the nature of the issue
it cannot be explained in any less technical language.

At the moment I still don't know what causes the problem.

Do you experience this issue often?

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl