mongrel crashing
0 views
Skip to first unread message
Tom Anderson
May 7, 2008, 12:06:40 AM5/7/08
to derailed
Anyone else have any experiences with Mongrel processes crashing? Our server loads are increasing and mongrel processes seem to be dying unexpectedly more and more frequently. Unfortunately the mongrel logging is not very helpful in debugging the problem.
Our basic setup uses Apache 2 with mod_proxy and mongrel. Our database is postgresql if that makes any difference.
Have you experienced crashes? If so, how do you deal with it? Do you just live with it and restart the processes or is your configuration more stable than ours?
~Tom
Our basic setup uses Apache 2 with mod_proxy and mongrel. Our database is postgresql if that makes any difference.
Have you experienced crashes? If so, how do you deal with it? Do you just live with it and restart the processes or is your configuration more stable than ours?
~Tom
Kevin Williams
May 7, 2008, 7:55:06 AM5/7/08
to dera...@googlegroups.com
I can't really help with mongrel, but I can say that I get a lot more
performance from thin and nginx than I ever did from apache2 and
mongrel. That's how I solved my crashing and memory issues.
performance from thin and nginx than I ever did from apache2 and
mongrel. That's how I solved my crashing and memory issues.
--
Cheers,
Kevin Williams
http://bantamtech.com/
http://almostserio.us/
http://kevwil.com/
bretweinraub
May 7, 2008, 1:14:15 PM5/7/08
to DeRailed - Denver Rails UG
OS?
Not sure about Macs; but Linux boxes support strace. There are
similar tools out
there (Solaris has "truss"). You might want to start a mongrel
instance or two inside
of strace and send the logging to a file. You'll get a dump of every
system call generated
by the process; you might find a smoking gun pattern in there.
Also you'll want to identify why the process exited (signal, SEGV,
exit?) This information
is available to the process that spawns mongrel; so you might want to
start a few up outside
of mongrel cluster. Your strace output may or may not help you see
this.
If you still have no luck; you might want to get a build a ruby
executable w/ the debugging on
and rig it (via ulimit) to generate a core file (if it is indeed core
dumping).
Then you can load it into GDB (or other tool) and look at the stack
trace.
Also I've got a little script called blackBox that I've used for
years. Basically it work like a
flight recorder so I get a snapshot of activity on the box when some
event occurs. I'll usually
monitor things like vmstat, netstat, iostat to see if there's some
sort of correlation between some
environmental issue and the problem. For example you might be
crashing because the process runs
out of file descriptors; but you might not see this in the logs but
you would see this in strace and
you might see it via data collection in netstat and/or lsof.
Its a bash script ... but I can get you hooked up w/ it if you want.
good luck!
-bdw
Not sure about Macs; but Linux boxes support strace. There are
similar tools out
there (Solaris has "truss"). You might want to start a mongrel
instance or two inside
of strace and send the logging to a file. You'll get a dump of every
system call generated
by the process; you might find a smoking gun pattern in there.
Also you'll want to identify why the process exited (signal, SEGV,
exit?) This information
is available to the process that spawns mongrel; so you might want to
start a few up outside
of mongrel cluster. Your strace output may or may not help you see
this.
If you still have no luck; you might want to get a build a ruby
executable w/ the debugging on
and rig it (via ulimit) to generate a core file (if it is indeed core
dumping).
Then you can load it into GDB (or other tool) and look at the stack
trace.
Also I've got a little script called blackBox that I've used for
years. Basically it work like a
flight recorder so I get a snapshot of activity on the box when some
event occurs. I'll usually
monitor things like vmstat, netstat, iostat to see if there's some
sort of correlation between some
environmental issue and the problem. For example you might be
crashing because the process runs
out of file descriptors; but you might not see this in the logs but
you would see this in strace and
you might see it via data collection in netstat and/or lsof.
Its a bash script ... but I can get you hooked up w/ it if you want.
good luck!
-bdw
gvarela
May 7, 2008, 4:58:40 PM5/7/08
to DeRailed - Denver Rails UG
You might want to double check the memory on the mongrel box and make
sure it is sufficient. We had a box where the linux kernel was
limiting the memory to 3gb and we were seeing mongrels die
occasionally. We patch the Kernel to make full use of the hardware
(8gb of Ram) and now the mongrels seem to be more stable. Another
approach might be to check out thin. Apparently it is more stable than
mongrel though I haven't had time to verify.
sure it is sufficient. We had a box where the linux kernel was
limiting the memory to 3gb and we were seeing mongrels die
occasionally. We patch the Kernel to make full use of the hardware
(8gb of Ram) and now the mongrels seem to be more stable. Another
approach might be to check out thin. Apparently it is more stable than
mongrel though I haven't had time to verify.
Reply all
Reply to author
Forward
0 new messages