mongrel goes unresponsive after 30 minutes
Bill Walton
I'm in the process of moving my app from a shared hosting arrangement to a
VPS and am seeing some mongrel behavior I haven't seen before.
Specifically, after about 30 minutes of inactivity mongrel goes unresponsive
and has to be restarted. When I restart it writes the messages below.
Before that, there's nothing in the logs that shows any problem at all.
In mongrel.log:
** USR2 signal received.
Thu Feb 08 08:43:51 EST 2007: Reaping 2 threads for slow workers because of
'shutdown'
Thread # is too old, killing.
Thu Feb 08 08:43:51 EST 2007: Error calling Dispatcher.dispatch #Thread # is
too old, killing.
In production.log:
Mongrel timed out this thread: shutdown
A set of trace messages follow this which I'm happy to post if it would be
helpful.
Can anybody help me understand what's going on and what I need to do to fix
it?
Thanks much,
Bill
Aníbal Rojas
If you find that Mongrel stops working after a long idle time and
you're using MySQL then you're hitting a bug in the MySQL driver that
doesn't properly timeout connections. What happens is the MySQL server
side of the connection times out and closes, but the MySQL client
doesn't detect this and just sits there.
What you have to do is set:
ActiveRecord::Base.verification_timeout = 14400
Or to any value that is lower than the MySQL server's
interactive_timeout setting. This will make sure that ActiveRecord
checks the connection often enough to reset the connection.
Q: Mongrel stops working if it's left alone for a long time.
If you find that Mongrel stops working after a long idle time and
you're using MySQL then you're hitting a bug in the MySQL driver that
doesn't properly timeout connections. What happens is the MySQL server
side of the connection times out and closes, but the MySQL client
doesn't detect this and just sits there.
What you have to do is set:
ActiveRecord::Base.verification_timeout = 14400
Or to any value that is lower than the MySQL server's
interactive_timeout setting. This will make sure that ActiveRecord
checks the connection often enough to reset the connection.
--
Aníbal Rojas
http://www.rubycorner.com
http://www.hasmanydevelopers.com
Aníbal Rojas
http://mongrel.rubyforge.org/faq.html
On Feb 8, 4:54 pm, "Aníbal Rojas" <anibalro...@gmail.com> wrote:
> >From Q: Mongrel stops working if it's left alone for a long time.
>
> If you find that Mongrel stops working after a long idle time and
> you're using MySQL then you're hitting a bug in the MySQL driver that
> doesn't properly timeout connections. What happens is the MySQL server
> side of the connection times out and closes, but the MySQL client
> doesn't detect this and just sits there.
>
> What you have to do is set:
>
> ActiveRecord::Base.verification_timeout = 14400
>
> Or to any value that is lower than the MySQL server's
> interactive_timeout setting. This will make sure that ActiveRecord
> checks the connection often enough to reset the connection.
>
> Q: Mongrel stops working if it's left alone for a long time.
>
> If you find that Mongrel stops working after a long idle time and
> you're using MySQL then you're hitting a bug in the MySQL driver that
> doesn't properly timeout connections. What happens is the MySQL server
> side of the connection times out and closes, but the MySQL client
> doesn't detect this and just sits there.
>
> What you have to do is set:
>
> ActiveRecord::Base.verification_timeout = 14400
>
> Or to any value that is lower than the MySQL server's
> interactive_timeout setting. This will make sure that ActiveRecord
> checks the connection often enough to reset the connection.
>
> --
> Aníbal Rojashttp://www.rubycorner.comhttp://www.hasmanydevelopers.com
Bill Walton
Thanks! I've got the admin looking into now. I really appreciate the
response.
Best regards,
Bill
Aníbal Rojas
some other conditions that could cause this errors.
Robby Russell
Do you have a cluster of mongrels running or just the one?
-Robby
--
Robby Russell
http://www.robbyonrails.com/
http://www.planetargon.com/
Bill Walton
Robby Russell wrote:
>
> Do you have a cluster of mongrels running or just the one?
I'm not sure and asked similar question of my hosting service today. I'll
report back on that as soon as I find out. For development purposes I
started with a shared hosting plan with the same provider and the app I'm
having problems with is running just fine on that site. I haven't even
browsed to it in almost two weeks and when I did this afternoon I found that
it's just 'hummin along'. It's not _exactly_the same app, but there are
very few differences. My first instinct is to look for differences in the
code between the two but, IME, that's really only going to be a productive
approach if the underlying architecture is comparable. Any debugging advice
/ experience is tremendously appreciated.
Thanks!
Bill
Zed A. Shaw
"Bill Walton" <bill....@charter.net> wrote:
>
> Greetings!
>
> I'm in the process of moving my app from a shared hosting arrangement to a
> VPS and am seeing some mongrel behavior I haven't seen before.
> Specifically, after about 30 minutes of inactivity mongrel goes unresponsive
> and has to be restarted. When I restart it writes the messages below.
> Before that, there's nothing in the logs that shows any problem at all.
>
Restart (USR2) isn't that reliable because of how ruby/rails needs to
reload. You should use a full stop/start cycle instead.
Once you start doing that and you still get no response then run strace
on the mongrel process to see what it's doing. It's most likely that
you have a library that's messing things up.
Also, don't use pstore, don't set Logger to rotate, don't do file
locking, and don't use a crappy DNS.
--
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
Bill Walton
Thanks very much for your reply. I appreciate it very much. More below.
Zed A. Shaw wrote:
> Restart (USR2) isn't that reliable because of
> how ruby/rails needs to reload. You should
> use a full stop/start cycle instead.
I've been doing that. Should have said it that way.
> Once you start doing that and you still get no
> response then run strace on the mongrel process
> to see what it's doing.
The folks at the hosting service (a2hosting) have done that I think and they
tell me it hasn't given them any clues.
> It's most likely that you have a library that's messing
> things up.
If by 'library' you mean gems / plugins, I do have a couple installed:
PDF::Writer and BackgroundRb. But BackgroundRb is stopped and there are no
calls to PDF:Writer being used in the test scenerio I've been using. I
thought Rails didn't load code before it needed it. Anyway, here's what I'm
doing.
1) Start mongrel using mongrel_rails start -d -p 3002 -e production <
/dev/null >& /dev/null
2) Open a browser and browse to the index action of the Rails app (the index
action is empty and the view renders a page with a form on it that has a
group of 3 radio buttons and a submit button.)
3) Close the browser.
4) Wait 40 minutes, then do step 2 again.
The result of step 4 is a blank white screen, not an error message. Neither
production.log nor mongel.log have anything written to them at that point
other than the initial startup messages in mongrel.log and the initial
request from step 2 in production.log. When I stop mongrel, it writes a
message to both logs saying its killing a slow worker. That's the only clue
I get. Steps 1-4 can be repeated ad nauseum.
After their initial investigation the sys admins concluded the problem had
to be in my code (surprise, surprise ;-) ) I had a slightly different
version that was running flawlessly on my shared account with them, so I
decided to 'dig in.' I copied the entire Rails application directory for
both apps to my PC, then copied the one I was having problems with on the
VPS over to the shared account. It ran without a problem (and, a week
later, is *still* running without a problem.)
Yesterday they set up a cron job outside my space to invoke the index method
every 10 minutes. That's 'solved' the problem. It doesn't sit well with me
though. My experience leads me to fear that whatever the real cause of the
problem is will eventually manifest itself again. At the worst possible
time.
If you (or anyone else reading this) have any interest in understanding
what's really going on (since mongrel's behavior is the only clue from an
end user perspective), please let me know. I'll give you any access you
need. If not, I understand completely.
Thanks again for your time.
Best regards,
Bill
Zed A. Shaw
"Bill Walton" <bill....@charter.net> wrote:
> Yesterday they set up a cron job outside my space to invoke the index method
> every 10 minutes. That's 'solved' the problem. It doesn't sit well with me
> though. My experience leads me to fear that whatever the real cause of the
> problem is will eventually manifest itself again. At the worst possible
> time.
Interesting, make sure you've got the compiled mysql gem and you aren't
using the rails mysql.rb file.
> If you (or anyone else reading this) have any interest in understanding
> what's really going on (since mongrel's behavior is the only clue from an
> end user perspective), please let me know. I'll give you any access you
> need. If not, I understand completely.
Yeah, actually I'd be interested. Email me off list to coordinate it.
Alexey Verkhovsky
make sure you've got the compiled mysql gem and you aren't
using the rails mysql.rb file.
Zed,
Can you point me to the reason for this recommendation? Is there a known problem with pure-Ruby MySQL driver?
Best regards,
Alexey Verkhovsky
Bill Walton
both recognize the contributors and to document it for future Railers.
Thank you to all.
Best regards,
Bill
Bill Walton wrote:
> I'm in the process of moving my app from a shared
> hosting arrangement to a VPS and am seeing some
> mongrel behavior I haven't seen before. Specifically,
> after about 30 minutes of inactivity mongrel goes
> unresponsive and has to be restarted.
Zed A. Shaw wrote:
> make sure you've got the compiled mysql gem and
> you aren't using the rails mysql.rb file.
Ezra Zygmuntowicz wrote:
> Also if your host is running linux then you can tell for
> sure if it's the compiled driver from irb:
>
> ey00-s00059 ~ # irb
> irb(main):001:0> require 'mysql.so'
> => true
Bill Walton wrote:
> When I run 'gem list' the list includes:
> mysql (2.7)
>
> BUT ...
>
> bwa...@yourtimematters.com [~/emrec]# irb
> irb(main):001:0> require 'mysql.so'
> LoadError: no such file to load -- mysql.so
> from (irb):1:in `require'
> from (irb):1
> irb(main):002:0>
Aníbal Rojas wrote:
> As I undertand the ruby-mysql library 2.7 does
> require the libmysqlclient to be properly installed,
> and depending on the location of mysql.so it is
> possible that rubygems is unable to load it, while
> the library is actually using it...
Support at hosting service wrote:
>This issue has been rectified:
>
> bwa...@yourtimematters.com [~]# irb
> irb(main):001:0> require 'mysql.so'
> => true
> irb(main):002:0>
And that fixed the behavior mongrel was exhibiting.