stat file missing - fatal error

[Matt]

Hi folks,

Just set up god on a new server, running nginx with a mongrel
cluster. I'm getting this error in the log:

FATAL: Unhandled exception in driver loop - (Errno::ENOENT): No such
file or directory - /proc/5138/stat
/usr/local/lib/ruby/gems/1.8/gems/god-0.7.6/bin/../lib/god/system/
slash_proc_poller.rb:57:in `read'
/usr/local/lib/ruby/gems/1.8/gems/god-0.7.6/bin/../lib/god/system/
slash_proc_poller.rb:57:in `stat'
/usr/local/lib/ruby/gems/1.8/gems/god-0.7.6/bin/../lib/god/system/
slash_proc_poller.rb:28:in `percent_cpu'
/usr/local/lib/ruby/gems/1.8/gems/god-0.7.6/bin/../lib/god/system/
process.rb:27:in `percent_cpu'
/usr/local/lib/ruby/gems/1.8/gems/god-0.7.6/bin/../lib/god/conditions/
cpu_usage.rb:65:in `test'
/usr/local/lib/ruby/gems/1.8/gems/god-0.7.6/bin/../lib/god/task.rb:
317:in `handle_poll'
/usr/local/lib/ruby/gems/1.8/gems/god-0.7.6/bin/../lib/god/driver.rb:
68:in `handle_event'


I'm guessing that one of the mongrel processes has stopped but not
completely cleaned up after itself. any ideas on nudging god to
restart the process instead of blow up?

thanks

[Kevin Clark]

Are you running FreeBSD 7.0? We just got a patch in that fixed
detection of the proc file system when the directory exists but the fs
isn't mounted:

http://github.com/kevinclark/god/commit/b6f9cad5479c102da63fe0c614751851d6540027

Doesn't look like Tom has pulled it in yet. If this is hitting people,
we should probably push a point release. I'll update the history in
prep.

--
Kevin Clark
http://glu.ttono.us

[Matt]

Thanks for the reply. Sorry, should have mentioned - this is on debian
4.0

On Jun 17, 2:18 am, "Kevin Clark" <kevin.cl...@gmail.com> wrote:
> Are you running FreeBSD 7.0? We just got a patch in that fixed
> detection of the proc file system when the directory exists but the fs
> isn't mounted:
>

> http://github.com/kevinclark/god/commit/b6f9cad5479c102da63fe0c614751...

[Kevin Clark]

Tom just pushed 0.7.7 to Rubyforge. Try it out, let us know if it
fixes your issue.

[Matt Cowley]

So far so good. *Awesome*, thanks!

[Matt Cowley]

Hmm..got the same issue again:

[2008-06-18 13:44:21] FATAL: Unhandled exception in driver loop -
(Errno::ENOENT): No such file or directory - /proc/20082/stat
/usr/local/lib/ruby/gems/1.8/gems/god-0.7.7/bin/../lib/god/system/
slash_proc_poller.rb:71:in `read'

Is this possibly a configuration issue on my side?

thanks

On Jun 17, 2008, at 6:30 PM, Kevin Clark wrote:

[Kevin Clark]

Doh. It _could_ be configuration, but I don't think so.

When this occurs, is the process running? Seems like it could be a
race condition where the process has died, but god hasn't noticed, so
it goes to check stats in /proc, but the info for the now missing pid
isn't there.

[Matt Cowley]

That sounds likely. It ran for several hours before this exception
occurred, so my guess is that it didn't get thrown until a process died.

[Kevin Clark]

>> On Jun 18, 2008, at 1:40 PM, Kevin Clark wrote:
>>
>> Doh. It _could_ be configuration, but I don't think so.
>>
>> When this occurs, is the process running? Seems like it could be a
>> race condition where the process has died, but god hasn't noticed, so
>> it goes to check stats in /proc, but the info for the now missing pid
>> isn't there.
>

>On Wed, Jun 18, 2008 at 12:31 PM, Matt Cowley <madc...@gmail.com> wrote:
>
> That sounds likely. It ran for several hours before this exception
> occurred, so my guess is that it didn't get thrown until a process died.
>
>

Ok, I'll try to take a look today. In the meantime, if you want to
modify your install so SlashProcPoller.usable? always returns false,
it'll go back to using ps. This should probably be configurable
anyway.

[Kevin Clark]

Hey, sorry it's taken so long to do this. I believe the fix is here:

http://github.com/kevinclark/god/commit/a74780fca54642e7687a79e7f165b58587c9710a

I'm poking Tom about pushing an 0.7.9 release today to fix the issue.

[Matt Cowley]

cool, thanks. i'll try it out whenever the release is ready.