Errno::EMFILE: Too many open files - socket(2)

[Jesse Chan-Norris]

So I've been having a problem in production that I can't seem to track
down, nor can I figure out even how to go about debugging it.

After running for a day or so (sometimes less), Juggernaut stops
responding. The only messages in the log are the following:

ERROR -- : Bad request http://192.168.1.78/streams/connection_logout
(Errno::EMFILE: Too many open files - socket(2))

The URL that is being called is the callback when the connection is
terminated. But juggernaut won't respond to external requests either -
it looks like the process just has too many open sockets.

When I do an lsof on the juggernaut process, I get the following:

juggernau 12766 user 10u IPv4 119101789 TCP
192.168.1.78:commplex-link->adsl-xxx-xxx-xxx-
xxx.dsl.hstntx.sbcglobal.net:53495 (ESTABLISHED)
juggernau 12766 user 11u IPv4 119077236 TCP
192.168.1.78:commplex-link->xxx.xxx.xxx.xxx:51571 (ESTABLISHED)
juggernau 12766 user 12u IPv4 119068904 TCP
192.168.1.78:commplex-link->cpe-xx-xx-xx-xx.socal.res.rr.com:50261
(ESTABLISHED)
juggernau 12766 user 13u IPv4 119066637 TCP
192.168.1.78:commplex-link->ipxx-xx-xx-xx.ri.ri.cox.net:4139
(ESTABLISHED)

I have about 3 times the these connections to the juggernaut server as
I do actual clients that are connected according to
Juggernaut.show_clients.

So it looks like connections aren't getting cleaned up somehow, but I
don't really know how to test this.

Anyone have any ideas?

[heisee]

Hi Jesse,

maybe this could help you:

http://groups.google.com/group/Juggernaut-for-Rails/browse_thread/thread/c3c0c4a802a59e9c/2b675650482f2e40

In old versions of the Juggernaut gem the client connections were not
disconnected. I suggested a fix based on the version that was current
at that time. If I olook in the current code, it was solved in a
different way, but it should be solved.
Another interesting thread my be:

http://groups.google.com/group/Juggernaut-for-Rails/browse_thread/thread/eba93f9092e30f73/c39dbaec189c6dc9

especially the posting with the system settings
* hard nofile 65535
* soft nofile 65535

good luck,
Heiko Seebach

On Jan 18, 6:34 pm, Jesse Chan-Norris <j...@pith.org> wrote:
> So I've been having a problem in production that I can't seem to track
> down, nor can I figure out even how to go about debugging it.
>
> After running for a day or so (sometimes less), Juggernaut stops
> responding. The only messages in the log are the following:
>

> ERROR -- : Bad requesthttp://192.168.1.78/streams/connection_logout

[Jesse Chan-Norris]

Heiko,

I raised the nofile setting and this at least prevents the "too many
open files" error. However, looking at the lsof output, it doesn't
look like old connections are going away at all. Juggernaut's been
running for about a day and I have 900 connections in the ESTABLISHED
state, but we only have 261 actual connections (according to
Juggernaut). I wonder if eventmachine isn't letting go of the
connections for some reason...

jcn

On Jan 19, 6:23 am, heisee <heiko.seeb...@googlemail.com> wrote:
> Hi Jesse,
>
> maybe this could help you:
>

> http://groups.google.com/group/Juggernaut-for-Rails/browse_thread/thr...

>
> In old versions of the Juggernaut gem the client connections were not
> disconnected. I suggested a fix based on the version that was current
> at that time. If I olook in the current code, it was solved in a
> different way, but it should be solved.
> Another interesting thread my be:
>

> http://groups.google.com/group/Juggernaut-for-Rails/browse_thread/thr...

[heisee]

> I wonder if eventmachine isn't letting go of the
> connections for some reason...

hmm, as far as I can see in the new sources (I still have a quite old
juggernaut server version running), old client connections should be
disconnected...
Maybe someone running the newest jug server can confirm this?

thanks, Heiko