File descriptor in bad state

[Dusan Miloradovic]

Hi

I hope someone can  give me some directions about this issue I have . I recently upgraded the production server from HTTP Kit to Immutant, because effectively the server was becoming  unresponsive every week or so. I was hoping that the issue was with the HTTP Kit itself, and the move to Immutant would solve the issue.  With the HTTP Kit I didn't get any error logs when it was  hanging, but with Immutant, this is what I got:

Exception in thread "XNIO-1 I/O-4" java.lang.InternalError: File descriptor in bad state

        at sun.nio.ch.EventPortWrapper.release(EventPortWrapper.java:235)

        at sun.nio.ch.EventPortSelectorImpl.implDereg(EventPortSelectorImpl.java:144)

        at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:150)

        at sun.nio.ch.EventPortSelectorImpl.doSelect(EventPortSelectorImpl.java:75)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

        at org.xnio.nio.WorkerThread.run(WorkerThread.java:510)

Caused by: java.io.IOException: File descriptor in bad state

        at sun.nio.ch.SolarisEventPort.port_dissociate(Native Method)

        at sun.nio.ch.EventPortWrapper.release(EventPortWrapper.java:233)

        ... 6 more

I am running Immutant version 2.1.4 on Sun Solaris version SunOS 5.11 with Oracle JDK 1.7 and  clojure 1.8.0

Are there any Immutant/Undertow parameters I should be aware of with regards to the problem?

Thanks

[Toby Crawley]

I haven't seen that error before, but haven't ever used Immutant on
Solaris. I've forwarded your message to
undert...@lists.jboss.org[1], we'll see if the Undertow devs have
any insight.

- Toby

[1]: http://lists.jboss.org/pipermail/undertow-dev/2016-June/001606.html

> --
> You received this message because you are subscribed to the Google Groups
> "Immutant" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to immutant+u...@googlegroups.com.
> To post to this group, send email to immu...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/immutant/13c058c9-b079-4b82-8a01-4f3ed0c1ffda%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Dusan Miloradovic]

Thank you Toby,

I will follow the replies from the Undertow team.

The error looks like socket/connection leak to me, and I was wondering under what circumstances something like that  may happen in Immutant/Undertow?

I will check again my code, maybe I have potential leaks somewhere else .

Dusan

[Toby Crawley]

Dusan:

I don't know of anything in Immutant off-hand that would cause a
socket leak, and I would suspect that would manifest as an error
related to too many open file descriptors. The undertow devs
recommended trying a newer Undertow + XNIO, which you can do with an
incremental build of Immutant (we've using the latest stable Undertow
in build #739 and newer). Are you able to give that a try? See
http://immutant.org/builds/2x/ for details on using an incremental
build.

They also recommended trying a newer JDK, if you are able.

- Toby

> https://groups.google.com/d/msgid/immutant/d824f38e-5b04-4bc2-96f5-1ff0e714cb01%40googlegroups.com.

[Dusan Miloradovic]

Hi Toby,

I looked at it more closely, and it is not caused by the large number of opened sockets. It happened yesterday also for the testing environment.

At the time of the failure, I did the netstat -an and there were several sockets in the CLOSED_WAIT status, and in the log file I had the same exception as before.

Immutant is capturing the client close in on-close callback, as I can see from the log files, and I am closing the open channels at the end of the user session, but some sockets are not completely closed.

I already upgraded to the JDK 1.8, and I will try with the incremental build version you sent me

Thanks

Dusan