Constantly growing number of open file when running on Weblogic in cluster mode with master instance configured as JCA adapter

[Viacheslav Yakunin]

Hi Nitin,

We are continuing testing application with XADisk file store. As I mentioned, the configuration is that we are runing two nodes on Weblogic cluster. We have XADisk configured as JCA adapter which serves as transactions coordinator (disk operations are performed by application instances). During stress testing we discovered that any file operations done via XADisk in application instances result in increasing of number of open files, that seems never get closed. On some servers we hit limit of open files and application crashed. 

In the JCA adapter logs we we sometimes see:

2014-12-10 09:32:14

Caused By: java.io.IOException: Too many open files in system

        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:150)

        at org.xadisk.bridge.server.PointOfContact.run(PointOfContact.java:47)

On client side it may look like:

Caused By: java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
        at java.nio.channels.SocketChannel.open(SocketChannel.java:146)
        at org.xadisk.bridge.proxies.facilitators.RemoteMethodInvoker.ensureConnected(RemoteMethodInvoker.java:56)
        at org.xadisk.bridge.proxies.facilitators.RemoteMethodInvoker.invokeRemoteMethod(RemoteMethodInvoker.java:79)
        at org.xadisk.bridge.proxies.facilitators.RemoteObjectProxy.invokeRemoteMethod(RemoteObjectProxy.java:39)

I would like to ask if you have any hint why this issue could appear and may be advise what can be done to get rid of it?

Thanks and regards

Sla

[Viacheslav Yakunin]

Attaching chart from monitoring tool we used to profile application.

 

[Viacheslav Yakunin]

Seems it was related to the known bug: https://java.net/jira/browse/XADISK-152

We rebuilt xadisk from trunk and problem disappeared.

Thanks so much

Sla

[Nitin Verma]

Hi Sla,

Good to know that the problem got resolved.

Thanks,
Nitin