Frequent Out of Memory in the node where the Hazelcast Monitor is deployed

[Mohan]

Hi All,

We are getting frequent out of memory in the Weblogic node where we
installed the hazelcast monitor, though the monitor is not used by us,
but still it breaks the wl node.

Some logs for your information,

INFO: HazelcastClient is CLIENT_CONNECTION_OPENING
Mar 14, 2011 9:33:34 AM
com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is STARTED
Mar 15, 2011 7:20:55 AM
com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is STARTING
Mar 15, 2011 7:20:55 AM
com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_OPENING
Mar 15, 2011 7:20:55 AM
com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is STARTED
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:
124)
at
weblogic.servlet.internal.ServletOutputStreamImpl.commit(ServletOutputStreamImpl.java:
451)
at
weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:
1401)
at
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:
1375)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
Mar 16, 2011 6:29:15 AM
com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is STARTING
Mar 16, 2011 6:29:15 AM
com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_OPENING
Mar 16, 2011 6:29:15 AM
com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is STARTED
Mar 17, 2011 5:11:03 AM
com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is STARTING
Mar 17, 2011 5:11:03 AM
com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_OPENING
Mar 17, 2011 5:11:03 AM
com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is STARTED
Exception in thread "VDE Replication Thread"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "Timer-1" java.lang.OutOfMemoryError: Java heap
space
Mar 17, 2011 6:03:35 PM com.hazelcast.client.OutRunnable
INFO: OutRunnable is finished.
Exception in thread "hz.client.2.OutThread"
java.lang.OutOfMemoryError: Java heap space
Mar 17, 2011 6:28:18 PM com.hazelcast.nio.AbstractSerializer
SEVERE: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "[ACTIVE] ExecuteThread: '9' for queue:
'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError:
Java heap space
Exception in thread "[STUCK] ExecuteThread: '6' for queue:
'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError:
Java heap space
Mar 17, 2011 6:30:46 PM com.hazelcast.nio.AbstractSerializer
SEVERE: Java heap space
java.lang.OutOfMemoryError: Java heap space
Mar 17, 2011 6:31:31 PM com.hazelcast.nio.AbstractSerializer
SEVERE: Java heap space
java.lang.OutOfMemoryError: Java heap space
Mar 17, 2011 6:31:46 PM com.hazelcast.nio.AbstractSerializer
SEVERE: Java heap space
java.lang.OutOfMemoryError: Java heap space
Mar 17, 2011 6:31:46 PM com.hazelcast.nio.AbstractSerializer
SEVERE: Java heap space
java.lang.OutOfMemoryError: Java heap space
Mar 17, 2011 6:31:46 PM com.hazelcast.nio.AbstractSerializer
SEVERE: Java heap space

Could you help find out the reason, the tool is very much useful to us
but at the same time we are unable to find the reason of it getting
oom.

I've generated the Thread dump and there are 64 threads waiting for
monitoring entry related to Network operation,

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=3
tid=0x007d2480 nid=0x52 waiting for monitor entry
[0xc2e7f000..0xc2e7fb70]
at
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:
93)
- waiting to lock <0xd8849608> (a java.lang.String)
at
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
at
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:
42)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:
145)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

Thanks,
Mohan