Threads get blocked in java.lang.ClassLoader.loadClass() during put()/get() operations

[Dinesh Kumar]

Hello,

I'm facing a problem which is clearly identical to the issue mentioned in the SO link below.

http://stackoverflow.com/questions/29771612/blocking-in-classloading-while-quering-hazelcast

[text from SO]

We are using hazelcast as a distributed cache. After the application runs for a certain time we start to get blocking in classloading. Following is the stacktrace :

java.lang.Thread.State: BLOCKED (on object monitor)      at java.lang.ClassLoader.loadClass(ClassLoader.java:404)      - locked < 0x00002acaac4c4718> (a java.lang.Object)      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)      at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:124)      at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:97)      at com.hazelcast.nio.IOUtil$1.resolveClass(IOUtil.java:113)      at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)      at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)      at com.hazelcast.nio.serialization.DefaultSerializers$ObjectSerializer.read(DefaultSerializers.java:196)      at com.hazelcast.nio.serialization.StreamSerializerAdapter.toObject(StreamSerializerAdapter.java:65)      at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:260)      at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:186)      at com.hazelcast.map.impl.AbstractMapServiceContextSupport.toObject(AbstractMapServiceContextSupport.java:42)      at com.hazelcast.map.impl.DefaultMapServiceContext.toObject(DefaultMapServiceContext.java:28)      at com.hazelcast.map.impl.proxy.MapProxySupport.toObject(MapProxySupport.java:1038)      at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:84)

@noctarius replied :

his is not Hazelcast specific, whenever you create an instance you have to ask the classloader for the class, no matter you use reflection or a new call. The problem is actually when synchronized classloaders come into the play (like in webapps or stuff). Hazelcast, obviously, has to deserialize a lot and therefore requests a lot of classes.

The internal deserialization is kind of optimized by now (by caching the conctructor instances - as far as I remember) but the Java standard serialization (the one you use) always wants the class and classes aren't yet cached.

I'm facing this issue in performance runs where I have simulated 1500 threads doing login operations (which in turn does close to 10 get() calls and 5 put() calls.

I understand that issue is not directly associated with hazelcast and is linked to the synchronized block in ClassLoader.

Q1. However, can you suggest some work around ?

Q2. If @noctarius is pointing to make use of custom serialization, how will it solve the problem ?

Thanks,

Dinesh