ClassNotFoundException after upgrade from 3.4.2 to 3.7.5

[Tonda -]

Hi,

I am trying to upgrade Hazelcast from version 3.4.2 to 3.7.5. But I am getting exception listed below.

Environment is Weblogic 12.1.3 domain with 4 instances organized in 2 clusters, two applications accesing same ReplicatedMap are deployed in a way that each creates own Hazelcast node, resulting in 8 nodes in Hazelcast cluster. Main Hazelcast jar is in global classpath, StorageNode class is packed in both application. When Weblogic server is restarted Hazelcast node joins cluster and objects are replicated to newly started server and this exceptions occurs. This did not happen with Hazelcast 3.4.2. Are there any changes in replicated map between 3.4.2 and 3.7.5 that can cause this?

Thanks.

####<Feb 1, 2017 11:49:16 AM CET> <Error> <com.hazelcast.spi.impl.operationservice.impl.AsyncResponseHandler> <twls04> <extranet_c1_s2a> <hz._hzInstance_1_hypo-deva.response> <<anonymous>> <> <> <1485946156062> <BEA-000000> <[10.88.13.99]:5701 [hypo-deva] [3.7.5] Failed to process response: Packet{flags=3, isResponse=true, isOperation=true, isEvent=false, partitionId=-1, conn=Connection[id=6, /10.88.13.99:5701->/10.88.13.98:59364, endpoint=[10.88.13.98]:5703, alive=true, type=MEMBER]} on:hz._hzInstance_1_hypo-deva.response

com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: cz.csas.inet.castlebravo.api.storage.StorageNode

        at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:224)

        at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)

        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:251)

        at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600)

        at com.hazelcast.nio.IOUtil.readObject(IOUtil.java:120)

        at com.hazelcast.replicatedmap.impl.operation.VersionResponsePair.readData(VersionResponsePair.java:58)

        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:151)

        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:104)

        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:54)

        at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)

        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:251)

        at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600)

        at com.hazelcast.spi.impl.operationservice.impl.responses.NormalResponse.readData(NormalResponse.java:104)

        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:151)

        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:104)

        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:54)

        at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)

        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:172)

        at com.hazelcast.spi.impl.operationservice.impl.ResponseHandler.handle(ResponseHandler.java:79)

        at com.hazelcast.spi.impl.operationservice.impl.AsyncResponseHandler$ResponseThread.doRun(AsyncResponseHandler.java:151)

        at com.hazelcast.spi.impl.operationservice.impl.AsyncResponseHandler$ResponseThread.run(AsyncResponseHandler.java:138)

Caused By: java.lang.ClassNotFoundException: cz.csas.inet.castlebravo.api.storage.StorageNode

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:122)

        at com.hazelcast.nio.IOUtil$ClassLoaderAwareObjectInputStream.resolveClass(IOUtil.java:358)

        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)

        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)

        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)

        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)

        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)

        at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:219)

        at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)

        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:251)

        at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600)

        at com.hazelcast.nio.IOUtil.readObject(IOUtil.java:120)

        at com.hazelcast.replicatedmap.impl.operation.VersionResponsePair.readData(VersionResponsePair.java:58)

        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:151)

        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:104)

        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:54)

        at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)

        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:251)

        at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600)

        at com.hazelcast.spi.impl.operationservice.impl.responses.NormalResponse.readData(NormalResponse.java:104)

        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:151)

        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:104)

        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:54)

        at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)

        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:172)

        at com.hazelcast.spi.impl.operationservice.impl.ResponseHandler.handle(ResponseHandler.java:79)

        at com.hazelcast.spi.impl.operationservice.impl.AsyncResponseHandler$ResponseThread.doRun(AsyncResponseHandler.java:151)

        at com.hazelcast.spi.impl.operationservice.impl.AsyncResponseHandler$ResponseThread.run(AsyncResponseHandler.java:138)

[dsukho...@gmail.com]

Looks like a ClassLoader issue at Java (de-)serialization. Try to put jar with StorageNode class on the global classpath too.

HTH, Denis.

[Jaromir Hamala]

Hi Tonda,

there was indeed a big redesign of the Replicated Map between 3.5 and 3.6. It appears we do unnecessary de-serilization.  I created a github issue for this. Let's see what we can do about it. 

For now the workaround is either to deploy your domain classes into your app server lib directory (global classpath) or move Hazelcast into your application. 

Cheers,

Jaromir

[Jaromir Hamala]

actually, can you try to switch InMemoryFormat of your replicated map to BINARY? It's OBJECT by default (for faster GETs), but keeping data stored in BINARY format could help to workaround the issue you are experiencing. 

Cheers,

Jaromir

[Tonda]

Hi,

thank you for your reply. I will definitely try that. I was already considering switching to BINARY as a possible workaround.

Regards,
Tonda

[Tonda]

Hi,

now I am confused. You confirmed that there is a problem when using OBJECT memory format and than closed the issue you created. So is it planned to be fixed or not?

In case I want to try to fix it myself, how difficult it would be? Can you provide me some hint, please?

Thanks,

Tonda

[Jaromir Hamala]

It's related to classloading: with the OBJECT format user objects are stored de-serialized (by definition) The deserialization is eager and it's done on threads managed by Hazelcast. When Hazelcast is loaded by the WebLogic global classloader then it does not have access to classes deployed inside your application(s) -> ClassNotFound during deserilization. This is expected & perhaps even desired (classloading isolation). There isn't much we can do about it. 

with the BINARY format the de-serilization is postponed until the latest possible moment - ie. a _user thread_ calls `map.get()`. At this point the thread context classloader is set to the webapplication (because WebLogic set it so) -> Hazelcast will use the context classloader > deserialization succeed. 

Cheers,

Jaromir

[Tonda]

I know this, I already faced several issues caused by classes on global classpath not seeing classes in WARs, I was little bit expecting this answer and it even sounds correct. But it worked in 3.4.2 so I am trying to get this resolved.

In the first post you of issue 9794 you created you suggest that deserialization could and should be postponed until object is accessed. So did you change mind and current beahaviour is correct and desired?

Thanks,

Tonda

[Jaromir Hamala]

Hi,

actually I was not aware the ReplicatedMap uses the OBJECT format by default. I thought there was an eager deserialization going on even with the BINARY format. Apparently that's not the case hence I closed the issue. 

Cheers,

Jaromir

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAGW5m6wMNRyN%2B5Tfoe_jg8os8AhDnVohc4WipruTuQ1vR89dBA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Tonda]

OK, so this is a feature change (rather unpleasant since it degrades usability for us) and we have to deal with that. Just to satisfy my curiosity, how was that done in 3.4.2 where the problem did not occur?

Thanks,

Tonda