ClassNotFoundException when merging map

[Keidi]

Hi,

I'm encountering a ClassNotFoundException when HazelCast tries to
recover from a split brain. The components running the HazelCast nodes
are not identical, and most classes are not shared (including some of
the classes of distributed objects). But it is my understanding that
serialization should occur only when the client thread tries to access
a distributed object and not when HazelCast tries to recover from a
network problem.
In my case it seems that when recovering from a split brain, a node
that does not own a certain class tries to serialize an object
assigned to that class.

Here's an excerpt from the log file:


22 Nov 2010 02:53:16,211
[hz.executor._hzInstance_0_dev.cached.thread-95] DEBUG
com.hazelcast.impl.SplitBrainHandler - [dev] Address[127.0.0.1:5701]
is connecting to Address[127.0.0.1:5703]
22 Nov 2010 02:53:16,211
[hz.executor._hzInstance_0_dev.cached.thread-95] DEBUG
com.hazelcast.impl.SplitBrainHandler - [dev] Address[127.0.0.1:5701]
is connecting to Address[127.0.0.1:5700]
22 Nov 2010 02:53:16,211
[hz.executor._hzInstance_0_dev.cached.thread-95] DEBUG
com.hazelcast.nio.OutSelector - [dev] connect to
Address[127.0.0.1:5700]
22 Nov 2010 02:53:16,211
[hz.executor._hzInstance_0_dev.cached.thread-95] DEBUG
com.hazelcast.impl.SplitBrainHandler - [dev] Address[127.0.0.1:5701]
is connecting to Address[127.0.0.1:5699]
22 Nov 2010 02:53:16,211
[hz.executor._hzInstance_0_dev.cached.thread-95] DEBUG
com.hazelcast.nio.OutSelector - [dev] connect to
Address[127.0.0.1:5699]
22 Nov 2010 02:53:16,226 [hz.OutThread] DEBUG
com.hazelcast.nio.OutSelector - [dev] connecting to
Address[127.0.0.1:5700]
22 Nov 2010 02:53:16,226 [hz.OutThread] DEBUG
com.hazelcast.nio.OutSelector - [dev] connection check. connected:
false, Address[127.0.0.1:5700]
22 Nov 2010 02:53:16,226 [hz.OutThread] DEBUG
com.hazelcast.nio.OutSelector - [dev] connecting to
Address[127.0.0.1:5699]
22 Nov 2010 02:53:16,226 [hz.OutThread] DEBUG
com.hazelcast.nio.OutSelector - [dev] connection check. connected:
false, Address[127.0.0.1:5699]
22 Nov 2010 02:53:17,256 [hz.OutThread] DEBUG
com.hazelcast.nio.OutSelector - [dev] Couldn't connect to
Address[127.0.0.1:5700], cause: Connection refused: no further
information
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
22 Nov 2010 02:53:17,256 [hz.OutThread] DEBUG
com.hazelcast.nio.OutSelector - [dev] Couldn't connect to
Address[127.0.0.1:5699], cause: Connection refused: no further
information
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at com.hazelcast.nio.OutSelector$Connector.handle(OutSelector.java:
62)
at com.hazelcast.nio.SelectorBase.run(SelectorBase.java:118)
at java.lang.Thread.run(Thread.java:619)
22 Nov 2010 02:53:17,256 [hz.InThread] INFO
com.hazelcast.nio.Connection - [dev] Connection lost
Address[127.0.0.1:5703]
22 Nov 2010 02:53:17,256 [hz.InThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] Connection is removed
Address[127.0.0.1:5703]
22 Nov 2010 02:53:17,256 [hz.ServiceThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] Removing Address
Address[127.0.0.1:5703]
22 Nov 2010 02:53:17,272 [hz.InThread] INFO
com.hazelcast.nio.InSelector - [dev] 5701 this socket is connected
to /127.0.0.1:52979
22 Nov 2010 02:53:18,270 [hz.ServiceThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] false Handling
JoinInfo{request=true, memberCount=1 JoinRequest{nodeType=MEMBER,
address=Address[127.0.0.1:5703], groupName='dev', groupPassword='dev-
pass', buildNumber='20101105', packetVersion='5'}}
22 Nov 2010 02:53:18,270 [hz.ServiceThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] false Handling
JoinInfo{request=true, memberCount=1 JoinRequest{nodeType=MEMBER,
address=Address[127.0.0.1:5703], groupName='dev', groupPassword='dev-
pass', buildNumber='20101105', packetVersion='5'}}
22 Nov 2010 02:53:18,270 [hz.ServiceThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] false Handling
JoinInfo{request=true, memberCount=1 JoinRequest{nodeType=MEMBER,
address=Address[127.0.0.1:5703], groupName='dev', groupPassword='dev-
pass', buildNumber='20101105', packetVersion='5'}}
22 Nov 2010 02:53:19,284 [hz.ServiceThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] false Handling
JoinInfo{request=true, memberCount=1 JoinRequest{nodeType=MEMBER,
address=Address[127.0.0.1:5703], groupName='dev', groupPassword='dev-
pass', buildNumber='20101105', packetVersion='5'}}
22 Nov 2010 02:53:20,298 [hz.ServiceThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] false Handling
JoinInfo{request=true, memberCount=1 JoinRequest{nodeType=MEMBER,
address=Address[127.0.0.1:5703], groupName='dev', groupPassword='dev-
pass', buildNumber='20101105', packetVersion='5'}}
22 Nov 2010 02:53:21,312 [hz.ServiceThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] false Handling
JoinInfo{request=true, memberCount=1 JoinRequest{nodeType=MEMBER,
address=Address[127.0.0.1:5703], groupName='dev', groupPassword='dev-
pass', buildNumber='20101105', packetVersion='5'}}
22 Nov 2010 02:53:22,326 [hz.ServiceThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] false Handling
JoinInfo{request=true, memberCount=1 JoinRequest{nodeType=MEMBER,
address=Address[127.0.0.1:5703], groupName='dev', groupPassword='dev-
pass', buildNumber='20101105', packetVersion='5'}}
22 Nov 2010 02:53:23,340 [hz.ServiceThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] false Handling
JoinInfo{request=true, memberCount=1 JoinRequest{nodeType=MEMBER,
address=Address[127.0.0.1:5703], groupName='dev', groupPassword='dev-
pass', buildNumber='20101105', packetVersion='5'}}
22 Nov 2010 02:53:23,340 [hz.ServiceThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] MEMBERS UPDATE!!
22 Nov 2010 02:53:23,340 [hz.ServiceThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] ClusterManager adding
Member [127.0.0.1:5701] this
22 Nov 2010 02:53:23,340 [hz.ServiceThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] ClusterManager adding
Member [127.0.0.1:5702]
22 Nov 2010 02:53:23,340 [hz.ServiceThread] DEBUG
com.hazelcast.cluster.ClusterManager - [dev] ClusterManager adding
Member [127.0.0.1:5703]
22 Nov 2010 02:53:23,340 [hz.ServiceThread] INFO
com.hazelcast.cluster.ClusterManager - [dev]

Members [3] {
Member [127.0.0.1:5701] this
Member [127.0.0.1:5702]
Member [127.0.0.1:5703]
}

22 Nov 2010 02:53:23,387
[hz.executor._hzInstance_0_dev.cached.thread-60] ERROR
com.hazelcast.nio.AbstractSerializer - com.acme.request
java.lang.ClassNotFoundException: com.acme.request
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
com.hazelcast.nio.AbstractSerializer.classForName(AbstractSerializer.java:
107)
at
com.hazelcast.nio.AbstractSerializer.classForName(AbstractSerializer.java:
92)
at com.hazelcast.nio.AbstractSerializer
$2.resolveClass(AbstractSerializer.java:392)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:
1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:
1496)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readUnshared(ObjectInputStream.java:441)
at com.hazelcast.nio.AbstractSerializer
$ObjectSerializer.readNormal(AbstractSerializer.java:444)
at com.hazelcast.nio.AbstractSerializer
$ObjectSerializer.read(AbstractSerializer.java:414)
at
com.hazelcast.nio.AbstractSerializer.toObject(AbstractSerializer.java:
143)
at com.hazelcast.nio.Serializer.readObject(Serializer.java:104)
at com.hazelcast.impl.ThreadContext.toObject(ThreadContext.java:89)
at com.hazelcast.nio.IOUtil.toObject(IOUtil.java:86)
at
com.hazelcast.impl.base.DataRecordEntry.getValue(DataRecordEntry.java:
158)
at com.hazelcast.impl.ConcurrentMapManager
$StoreAwareOperationHandler.execute(ConcurrentMapManager.java:1846)
at com.hazelcast.impl.ConcurrentMapManager
$OrderedExecutionTask.run(ConcurrentMapManager.java:2132)
at com.hazelcast.impl.executor.ParallelExecutorService
$ParallelExecutorImpl
$ExecutionSegment.run(ParallelExecutorService.java:136)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
22 Nov 2010 02:53:23,387
[hz.executor._hzInstance_0_dev.cached.thread-60] DEBUG
com.hazelcast.impl.ConcurrentMapManager - [dev] Store thrown
exception for CONCURRENT_MAP_MERGE
java.lang.RuntimeException: java.lang.ClassNotFoundException:
com.acme.request
at
com.hazelcast.nio.AbstractSerializer.toObject(AbstractSerializer.java:
146)
at com.hazelcast.nio.Serializer.readObject(Serializer.java:104)
at com.hazelcast.impl.ThreadContext.toObject(ThreadContext.java:89)
at com.hazelcast.nio.IOUtil.toObject(IOUtil.java:86)
at
com.hazelcast.impl.base.DataRecordEntry.getValue(DataRecordEntry.java:
158)
at com.hazelcast.impl.ConcurrentMapManager
$StoreAwareOperationHandler.execute(ConcurrentMapManager.java:1846)
at com.hazelcast.impl.ConcurrentMapManager
$OrderedExecutionTask.run(ConcurrentMapManager.java:2132)
at com.hazelcast.impl.executor.ParallelExecutorService
$ParallelExecutorImpl
$ExecutionSegment.run(ParallelExecutorService.java:136)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.ClassNotFoundException: com.acme.request
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
com.hazelcast.nio.AbstractSerializer.classForName(AbstractSerializer.java:
107)
at
com.hazelcast.nio.AbstractSerializer.classForName(AbstractSerializer.java:
92)
at com.hazelcast.nio.AbstractSerializer
$2.resolveClass(AbstractSerializer.java:392)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:
1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:
1496)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readUnshared(ObjectInputStream.java:441)
at com.hazelcast.nio.AbstractSerializer
$ObjectSerializer.readNormal(AbstractSerializer.java:444)
at com.hazelcast.nio.AbstractSerializer
$ObjectSerializer.read(AbstractSerializer.java:414)
at
com.hazelcast.nio.AbstractSerializer.toObject(AbstractSerializer.java:
143)
... 10 more

This happens in both 1.9 and 1.9.1-SNAPSHOT.

Regards,
Shahar

[Talip Ozturk]

Shahar,

I just got this fixed and created a new snapshot so that you can
download and try out.
http://www.hazelcast.com/downloads.jsp

http://twitter.com/oztalip

> --
> You received this message because you are subscribed to the Google Groups "Hazelcast" group.
> To post to this group, send email to haze...@googlegroups.com.
> To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.
>
>

[Keidi]

Hi Talip,

Sorry for the late response. I was on holidays and now we are
encountering much bigger problems with HazelCast (a separate thread
will be opened shortly).
I will be able to check this issue only after the more critical issues
are solved.

Thanks,
Shahar

On Nov 25 2010, 11:41 pm, Talip Ozturk <ta...@hazelcast.com> wrote:
> Shahar,
>
> I just got this fixed and created a new snapshot so that you can

> download and try out.http://www.hazelcast.com/downloads.jsp

> ...
>
> read more »