ILock.destroy and ISemaphore.destroy exceptions on 2.5

101 views
Skip to first unread message

trey.hu...@gmail.com

unread,
Jul 16, 2013, 3:33:56 PM7/16/13
to haze...@googlegroups.com
I am experiencing very odd behavior when invoking destroy on either ILock or ISemaphore on Hazelcast 2.5. Here is my sequence:

In a DistributedTask (call it Coordinator), I call HazelcastInstance.getLock() and HazeclastInstance.getSemaphore. The Coordinator unlocks the lock, and releases one permit on the semaphore.

In another DistributedTask (call it Participant), I get local copies of the same lock and semaphore using the same names. The participant acquires the lock unlocked from the coordinator and acquires the semaphore. 

When everything is done, back in the Coordinator, I attempt to dispose of the acquired resources. Calling destroy on either the lock or the semaphore yields several but different exceptions. This is not an execution sequence issues. I can produce the exact same results when executing within a single thread on a single hazelcast node. 

Below is what I get when calling ILock.destroy():
// ILock.destroy()
on hazelcast node:
java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333)
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:988)
at com.hazelcast.impl.FactoryImpl.getOrCreateProxyByName(FactoryImpl.java:566)
at com.hazelcast.impl.ClientHandlerService$DestroyHandler.processCall(ClientHandlerService.java:366)
at com.hazelcast.impl.ClientHandlerService$ClientOperationHandler.handle(ClientHandlerService.java:1574)
at com.hazelcast.impl.ClientRequestHandler$1.run(ClientRequestHandler.java:57)
at com.hazelcast.impl.ClientRequestHandler$1.run(ClientRequestHandler.java:54)
at com.hazelcast.impl.ClientRequestHandler.doRun(ClientRequestHandler.java:63)
at com.hazelcast.impl.FallThroughRunnable.run(FallThroughRunnable.java:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)


on jboss:
java.lang.RuntimeException
at com.hazelcast.impl.ClientServiceException.readData(ClientServiceException.java:63)
at com.hazelcast.nio.Serializer$DataSerializer.read(Serializer.java:104)
at com.hazelcast.nio.Serializer$DataSerializer.read(Serializer.java:79)
at com.hazelcast.nio.AbstractSerializer.toObject(AbstractSerializer.java:121)
at com.hazelcast.nio.AbstractSerializer.toObject(AbstractSerializer.java:156)
at com.hazelcast.client.ClientThreadContext.toObject(ClientThreadContext.java:72)
at com.hazelcast.client.IOUtil.toObject(IOUtil.java:34)
at com.hazelcast.client.ProxyHelper.getValue(ProxyHelper.java:186)
at com.hazelcast.client.ProxyHelper.doOp(ProxyHelper.java:146)
at com.hazelcast.client.ProxyHelper.doOp(ProxyHelper.java:140)
at com.hazelcast.client.ProxyHelper.destroy(ProxyHelper.java:197)
at com.hazelcast.client.LockClientProxy.destroy(LockClientProxy.java:92)


And below is what I get when I call ISemaphore.destroy():
// ISemaphore.destroy()

// hazelcast node
Jul 16, 2013 2:19:50 PM com.hazelcast.cluster.ClusterService
SEVERE: [localhost]:5701 [dev-local] error processing messages  processable=com.hazelcast.cluster.ClusterService$1@69c41ba8
java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1144)
at com.hazelcast.impl.FactoryImpl.createProxy(FactoryImpl.java:785)
at com.hazelcast.impl.FactoryImpl$8.process(FactoryImpl.java:840)
at com.hazelcast.cluster.ClusterService$1.process(ClusterService.java:112)
at com.hazelcast.cluster.ClusterService.processProcessable(ClusterService.java:180)
at com.hazelcast.cluster.ClusterService.dequeueProcessables(ClusterService.java:245)
at com.hazelcast.cluster.ClusterService.run(ClusterService.java:191)
at java.lang.Thread.run(Thread.java:722)

Jul 16, 2013 2:19:50 PM com.hazelcast.cluster.ClusterService
SEVERE: [localhost]:5701 [dev-local] null
java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1144)
at com.hazelcast.impl.FactoryImpl.createProxy(FactoryImpl.java:785)
at com.hazelcast.impl.FactoryImpl$8.process(FactoryImpl.java:840)
at com.hazelcast.cluster.ClusterService$1.process(ClusterService.java:112)
at com.hazelcast.cluster.ClusterService.processProcessable(ClusterService.java:180)
at com.hazelcast.cluster.ClusterService.dequeueProcessables(ClusterService.java:245)
at com.hazelcast.cluster.ClusterService.run(ClusterService.java:191)
at java.lang.Thread.run(Thread.java:722)


// jboss 
14:20:00,346 INFO  [com.hazelcast.client.ProxyHelper] (http--0_0_0_0_0_0_0_0-8180-6) There is no response for Call [11] operation=DESTROY in 5 seconds.
14:20:05,346 INFO  [com.hazelcast.client.ProxyHelper] (http--0_0_0_0_0_0_0_0-8180-6) There is no response for Call [11] operation=DESTROY in 10 seconds.
14:20:10,347 INFO  [com.hazelcast.client.ProxyHelper] (http--0_0_0_0_0_0_0_0-8180-6) There is no response for Call [11] operation=DESTROY in 15 seconds.
14:20:15,347 INFO  [com.hazelcast.client.ProxyHelper] (http--0_0_0_0_0_0_0_0-8180-6) There is no response for Call [11] operation=DESTROY in 20 seconds.



Peter Veentjer

unread,
Jul 16, 2013, 3:39:08 PM7/16/13
to haze...@googlegroups.com
Can you give us the code that reproduces this problem?

It might have to do with a bug Mehmet solved today where a lock with a null id was causing problems (and then you get this NPE). 



--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

trey.hu...@gmail.com

unread,
Jul 16, 2013, 4:36:36 PM7/16/13
to haze...@googlegroups.com
Unfortunately my company will not allow me to share the code. That policy can be quite a pain sometimes.

I've tried to distill the code into its simplest pieces and execute it in just a normal java main, and I cannot reproduce the problem. This only happens after a node is up and running for a bit. I'll try to research it more. 

trey.hu...@gmail.com

unread,
Jul 16, 2013, 6:45:43 PM7/16/13
to haze...@googlegroups.com
And... I figured it out. I was executing the destroy method on ILock and ISemaphore from a HazelcastClient (from jboss), not from within the actual hazelcast node itself. I moved the code to inside of a distributed task executed on the datagrid itself, and the problem went away.

Peter Veentjer

unread,
Jul 17, 2013, 12:46:52 AM7/17/13
to haze...@googlegroups.com
I managed to reproduce the problem:

public class Foo {
    public static void main(String[] args){
        HazelcastInstance hz = Hazelcast.newHazelcastInstance();
//        hz.getLock("lock");
        hz.getSemaphore("s");
        HazelcastInstance client = HazelcastClient.newHazelcastClient(new ClientConfig());
//        client.getLock("lock").destroy();
        client.getSemaphore("s").destroy();
        System.exit(0);
    }
}

It fails with an NPE with semaphore and with lock. 

We should have a fix available today, but it will be in the 2.6.1-SNAPSHOT branch.


On Wed, Jul 17, 2013 at 1:45 AM, <trey.hu...@gmail.com> wrote:
And... I figured it out. I was executing the destroy method on ILock and ISemaphore from a HazelcastClient (from jboss), not from within the actual hazelcast node itself. I moved the code to inside of a distributed task executed on the datagrid itself, and the problem went away.

--

Mehmet Dogan

unread,
Jul 17, 2013, 2:05:53 AM7/17/13
to haze...@googlegroups.com
Thanks for the report. Issue is now fixed in master branch and fix will be available in 2.6.1 patch release.


@mmdogan

trey.hu...@gmail.com

unread,
Jul 17, 2013, 8:24:58 AM7/17/13
to haze...@googlegroups.com
Wow, that was fast. Thanks guys. 
Reply all
Reply to author
Forward
0 new messages