How hazelcast guarantee only one semaphore in the cluster?

568 views
Skip to first unread message

xinzh...@gmail.com

unread,
Aug 4, 2013, 11:06:43 PM8/4/13
to haze...@googlegroups.com

Hi all,

 

I just did a test about the hazelcast cluster semaphore:

I start 2 hazelcast instances with the same configuration file:

                <semaphore name="SMSBINDS">

                                <initial-permits>1</initial-permits>

                </semaphore>

 

And I start 2 HzClient, which will loop to get the semaphore, so the 1st started HzClient will get the semaphore and the other is looping.

However, when I stop one hazelcast instance(the 2nd one in my test), there will be an extra semaphore in the hazelcast cluster.

So the 2nd started HzClient can get the semaphore before the 1st started HzClient release the semaphore.

 

So I think, the 1nd hazelcast instance create a semaphore when the 2nd instance stop, that’s why the 2nd started HzClient can get a semaphore.

I doubt if the hazelcast can guarantee only one semaphore in the cluster, of maybe we need to use Lock instead. Thank you for your comments!

 

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 

-----------------------------------------------1st hazelcast instance log when 2nd instance stop

WARNING: [10.36.98.156]:5701 [local] Removing connection to endpoint Address[10.36.98.156]:5702 Cause => java.net.ConnectException {Connection refused: connect}, Error-Count: 5

Jul 24, 2013 2:29:50 PM com.hazelcast.cluster.ClusterManager

INFO: [10.36.98.156]:5701 [local] Removing Address Address[10.36.98.156]:5702

Jul 24, 2013 2:29:50 PM com.hazelcast.impl.PartitionManager

INFO: [10.36.98.156]:5701 [local] Starting to send partition replica diffs...true

Jul 24, 2013 2:29:50 PM com.hazelcast.cluster.ClusterManager

INFO: [10.36.98.156]:5701 [local]

 

Members [1] {

                Member [10.36.98.156]:5701 this

}

 

 

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 

-------------------------------------------------1st started HzClient

Logger>>Wed Jul 24 14:29:34 CST 2013        com.hazelcast.client.HazelcastClient@42704baa

Logger>>Wed Jul 24 14:29:34 CST 2013        Cluster [2] {

                Member [10.36.98.156]:5701

                Member [10.36.98.156]:5702

}

 

Created SEMAPHORE,4:SMSBINDS

Logger>>Wed Jul 24 14:29:35 CST 2013        Lock status: SMSBINDS, attached:0 available:1

Logger>>Wed Jul 24 14:29:35 CST 2013        Got Semaphore: SMSBINDS

Jul 24, 2013 2:29:46 PM com.hazelcast.client.ConnectionManager

WARNING: Connection to Connection [0] [CNBJADM0026L.int.asurion.com/10.36.98.156:5702 -> 10.36.98.156:5702] is lost

Jul 24, 2013 2:29:46 PM com.hazelcast.client.LifecycleServiceClientImpl

INFO: HazelcastClient is CLIENT_CONNECTION_LOST

Jul 24, 2013 2:29:46 PM com.hazelcast.client.LifecycleServiceClientImpl

INFO: HazelcastClient is CLIENT_CONNECTION_OPENING

Jul 24, 2013 2:29:47 PM com.hazelcast.client.LifecycleServiceClientImpl

INFO: HazelcastClient is CLIENT_CONNECTION_OPENED

 

 

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 

-------------------------------------------------2nd started HzClient

Logger>>Wed Jul 24 14:29:40 CST 2013        com.hazelcast.client.HazelcastClient@4139eeda

Logger>>Wed Jul 24 14:29:40 CST 2013        Cluster [2] {

                Member [10.36.98.156]:5701

                Member [10.36.98.156]:5702

}

 

Logger>>Wed Jul 24 14:29:40 CST 2013        Lock status: SMSBINDS, attached:1 available:0

Logger>>Wed Jul 24 14:29:40 CST 2013        Can't get Semaphore, sleep 3s.

Logger>>Wed Jul 24 14:29:43 CST 2013        Can't get Semaphore, sleep 3s.

Jul 24, 2013 2:29:46 PM com.hazelcast.client.ConnectionManager

WARNING: Connection to Connection [0] [CNBJADM0026L.int.asurion.com/10.36.98.156:5702 -> 10.36.98.156:5702] is lost

Jul 24, 2013 2:29:46 PM com.hazelcast.client.LifecycleServiceClientImpl

INFO: HazelcastClient is CLIENT_CONNECTION_LOST

Jul 24, 2013 2:29:46 PM com.hazelcast.client.LifecycleServiceClientImpl

INFO: HazelcastClient is CLIENT_CONNECTION_OPENING

Jul 24, 2013 2:29:47 PM com.hazelcast.client.LifecycleServiceClientImpl

INFO: HazelcastClient is CLIENT_CONNECTION_OPENED

Logger>>Wed Jul 24 14:29:51 CST 2013        Got Semaphore: SMSBINDS

 

Regards,

Austin

Peter Veentjer

unread,
Aug 5, 2013, 1:53:25 AM8/5/13
to haze...@googlegroups.com
Hi Austin,

which version of Hazelcast are you using?

And can you create a test that reproduces this problem so we can investigate it.

Peter.


--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Austin Cao

unread,
Dec 25, 2013, 5:30:06 AM12/25/13
to haze...@googlegroups.com
Hi Peter,

Very sorry for have you waiting so long a time!

I've just attached 2 classes and 1 config, for reproducing this issue, you can run 2 HzServer and then start 2 HzClient.
For now, the 1st HzClient will get the semaphore and the 2nd HzClient will loop to try to acquire the semaphore.
Here is the log from the 1st HzClient
Logger>>Wed Dec 25 18:17:47 CST 2013 Semaphore status: SMSBINDS, attached:0 available:1
Logger>>Wed Dec 25 18:17:47 CST 2013 Got Semaphore: SMSBINDS, attached:1 available:0
Dec 25, 2013 6:18:08 PM com.hazelcast.client.ConnectionManager
WARNING: Connection to Connection [0] [localhost-web/127.0.0.1:5702 -> 127.0.0.1:5702] is lost
Dec 25, 2013 6:18:08 PM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_LOST
Dec 25, 2013 6:18:08 PM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_OPENING
Dec 25, 2013 6:18:09 PM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_OPENED
Logger>>Wed Dec 25 18:18:47 CST 2013 Release Semaphore: SMSBINDS, attached:2 available:1

You can stop the 2nd HzServer, and you will see log from the 2nd HzClient that it can get the semaphore:
Logger>>Wed Dec 25 18:17:58 CST 2013 Semaphore status: SMSBINDS, attached:1 available:0
Logger>>Wed Dec 25 18:17:59 CST 2013 Can't get Semaphore, sleep 3s.
Logger>>Wed Dec 25 18:18:03 CST 2013 Can't get Semaphore, sleep 3s.
Logger>>Wed Dec 25 18:18:07 CST 2013 Can't get Semaphore, sleep 3s.
Dec 25, 2013 6:18:08 PM com.hazelcast.client.ConnectionManager
WARNING: Connection to Connection [0] [localhost-web/127.0.0.1:5702 -> 127.0.0.1:5702] is lost
Dec 25, 2013 6:18:08 PM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_LOST
Dec 25, 2013 6:18:08 PM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_OPENING
Dec 25, 2013 6:18:09 PM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_OPENED
Logger>>Wed Dec 25 18:18:13 CST 2013 Got Semaphore: SMSBINDS, attached:1 available:0

------------------------------------------------------------
My target is to hold one semaphore/lock in the hazelcast cluster so that we can have several client that will compete to get it
As my test, seems my usage is not proper, please help shed some lights on it, thank you!

Regards,
Austin
HzClient.java
HzServer.java
hazelcast-cluster.xml

Austin Cao

unread,
Dec 25, 2013, 10:06:12 PM12/25/13
to haze...@googlegroups.com
Hi Peter,

I also test the "Lock" with the same case as "Semaphore", the test result is almost the same.
When the 2nd HzServer being turned off, the 2nd HzClient will get the the lock, which means 2 clients can hold the lock at the same time.

Please note that if you turn off 1st HzServer rather than 2nd HzServer, you won't get this test result, thank you!

R,
Austin

Austin Cao

unread,
Dec 26, 2013, 1:47:42 AM12/26/13
to haze...@googlegroups.com
Hi Peter,

I am using hazelcast 2.5. And I download the latest package (version 3.1.3), seems this issue is fixed.
However, I really want to know the story about this issue, thank you!

R,
Austin


On Monday, August 5, 2013 1:53:25 PM UTC+8, peter veentjer wrote:

Mehmet Dogan

unread,
Dec 26, 2013, 2:10:55 AM12/26/13
to haze...@googlegroups.com

That was a client design problem in version 2.x, only occurs when clients are involved and not easy to fix in 2.x architecture.

By 3.0 we redesigned nearly every aspect of server and client (even client will get a further improvement by 3.2). One of the design changes is related to resource (lock, semaphore..) ownership in cluster.

@mmdogan

~Sent from mobile

rick...@gmail.com

unread,
Jan 2, 2014, 3:44:01 AM1/2/14
to haze...@googlegroups.com
hi Mehmet,

Could you share any hints to us?
Thank you.

Is there any workaround?
Is there anything we can do to avoid or reduce the occurrence of such bug?
We cannot upgrade it from 2.x to 3.x shortly.

seems this issue and changeset?
this change?

Regards,
Rick
Reply all
Reply to author
Forward
0 new messages