Tuning Hazelcast Map

[Amila De Silva]

Hi,

In one of my tests I'm using a Hazelcast map to keep a distributed map across 4 JVMs. The four JVMs handle about 500 transactions per second and for each transaction a put is done on the map.In a single JVM there are 500 threads writing to this map. At this TPS I'm seeing lot of threads going into TIMED_WAITING state. This is an excerpt from the stack trace;

"PassThroughMessageProcessor-398" - Thread t@668

   java.lang.Thread.State: TIMED_WAITING

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for <36adb967> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)

        at com.hazelcast.spi.impl.InvocationImpl$InvocationFuture.waitForResponse(InvocationImpl.java:323)

        at com.hazelcast.spi.impl.InvocationImpl$InvocationFuture.get(InvocationImpl.java:291)

        at com.hazelcast.spi.impl.InvocationImpl$InvocationFuture.get(InvocationImpl.java:283)

        at com.hazelcast.map.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:197)

        at com.hazelcast.map.proxy.MapProxySupport.putInternal(MapProxySupport.java:167)

        at com.hazelcast.map.proxy.MapProxyImpl.put(MapProxyImpl.java:71)

        at com.hazelcast.map.proxy.MapProxyImpl.put(MapProxyImpl.java:59)

Even after disabling backups the same behavior is there. Is this a normal behavior or is there a place to increase hazelcast threads so that I can reduce the number of Waiting threads?

I'm using 3.0.1.

Thanks

Amila

[Nick Pratt]

Why so many threads per JVM?

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/1f71938e-a471-4844-8e13-1cedd4cbb792%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Amila De Silva]

That many threads are needed to handle transactions. Put is done in an attempt to update a cache. Do you have a recommended number of threads per a JVM?

[Nick Pratt]

I'd start with 1 per CPU and then test from there.  Each thread has overhead - 500 is a lot.

N

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/d636e16c-a986-45f0-b791-286fb912ea3d%40googlegroups.com.

[Amila De Silva]

Thanks for the suggestion - that's one thing I could try.

When doing a profiling it showed com.hazelcast.spi.impl.NodeEngineImpl.send(com.hazelcast.nio.Packet, com.hazelcast.nio.Address, com.hazelcast.spi.impl.NodeEngineImpl$FutureSend) taking most of the time. Is this the part related to syncing data across the map? Is there a way to increase the number of threads executing this method?

[Nick Pratt]

It certainly looks like Hazelcast communication. As a general guideline, just increasing threads often doesn't translate into increased throughput - often it decreases it (due to increased context switching). While you can make educated/informed guesses on the number (one thread per core is a good starting point) the optimal figure for your application can only be found through load testing a known revision on known hardware. Code changes, jdk changes and hardware changes all have big impacts on multi thread app performance.  This implies a new round of load/performance testing ideally at every single build and every time hardware changes.  Don't overlook environmental factors, especially with networked apps.

What's your goal here (puts per second)?
Are you using distributed/XA transactions?
What's your network speed / connectivity?
What's your HZ map config?
What are the machine specs for each node?
What JVM are you using?
How big are the objects you are putting in the map?

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/2dcf693a-9d03-415c-8d5a-852629461aa0%40googlegroups.com.