Change number of threads for Hibernate 2nd level cache

[Barry Lagerweij]

Hello,

How do I configure the number of threads which will be used by the Hibernate cache provider ? Currently, Hazelcast starts around 100 threads for the cache:

3XMTHREADINFO      "hz.1.threads._hzInstance_1_eframe-test.cached.thread-97" (TID:0x0AD1FF00, sys_thread_t:0x0AD05178, state:P, native ID:0x0000344E) prio=5
(100 times)
3XMTHREADINFO      "hz.1.threads._hzInstance_1_eframe-test.cached.thread-188" (TID:0x09F1EE00, sys_thread_t:0x0A189DA8, state:P, native ID:0x000060A4) prio=5
3XMTHREADINFO      "hz.1.threads._hzInstance_1_eframe-test.cached.thread-190" (TID:0x0931A900, sys_thread_t:0x0933C008, state:P, native ID:0x00004569) prio=5

We can tune the number of threads for the Executor, e.g.:
       <property name="hazelcast.executor.client.thread.count">10</property>
       <property name="hazelcast.executor.event.thread.count">2</property>
       <property name="hazelcast.executor.migration.thread.count">5</property>
       <property name="hazelcast.executor.query.thread.count">2</property>
       <property name="hazelcast.executor.store.thread.count">10</property>      

Are there similar properties for the 2nd level cache ?

Barry

[Mehmet Dogan]

Hi,

Hazelcast internal executor threads are used for operations like map store/load, event or client requests processing. Member communications are done in IO threads and data operations are done in service thread.

Hibernate cache provider operations are synchronous operations such as get, put, remove, lock and nearly all done in user/application threads.

http://hazelcast.com/docs/1.9.4/manual/multi_html/ch15.html#InternalsThreads

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To post to this group, send email to haze...@googlegroups.com.
To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.

[Barry Lagerweij]

Thanks for the information, but it has not really answered my question.

If I read your e-mail, you're saying that 'Hibernate cache provider operations are synchronous operations such as get, put, remove, lock and nearly all done in user/application threads.'
This does not explain why I'm seeing so many threads called 'hz.1.threads._hzInstance_1_eframe-test.cached.thread-97' etc.

Are you saying that each Hibernate region (entity, collection) results in a thread ? And that there is nothing we can do to reduce this ?

With kind regards,

Barry

[Mehmet Dogan]

Threads containing "cached" in their names are Hazelcast executor pool threads, pool grows as more threads are required. Those threads have 60 seconds max-idle time.  

>> "Are you saying that each Hibernate region (entity, collection) results in a thread ? And that there is nothing we can do to reduce this ?"

No, each region does not result in a separate thread. An operation is started to be processed in app thread, flow goes into service thread and/or IO threads and then result is passed to app thread back. Some operations like connection handling, member pinging, event firing, cleanups are done in executor threads.

@mmdogan

[Barry Lagerweij]

Even when a system is completely idle, I still see about 50 threads with names like 'hz.1.threads._hzInstance_1_clustername.cached.thread-1355'. All of these threads have the following stacktrace:

at sun/misc/Unsafe.park(Native Method)
at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:197(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:817(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1146(Compiled Code))
at java/util/concurrent/SynchronousQueue$Node.waitForPut(SynchronousQueue.java:306(Compiled Code))
at java/util/concurrent/SynchronousQueue.poll(SynchronousQueue.java:458(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:499(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:702(Compiled Code))
at java/lang/Thread.run(Thread.java:811(Compiled Code))

Is there anything we can do to reduce the number of threads ?

With kind regards,

Barry

[Mehmet Dogan]

I forgot to ask, which version are you using? Because executor thread pool management part got some important changes recently.

[Barry Lagerweij]

We are using version 1.9.4.3, in which version were those changes to threadpool management introduced?

Barry

[Mehmet Dogan]

It was 1.9.4.3 and version 1.9.4.4 has some fixes on new threadpool. Are you using any type of eviction, I mean max-size, ttl, max-idle etc?

[Barry Lagerweij]

We tried again using 1.9.4.4, and it seems to be fixed now. (just a few threads)

Thanks!

Barry

[Mehmet Dogan]

Good to hear that!

@mmdogan

[Barry Lagerweij]

It seems that the issue with the number of threads is not completely solved: After running for a few days, the performance of the Hibernate 2nd level caches degrades, and becomes unusable after a while. CPU is at a few percent utilization, heap is about 47% used. Also, the number of threads for the cache seems to increase to about 50 threads after a day or two.

I'm not 100% sure that the performance decrease is caused by the high number of 2nd level cache threads, but it's odd to see this number increase, especially after a night of complete inactivity (this is an internal test system). I expected the threadpool to free these unused threads after a certain timeout of inactivity...

All the (idle?) threads have exactly identical stacks:
3XMTHREADINFO      "hz.1.threads._hzInstance_1_eFrameDevCluster.cached.thread-12610" (TID:0x0000002C6B702100, sys_thread_t:0x0000002C6B7010D0, state:P, native ID:0x000000000000114D) prio=5
4XESTACKTRACE          at sun/misc/Unsafe.park(Native Method)
4XESTACKTRACE          at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:197(Compiled Code))
4XESTACKTRACE          at java/util/concurrent/locks/AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:817(Compiled Code))
4XESTACKTRACE          at java/util/concurrent/locks/AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1146(Compiled Code))
4XESTACKTRACE          at java/util/concurrent/SynchronousQueue$Node.waitForPut(SynchronousQueue.java:306(Compiled Code))
4XESTACKTRACE          at java/util/concurrent/SynchronousQueue.poll(SynchronousQueue.java:458(Compiled Code))
4XESTACKTRACE          at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:499(Compiled Code))
4XESTACKTRACE          at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:702(Compiled Code))
4XESTACKTRACE          at java/lang/Thread.run(Thread.java:811(Compiled Code))

This is on an IBM JDK: J2RE 5.0 IBM J9 2.3 Linux amd64-64 build j9vmxa6423-20090707, running on 64-bit Linux 2.6.9-89.0.18.ELsmp

The machine is running IBM WebSphere 6.1, other web-applications are using ehcache as 2nd level cache, and the performance of these webapps do not seem to be affected. Restarting the web-application seems to solve the issue for a while, but we would like to get this fixed.

Any clues ?

Barry