Number of executor threads keep on growing

[Barry Lagerweij]

Currently when I use the Hazelcast executorService, a new Thread is created for each Task. I have the feeling that these threads are never stopped:

        threadPoolExecutor = new ThreadPoolExecutor(
                0, Integer.MAX_VALUE,
                60L,
                TimeUnit.SECONDS,
                new SynchronousQueue(),
                new ExecutorThreadFactory(node.threadGroup, node.getThreadPoolNamePrefix("cached"), classLoader),
                new RejectionHandler()) {
            protected void beforeExecute(Thread t, Runnable r) {
                ThreadContext threadContext = ThreadContext.get();
                threadContext.setCurrentFactory(node.factory);
                CallContext callContext = mapThreadCallContexts.get(t);
                if (callContext == null) {
                    callContext = new CallContext(threadContext.createNewThreadId(), false);
                    mapThreadCallContexts.put(t, callContext);
                }
                threadContext.setCallContext(callContext);
            }
        };

The ThreadPoolExecutor will create new threads, since the max pool is configured to Integer.MAX_VALUE.
Also, the keep-alive-seconds is completely ignored, and the ThreadPoolExecutor does not timeout after waiting for work (allowCoreThreadTimeOut defaults to false)

The result of this is a new Thread for each executed DistributedTask.... (!) Can someone else using the ExecutorService please verify by executing a few tasks, and then create a thread-dump ?

Barry

[Mehmet Dogan]

'keep-alive-seconds' defined in configuration is deprecated some time ago and it is not used anymore. Thread-pool 'keepAliveTime' is configured to  60 seconds (third parameter in ThreadPoolExecutor constructor). And 'core-pool-size' is zero (0). So, after 60 seconds idle time, all threads will be terminated. In 60 seconds period, alive and idle threads will be re-used. 

'allowCoreThreadTimeOut' is introduced in Java 6,  Hazelcast can not use that because it is Java 5 compatible. But also Hazelcast does not need that, because it is meaningful for thread-pools those have some core threads. Hazelcast's core thread count is zero. 'keepAliveTime' applies to all threads.

By using test below this behavior can be seen;

----------------------------------------------------------------------------------------------------------------------------------------

public static void main(String[] args) throws Exception {

        ExecutorService ex = Hazelcast.newHazelcastInstance(null).getExecutorService();

        for (int i = 0; i < 20; i++) {

            ex.execute(new DummyTask());

        }

        Thread.sleep(1000);

        com.hazelcast.impl.management.ThreadDumpGenerator tdg = com.hazelcast.impl.management.ThreadDumpGenerator.newInstance();

        System.out.println(tdg.dumpAllThreads());

        Thread.sleep(1000 * 70); // make sure all idle terminated

        System.out.println(tdg.dumpAllThreads());

    }

    static class DummyTask implements Runnable, Serializable {

        public void run() {

            try {

                Thread.sleep(5 * 1000);

            } catch (InterruptedException e) {

                e.printStackTrace();

            }

        }

    }

In the first dump there will be 20 threads named 'hz._hzInstance_1_dev.cached.thread-1' to 'hz._hzInstance_1_dev.cached.thread-20'.

In the second dump you should not see those threads (there may be one that not terminated because of Hazelcast internal tasks).

@mmdogan

Barry

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To post to this group, send email to haze...@googlegroups.com.
To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.

[Tim Peierls]

Since there's no way (is there?) to get programmatic access to the underlying ThreadPoolExecutor, those who want to throttle distributed task submission (to limit creation of threads, for example) have to do it themselves. I've fleshed out one way of doing this using Semaphore:

http://pastebin.com/PsYqV5DA 

I added some very basic tests for this class:

http://pastebin.com/xmBkyVC3 

Hazelcast team -- I know you're phasing out the "gotchas" page, so the fact that the ExecutorService instances returned by HazelcastInstance are not true ExecutorServices (they do not support invokeAll/Any and they don't have standard shutdown behavior) should be loudly announced in the javadocs.

--tim

[Fuad Malikov]

Thanks Tim, I have updated the Javadocs. 

-fuad