java.lang.OutOfMemoryError: unable to create new native thread

[Sebastien Blind]

Hello - seeing this exception in hazelcast 3.2.1 and then the server crashes

java.lang.OutOfMemoryError: unable to create new native thread

        at java.lang.Thread.start0(Native Method)

        at java.lang.Thread.start(Thread.java:640)

        at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)

        at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706)

        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650)

        at com.hazelcast.client.ClientEngineImpl.handlePacket(ClientEngineImpl.java:119)

        at com.hazelcast.nio.NodeIOService.handleClientPacket(NodeIOService.java:103)

        at com.hazelcast.nio.SocketClientDataReader.read(SocketClientDataReader.java:52)

        at com.hazelcast.nio.ReadHandler.handle(ReadHandler.java:72)

        at com.hazelcast.nio.InSelectorImpl.handleSelectionKey(InSelectorImpl.java:33)

        at com.hazelcast.nio.AbstractIOSelector.run(AbstractIOSelector.java:126)

stack dumps shows (at least the one i captured before the crash):

232 times com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59)

2 times com.hazelcast.management.ManagementCenterService$StateSendThread.sleep(ManagementCenterService.java:376)

1 time  com.hazelcast.util.HealthMonitor.run(HealthMonitor.java:104)

3 times com.hazelcast.util.executor.SingleExecutorThreadFactory$ManagedThread.run(SingleExecutorThreadFactory.java:45)

1 time com.hazelcast.nio.SocketAcceptor.run(SocketAcceptor.java:49)

6 time com.hazelcast.nio.AbstractIOSelector.run(AbstractIOSelector.java:112)

48 times com.hazelcast.spi.impl.BasicOperationScheduler$PartitionThread.doRun(BasicOperationScheduler.java:257)

1 time com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.doRun(InternalPartitionServiceImpl.java:1479)

1 time com.hazelcast.spi.impl.WaitNotifyServiceImpl$ExpirationTask.run(WaitNotifyServiceImpl.java:375)

There are maybe 40 or so maps and a couple of queues

Adding  "-Dhazelcast.operation.thread.count=50" or  -Xss164k didn't help.

There is only one other process in the cluster. It stays alive and it has about 100 live threads. 

If i start another server process on a separate (but similar) machine similar to the process that crashes, it stays up. At the same time, there are a couple of java clients connected to the cluster

The first time it occurred was while loading data (one entry at a time in a map of less than 10k entries). Then I am not able to bring the server back

Any thoughts/extra info needed?

Thanks in advance,

Sebastien

[noctarius]

Hi Sebastien,

This normally is an issue on Linux with file limits. Normally users have a max soft and hard limit for open files.

If you Google for that exception (especially with the native thread) you should be able to find out how to change those limits.

Actually I'm on my mobile so sorry I can't offer the page and believe me the OOME is totally misleading here.

Chris

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/7472aee7-2f21-44fe-addf-820e85c8d33c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[sebastien]

file limits appear to be pretty high on the box

[javuser@chi-uat-app01 bin]$ ulimit -a | grep file

core file size          (blocks, -c) 0

file size               (blocks, -f) unlimited

open files                      (-n) 65535

file locks                      (-x) unlimited

and lsof -a -p <hazelcastpid> | wc

goes up to ~250 before process dies

(and lsof returns ~3k entries)

Thanks,

Sebastien

[Noctarius]

Is it the account the server runs with?

[sebastien]

yes - the hazelcast process run w/ javuser unix id

[sebastien]

the crash behavior doesn't seem to happen w/ hazelcast 3.1.2 - we are downgrading from 3.2.1 to 3.1.2 for now

[Jaromir Hamala]

Hi Sebastien,

just to double check: Are you using 64b Java? What's your Xmx setting?

Cheers,

Jaromir

[sebastien]

JVM details below, and settings are -Xms1024m -Xmx1024m -XX:MaxPermSize=128m - we bumped the settings up to 2gb to try with no better end result (visualvm was showing a usage of less than 400mb until process crashed)

java version "1.6.0_45"

Java(TM) SE Runtime Environment (build 1.6.0_45-b06)

Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

[noctarius]

As I said before "unable to create native thread" means in 99% of the cases that the operating system rejected creating a new thread. This is normally due to limitations. So if you're not running as root check limits on the corresponding user account.

Chris

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/dfecc267-408f-44bc-a1de-6e63db629be5%40googlegroups.com.