Outofmemory when initializing client in 3.6.4

[Hari M S]

Aug 16, 2016 11:44:29 AM com.hazelcast.core.LifecycleService

INFO: HazelcastClient[hz.client_0_dev][3.6.4] is CLIENT_CONNECTED

Exception in thread "Thread-1" java.lang.OutOfMemoryError: unable to create new native thread

        at java.lang.Thread.start0(Native Method)

        at java.lang.Thread.start(Thread.java:640)

        at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)

        at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706)

        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650)

        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:78)

        at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:599)

        at com.hazelcast.client.spi.impl.listener.ClientSmartListenerService.init(ClientSmartListenerService.java:197)

        at com.hazelcast.client.spi.impl.ClientClusterServiceImpl.initMembershipListener(ClientClusterServiceImpl.java:181)

        at com.hazelcast.client.spi.impl.ClientClusterServiceImpl.addMembershipListener(ClientClusterServiceImpl.java:164)

        at com.hazelcast.client.spi.impl.listener.ClientSmartListenerService.start(ClientSmartListenerService.java:146)

        at com.hazelcast.client.impl.HazelcastClientInstanceImpl.start(HazelcastClientInstanceImpl.java:313)

        at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:86)

        at com.newbreak.common.server.HazelCastProvider._initialize(HazelCastProvider.java:60)

my client code to initialize client :

ClientConfig clientConfig = new ClientConfig();

clientConfig.getNetworkConfig().setConnectionAttemptLimit(Integer.MAX_VALUE);

clientConfig.getNetworkConfig().setConnectionTimeout(5000);

clientConfig.getNetworkConfig().addAddress(host);

h = HazelcastClient.newHazelcastClient(clientConfig); <-- this is line 60 shown above

I dont see this on hazelcast 3.6, but only in 3.6.4

[Noctarius]

Hey Hari,

This is not a real OOME but means the user account running your app ran out of native file handles (in Linux basically everything is a file, including threads). That’s the reason that the message is “unable to create new native thread”.

To see how to fix your problem see our FAQ section in the documentation:

http://docs.hazelcast.org/docs/3.6/manual/html-single/index.html#how-to-get-rid-of-java-lang-outofmemoryerror-unable-to-create-new-native-thread-

Chris

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/1bd49309-37a0-4a30-8e51-bdc95ee9305a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hari M S]

thanks Chris. I will make changes to fix this.

what changed between 3.5 - 3.6.4 to require a higher open file limit ?

thanks

hari

[Noctarius]

Hey Hari,

The whole client architecture was rewritten for better throughput and the new client protocol. On the other hand we also have a new way on how we utilize threads :-)

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/da65319a-5cd2-4b2e-b05c-9d355c772da1%40googlegroups.com.

[Hari M S]

Thanks Chris. I made changes, but still see the same issue

ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 120584

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 16383

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 8182

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

thanks

hari

[ih...@hazelcast.com]

Can you also write output of this command: 

lsof | wc -l

it will show number of opened files.