Error occurred during initialization java.lang.OutOfMemoryError: unable to create new native thread
kkuma...@gmail.com
/apps/hadoop/2.7.2/bin/../libexec/hadoop-config.sh: fork: retry: Resource temporarily unavailable Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371) at java.lang.UNIXProcess.initStreams(UNIXProcess.java:226) at java.lang.UNIXProcess$3.run(UNIXProcess.java:199) at java.lang.UNIXProcess$3.run(UNIXProcess.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:197) at java.lang.ProcessImpl.start(ProcessImpl.java:130) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028) at org.apache.hadoop.util.Shell.runCommand(Shell.java:486) at org.apache.hadoop.util.Shell.run(Shell.java:456) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) at org.apache.hadoop.util.Shell.isSetsidSupported(Shell.java:391) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:381) at org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:440) at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:486) at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170) at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.fs.FsShell.main(FsShell.java:340) 2018-03-26 01:04:39.209 Will retry in 5 seconds to ensure that this is not a transient error Error occurred during initialization of VM java.lang.OutOfMemoryError: unable to create new native thread 2018-03-26 01:04:44.429 Will retry in 5 seconds to ensure that this is not a transient error # # There is insufficient memory for the Java Runtime Environment to continue. # Cannot create GC thread. Out of system resources.
######################################################################
below is some of the stats from our genie instance.
Total amount of memory available on the AWS EC2 instance where genie is running is 120GB.
[root@ip-xx-xxx-xxx-xxx security]# java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'
uintx AdaptivePermSizeWeight = 20 {product}
intx CompilerThreadStackSize = 0 {pd product}
uintx ErgoHeapSizeLimit = 0 {product}
uintx HeapSizePerGCThread = 87241520 {product}
uintx InitialHeapSize := 2015071232 {product}
uintx LargePageHeapSizeThreshold = 134217728 {product}
uintx MaxHeapSize := 32038191104 {product}
uintx MaxPermSize = 174063616 {pd product}
uintx PermSize = 21757952 {pd product}
intx ThreadStackSize = 1024 {pd product}
intx VMThreadStackSize = 1024 {pd product}
java version "1.7.0_111"
JAVA Heap Size - Initial Heap Size - 2015071232 bytes (2015mb) and MaxHeapSize 32038191104 bytes (32038 MB)
Number of threads the OS can create is ulimit -u = 491887
we don't think it is an issue with the number of threads.
do we need to decrease the heap size allocated to JVM, currently maximum number of jobs allowed to run
is set as com.netflix.genie.server.max.running.jobs=100.
how much amount of memory is required for each genie job to run?
can someone please let me know what changes do we need to do in order to make sure that we do not face out of memory issue.
Thanks,
Kiran
