java vm out of memery ，java.lang.OutOfMemoryError: unable to create native thread

9,746 views
Skip to first unread message

dong liang

unread,
Jun 10, 2019, 10:30:01 AM6/10/19
to Android Building

when i compile  with java code ，i found some error like this ：


[warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
[ 90% 26803/29520] //hardware/interfaces/contexthub/1.0:android.hardware.contexthub-V1.0-java javac [common]
FAILED: out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/android.hardware.contexthub-V1.0-java.jar
rm -rf "out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/classes" "out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/anno" "out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/srcjars" && mkdir -p "out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/classes" "out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/anno" "out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/srcjars" && out/soong/host/linux-x86/bin/zipsync -d out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/srcjars -l out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/srcjars/list -f "*.java" out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java_gen_java/gen/srcs.srcjar && (if [ -s out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/srcjars/list ] || [ -s out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/android.hardware.contexthub-V1.0-java.jar.rsp ] ; then out/soong/host/linux-x86/bin/soong_javac_wrapper prebuilts/jdk/jdk9/linux-x86/bin/javac -J-Xmx2048M -Xmaxerrs 9999999 -encoding UTF-8 -sourcepath "" -g -XDskipDuplicateBridges=true -XDstringConcat=inline  -proc:none  -bootclasspath out/soong/.intermediates/libcore/core.current.stubs/android_common/turbine-combined/core.current.stubs.jar:out/soong/.intermediates/libcore/core-lambda-stubs/android_common/turbine-combined/core-lambda-stubs.jar -classpath out/soong/.intermediates/frameworks/base/hwbinder.stubs/android_common/turbine-combined/hwbinder.stubs.jar:out/soong/.intermediates/system/libhidl/transport/base/1.0/android.hidl.base-V1.0-java/android_common/turbine-combined/android.hidl.base-V1.0-java.jar -source 1.8 -target 1.8 -d out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/classes -s out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/anno @out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/android.hardware.contexthub-V1.0-java.jar.rsp @out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/srcjars/list ; fi ) && out/soong/host/linux-x86/bin/soong_zip -jar -o out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/android.hardware.contexthub-V1.0-java.jar -C out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/classes -D out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/classes && rm -rf "out/soong/.intermediates/hardware/interfaces/contexthub/1.0/android.hardware.contexthub-V1.0-java/android_common/javac/srcjars"
[0.585s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached


please help me，tks 

the memery is 500G ：

free -g :
 total        used        free      shared  buff/cache   available
Mem:            503           2         388           0         112         499
Swap:            46           0          46

uname -r ：
4.4.0-131-generic

ulimit  -a ：
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2062255
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 102400
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2062255
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


error pid ：
hs_err_pid79581.log ：



---------------  T H R E A D  ---------------

Current thread (0x0000000001a0c000):  JavaThread "Unknown thread" [_thread_in_vm, id=79581, stack(0x00007fffe9486000,0x00007fffe9586000)]

Stack: [0x00007fffe9486000,0x00007fffe9586000],  sp=0x00007fffe957a6f0,  free space=977k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xc5d97c]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x15c
V  [libjvm.so+0xc5e653]  VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, __va_list_tag*)+0x43
V  [libjvm.so+0x639778]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0xe8
V  [libjvm.so+0xc85f69]  AbstractWorkGang::add_workers(unsigned int, bool)+0x209
V  [libjvm.so+0xc86036]  AbstractWorkGang::initialize_workers()+0x96
V  [libjvm.so+0x6f5578]  G1CollectedHeap::G1CollectedHeap(G1CollectorPolicy*)+0x558
V  [libjvm.so+0xc260c4]  Universe::create_heap()+0x1c4
V  [libjvm.so+0xc2648d]  Universe::initialize_heap()+0x39d
V  [libjvm.so+0xc265a3]  universe_init()+0x43
V  [libjvm.so+0x7ae77f]  init_globals()+0x5f
V  [libjvm.so+0xc073d7]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x3a7
V  [libjvm.so+0x7feeac]  JNI_CreateJavaVM+0x5c
C  [libjli.so+0x3934]  JavaMain+0x84
C  [libjli.so+0x8a2a]  ContinueInNewThread0+0x5a
C  [libjli.so+0x5337]  ContinueInNewThread+0x57
C  [libjli.so+0x6378]  JLI_Launch+0xe38
C  [javac+0x9b4]
C  [libc.so.6+0x20830]  __libc_start_main+0xf0


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

Other Threads:

[error occurred during error reporting (printing all threads), id 0xe0000001]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None


[error occurred during error reporting (printing compressed oops mode), id 0xe0000001]

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Classes redefined (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events

VM Arguments:
jvm_args: -Dapplication.home=/home/xxx/hione_java/prebuilts/jdk/jdk9/linux-x86 --add-modules=ALL-DEFAULT -Xms8m -Xmx2048M -Djdk.module.main=jdk.compiler
java_command: jdk.compiler/com.sun.tools.javac.Main -Xmaxerrs 9999999 -encoding UTF-8 -sourcepath  -g -XDskipDuplicateBridges=true -XDstringConcat=inline -proc:none -bootclasspath out/soong
/.intermediates/libcore/mmodules/core_platform_api/core.platform.api.stubs/android_common/turbine-combined/core.platform.api.stubs.jar:out/soong/.intermediates/libcore/core-lambda-stubs/and
roid_common/turbine-combined/core-lambda-stubs.jar -classpath out/soong/.intermediates/frameworks/base/ext/android_common/turbine-combined/ext.jar:out/soong/.intermediates/frameworks/base/f
ramework/android_common/turbine-jarjar/framework.jar:out/soong/.intermediates/frameworks/base/media/updatable_media_stubs/android_common/turbine-combined/updatable_media_stubs.jar:out/soong
/.intermediates/external/protobuf/libprotobuf-java-micro/android_common/turbine-combined/libprotobuf-java-micro.jar -source 1.8 -target 1.8 -d out/soong/.intermediates/hardware/ril/librilut
ils/sap-api-java-static/android_common/javac/classes -s out/soong/.intermediates/hardware/ril/librilutils/sap-api-java-static/android_common/javac/anno @out/soong/.intermediates/hardware/ri
l/librilutils/sap-api-java-static/android_common/javac/sap-api-java-static.jar.rsp @out/soong/.intermediates/hardware/ril/librilutils/sap-api-java-static/android_common/javac/srcjars/list
java_class_path (initial): <not set>
Launcher Type: SUN_STANDARD


Logging:
Log output configuration:
#0: stdout all=warning uptime,level,tags
#1: stderr all=off uptime,level,tags

Environment Variables:
JAVA_HOME=/home/xxx/hione_java/prebuilts/jdk/jdk9/linux-x86
PATH=/home/xxx/hione_java/prebuilts/jdk/jdk9/linux-x86/bin:/home/xxx/hione_java/prebuilts/jdk/jdk9/linux-x86/bin:/home/xxx/hione_java/out/soong/host/linux-x86/bin:/home/xxx/
hione_java/out/host/linux-x86/bin:/home/xxx/hione_java/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin:/home/xxx/hione_java/prebuilts/gcc/linux-x86/arm/arm-linux-andro
ideabi-4.9/bin:/home/xxx/hione_java/development/scripts:/home/xxx/hione_java/prebuilts/devtools/tools:/home/xxx/hione_java/external/selinux/prebuilts/bin:/home/xxx/hione_jav
a/prebuilts/misc/linux-x86/dtc:/home/xxx/hione_java/prebuilts/misc/linux-x86/libufdt:/home/xxx/hione_java/prebuilts/gcc/linux-x86/arm/arm-eabi-4.8/bin:/home/xxx/hione_java/prebu
ilts/gcc/linux-x86/arm/arm-none-eabi-6.2.1/bin:/home/xxx/hione_java/prebuilts/asuite/acloud/linux-x86:/home/xxx/hione_java/prebuilts/asuite/aidegen/linux-x86:/home/xxx/hione_jav
a/prebuilts/asuite/atest/linux-x86:/home/xxx/bin:/home/xxx/.local/bin:/home/xxx/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:None
:None/bin::
SHELL=/bin/bash


Signal Handlers:
SIGSEGV: [libjvm.so+0xc5e920], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xc5e920], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0xc5e920], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0xa984e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0xa984e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0xc5e920], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR2: [libjvm.so+0xa983a0], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none


---------------  S Y S T E M  ---------------

OS:DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS"
uname:Linux 4.4.0-131-generic #157-Ubuntu SMP Thu Jul 12 15:51:36 UTC 2018 x86_64
libc:glibc 2.23 NPTL 2.23
rlimit: STACK 16384k, CORE 0k, NPROC infinity, NOFILE 1048576, AS infinity
load average:1.65 12.57 19.96

/proc/meminfo:
MemTotal:       527979248 kB
MemFree:        393526888 kB
MemAvailable:   521512832 kB
Buffers:         2808744 kB
Cached:         121445536 kB
SwapCached:            0 kB
Active:         93355136 kB
Inactive:       34175464 kB
Active(anon):    3279544 kB
Inactive(anon):    22512 kB
Active(file):   90075592 kB
Inactive(file): 34152952 kB
Unevictable:        4136 kB
Mlocked:            4136 kB
SwapTotal:      48828412 kB
SwapFree:       48828412 kB
Dirty:             52972 kB
Writeback:             0 kB
AnonPages:       3236204 kB
Mapped:           121656 kB
Shmem:             23460 kB
Slab:            5441964 kB
SReclaimable:    5157688 kB
SUnreclaim:       284276 kB
KernelStack:      128272 kB
PageTables:        91132 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    312818036 kB
Committed_AS:   51082884 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:      8192 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      221056 kB
DirectMap2M:     4610048 kB
DirectMap1G:    533725184 kB




CPU:total 88 (initial active 88) (22 cores per cpu, 2 threads per core) family 6 model 79 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, c
lmul, erms, rtm, 3dnowpref, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx, fma
CPU Model and flags from /proc/cpuinfo:
model name      : Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perf
mon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc
_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx
2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm arat pln pts

Memory: 4k page, physical 527979248k(393536968k free), swap 48828412k(48828412k free)

vm_info: OpenJDK 64-Bit Server VM (9+181-android4685155) for linux-amd64 JRE (9+181-android4685155), built on Mar 28 2018 20:02:59 by "androidbuild" with gcc 4.8

END









Chunlin Zhang

unread,
Oct 25, 2019, 11:32:02 AM10/25/19
to Android Building
All the exception is throw when call "AbstractWorkGang::add_workers", and in add_workers, if the new worker not created, will exit with this exception
( https://hg.openjdk.java.net/jdk9/jdk9/hotspot/file/b756e7a2ec33/src/share/vm/gc/shared/workerManager.hpp#l70 )
    for (uint worker_id = start; worker_id < end; worker_id += 1) {
      WorkerThread* new_worker = NULL;
      if (initializing || !InjectGCWorkerCreationFailure) {
        new_worker = holder->install_worker(worker_id);
      }
      if (new_worker == NULL || !os::create_thread(new_worker, worker_type)) {
        log_trace(gc, task)("WorkerManager::add_workers() : "
                            "creation failed due to failed allocation of native %s",
                            new_worker == NULL ?  "memory" : "thread");
        if (new_worker != NULL) {
           delete new_worker;
        }
        if (initializing) {
          vm_exit_out_of_memory(0, OOM_MALLOC_ERROR, "Cannot create worker GC thread. Out of system resources.");
        }
        break;
      }
      created_workers++;
      os::start_thread(new_worker);
    }


So I think there are 2 possibilities:
exceed process number limit, so new thead ceation failed
memory not enough, so creation failed

Because in all the hs_err file, we can see the memory is enough(MemAvailable:   521512832 kB), so the most possible is exceeding process number limt.

在 2019年6月10日星期一 UTC+8下午10:30:01，dong liang写道：

divya govind

unread,
Jan 14, 2020, 4:47:22 PM1/14/20
to Android Building
generate thread dumps each 1-5 seconds gap there should thread leak (recently we had encountered similar issue found one of the class just create thread never being released) 
please post full stack trace ?

Agostinho Carvalho

unread,
Jan 15, 2020, 11:40:17 AM1/15/20
to android-...@googlegroups.com
Hello.

Have you tried increasing the JAVA memory Heap size?

To show the current JAVA Heap size, run the following command under Linux terminal:
java -XshowSettings 2>&1  | grep -i Heap

To adjust the JAVA Heap size to 4GB (should be enough):
export _JAVA_OPTIONS="-Xmx4G"

After those adjustments (and using the same terminal session), try to recompile AOSP.

Regards.
Agostinho Carvalho

--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to android-...@googlegroups.com
To unsubscribe from this group, send email to
android-buildi...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to the Google Groups "Android Building" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-buildi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/957563b0-064c-4389-8936-2e1e49c84d7b%40googlegroups.com.


--
Agostinho Carvalho

Chunlin Zhang

unread,
Jan 15, 2020, 11:40:20 AM1/15/20
to android-...@googlegroups.com
After google with error message "pthread_create failed (EAGAIN) for attributes", we found a bug report to elasticsearch, then a blog posted by elasticsearch team: We are out of memory (or: Why systemd process limits ruined my day), the cgroup pid limit look possible connect with the error we get.
In the kernel log,we can found log similar like this:
kernel: [10199168.480630] cgroup: fork rejected by pids controller in /user.slice/user-23152.slice/session-46935.scope

In this article mention a command to show all the pids.max:
for f in `find /sys/fs/cgroup/pids/user.slice -name 'pids.max'`; do echo "$f"; cat "$f"; done
Command to modify the pids.max
systemctl set-property user-1000.slice TasksMax=512
We found it works after modify the pids.max.

--
--
You received this message because you are subscribed to the "Android Building" mailing list.
To post to this group, send email to android-...@googlegroups.com
To unsubscribe from this group, send email to
android-buildi...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-building?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Android Building" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/android-building/Xw-k0Q0x10k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to android-buildi...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages