2016-05-05T08:11:03.408-0400 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable2016-05-05T08:11:03.408-0400 I NETWORK [initandlisten] failed to create thread after accepting new connection, closing connection
$ ulimit -a
core file size (blocks, -c) 0data seg size (kbytes, -d) unlimitedscheduling priority (-e) 0file size (blocks, -f) unlimitedpending signals (-i) 1546963max locked memory (kbytes, -l) unlimitedmax memory size (kbytes, -m) unlimitedopen files (-n) 131072pipe size (512 bytes, -p) 8POSIX message queues (bytes, -q) 819200real-time priority (-r) 0stack size (kbytes, -s) 8192cpu time (seconds, -t) unlimitedmax user processes (-u) unlimitedvirtual memory (kbytes, -v) unlimitedfile locks (-x) unlimited
$ cat /etc/security/limits.d/20-nproc.conf# Default limit for number of user's processes to prevent# accidental fork bombs.# See rhbz #432903 for reasoning.
* soft nproc 4096root soft nproc unlimited
oasprod soft nofile unlimited#oasprod hard nofile 65535oasprod hard nofile 131072oasprod soft nproc unlimitedoasprod hard nproc unlimitedoasprod soft fsize unlimitedoasprod hard fsize unlimitedoasprod soft memlock unlimitedoasprod hard memlock unlimitedoasprod soft cpu unlimitedoasprod hard cpu unlimitedoasprod soft as unlimitedoasprod hard as unlimited
# Sample limits of a Mongod process
$ cat /proc/44954/limitsLimit Soft Limit Hard Limit UnitsMax cpu time unlimited unlimited secondsMax file size unlimited unlimited bytesMax data size unlimited unlimited bytesMax stack size 8388608 unlimited bytesMax core file size 0 unlimited bytesMax resident set unlimited unlimited bytesMax processes unlimited unlimited processesMax open files 131072 131072 filesMax locked memory unlimited unlimited bytesMax address space unlimited unlimited bytesMax file locks unlimited unlimited locksMax pending signals 1546963 1546963 signalsMax msgqueue size 819200 819200 bytesMax nice priority 0 0Max realtime priority 0 0Max realtime timeout unlimited unlimited us
$ cat /proc/sys/net/ipv4/ip_local_port_range32768 61000
$ cat /proc/sys/vm/max_map_count
65530
$ cat /proc/sys/net/ipv4/tcp_keepalive_time35
$ cat /proc/sys/kernel/threads-max3093926
pthread_create failed: errno:11 Resource temporarily unavailable
This message is continuously printed multiple times in the log and then everything seems to be fine after 15-30 seconds.
Hi Darshan,
Generally the error message is an indication that the OS runs out of sockets for mongod
to open (pthread_create
returns EAGAIN
see linux pthread_create:ERRORS).
$ cat /proc/sys/kernel/threads-max
3093926
Based on your /proc/sys/kernel/threads-max
, you have increased the maximum number of threads on the system to be 3093926
. Depending on your hardware specs and applications requirements, you need to ensure that this is a reasonable number. Try tuning the number of maximum threads accordingly.
$ cat /etc/security/limits.d/20-nproc.conf
- soft nproc 4096
root soft nproc unlimited
Based on the output of /etc/security/limits.d/20-nproc.conf
, although you have the root user setting to unlimited, generally mongod
in CentOS is executed via mongod
user. Check whether the mongod
process is actually under the soft limits of 4096
.
Another thing to check, sometimes multiple short-lived connections can caused the machine to reach the system limits. Check on how your applications utilise the connections. i.e. depending on your use case see Bulk() operations.
You may also be benefited from reading these deployments/operational resources:
Kind regards,
Wan.