mongodb: pthread_create failed: errno:11 on ubuntu 16.04

Love Nyberg

unread,

Jun 15, 2016, 1:33:43 AM6/15/16

to mongodb-user

Running sharded setup on dedicated servers (ubuntu). We've updated our one of our servers to ubuntu 16.04, while the other replica set is still running ubuntu 12.04. When switching over primary to upgraded server (server1) it can only handle up to 417 connections (same count each time). Running MongoDB 3.2.2 on both servers.

2016-06-14T11:29:27.002+0200 I NETWORK  [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
2016-06-14T11:29:27.002+0200 I NETWORK  [initandlisten] failed to create thread after accepting new connection, closing connection

Aligned configuration around ulimits as described at https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings

Here's all current limits

server1 (ubuntu 16.04 not working)

[mongod #33318 -- limits]
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             64000                64000                processes 
Max open files            64000                64000                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       515435               515435               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

root@server1:~# ulimit -Ha
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515435
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 515435
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

server2 (ubuntu12.04 working)

[mongod #45274 -- limits]
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             64000                64000                processes 
Max open files            64000                64000                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       1031036              1031036              signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

root@ssdb2:~# ulimit -Ha
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1031036
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1031036
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Added the following to systemd start script /lib/systemd/system/mongod.service to server1

# (file size)
LimitFSIZE=infinity
# (cpu time)
LimitCPU=infinity
# (virtual memory size)
LimitAS=infinity
# (open files)
LimitNOFILE=64000
# (processes/threads)
LimitNPROC=64000

It always stops receiving new connections after 417 connections. Any other limit or configuration that needs to be set?

Message has been deleted

Henrik Hofmeister

unread,

Jun 29, 2016, 4:44:01 AM6/29/16

to mongodb-user

We were facing the exact same issue on Ubuntu 16 and after much hair pulling, yelling and general profanities we found that it was related to a new feature that came with systemd - namely "MaxTasks" which will by default limit services to 512 threads and subprocesses - a number easily used by a busy mongod it would seem.

The fix is using this systemd script:

https://gist.github.com/hofmeister/a4706604c3b899eb0288f0442be5f6c4

Andrew Morrow

unread,

Jun 29, 2016, 1:07:19 PM6/29/16

to mongod...@googlegroups.com

That is a very interesting discovery. Thank you for taking the time to do a full investigation. Would you please file a JIRA ticket in the SERVER project describing your findings? Or, let me know and I can file one on your behalf.

Thanks,

Andrew

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.org/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/e6eee5fd-05a5-4283-a2d6-1c601021a3d0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Andrew Morrow

unread,

Jul 2, 2016, 11:39:55 AM7/2/16

to mongod...@googlegroups.com

Hi -

I've filed https://jira.mongodb.org/browse/SERVER-24885 so that we can investigate this issue.