pthread_create error and crash

[Max Michaels]

Hello,

I've added a couple of shards to my cluster. The only difference on
these shards is that they are running CentOS 6 rather than 5 on the
older shards. The servers run for about an hour and then crash with
the following errors in the logs:

Tue Nov 29 09:56:08 [initandlisten] pthread_create failed: errno:11
Resource temporarily unavailable
Tue Nov 29 09:56:08 [initandlisten] can't create new thread, closing connection
Tue Nov 29 09:56:19 [conn1072] ERROR: Uncaught std::exception:
boost::thread_resource_error, terminating
Tue Nov 29 09:56:19 dbexit:
Tue Nov 29 09:56:19 [conn1072] shutdown: going to close listening sockets...
Tue Nov 29 09:56:19 [conn1072] closing listening socket: 6
Tue Nov 29 09:56:19 [conn1072] closing listening socket: 7
Tue Nov 29 09:56:19 [conn1072] closing listening socket: 8
Tue Nov 29 09:56:19 [conn1072] removing socket file: /tmp/mongodb-27017.sock
Tue Nov 29 09:56:19 [conn1072] shutdown: going to flush diaglog...
Tue Nov 29 09:56:19 [conn1072] shutdown: going to close sockets...
Tue Nov 29 09:56:19 shutdown failed with exception
Tue Nov 29 09:56:19 dbexit: really exiting now

There's a ton of the pthread create errors first and then eventually
the crash happens.

I've also added these to the top of the mongod startup script:

ulimit -n 80000
ulimit -u 250000

Any ideas?

Thanks,

-Max

[Max Michaels]

Another note on this is that the connection count is roughly the same
as it is on the servers that aren't having issues.

[Eliot Horowitz]

What version of mongod is this with?
If its < 2.0, can you set the stack size to 1024 (-s 1024)

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[Max Michaels]

It's 2.0.1.

Funny you mentioned the -s switch. I just changed it but am getting
the same result. The following is now in my mongod startup script:

ulimit -n 80000
ulimit -u 250000

ulimit -s 1024

Latest error is the same:

Tue Nov 29 14:30:49 [initandlisten] connection accepted from
10.20.20.52:55005 #1040
Tue Nov 29 14:30:49 [initandlisten] pthread_create failed: errno:11
Resource temporarily unavailable
Tue Nov 29 14:30:49 [initandlisten] can't create new thread, closing connection
Tue Nov 29 14:30:51 [conn166] ERROR: Uncaught std::exception:
boost::thread_resource_error, terminating
Tue Nov 29 14:30:51 dbexit:
Tue Nov 29 14:30:51 [conn166] shutdown: going to close listening sockets...
Tue Nov 29 14:30:51 [conn166] closing listening socket: 6
Tue Nov 29 14:30:51 [conn166] closing listening socket: 7
Tue Nov 29 14:30:51 [conn166] closing listening socket: 8
Tue Nov 29 14:30:51 [conn166] removing socket file: /tmp/mongodb-27017.sock
Tue Nov 29 14:30:51 [conn166] shutdown: going to flush diaglog...
Tue Nov 29 14:30:51 [conn166] shutdown: going to close sockets...
Tue Nov 29 14:30:51 shutdown failed with exception
Tue Nov 29 14:30:51 dbexit: really exiting now

[Max Michaels]

Any ideas on this one? This is kind of a show stopper.

Since last email, I've tried building from source against CentOS6
libraries, same result.

Any debugging ideas? I just have no clue why adding these shards has
ground the cluster to a halt. Are there known issues with CentOS 6 and
Mongo 2.0? Should I downgrade to CentOS 5?

Thanks,

-Max

[Richard Kreuter]

If you're running the ulimit commands as an non-root user, it could be
that the system-wide hard limits are lower than what you're asking for
(and so as an unprivileged user you can't increase your process's
resource limits above the hard limit). Can you report back the output
of the following?

ulimit -Sa
ulimit -Ha

Thanks,
Richard

On Nov 30, 2:04 pm, Max Michaels <maxmicha...@gmail.com> wrote:
> Any ideas on this one? This is kind of a show stopper.
>
> Since last email, I've tried building from source against CentOS6
> libraries, same result.
>
> Any debugging ideas? I just have no clue why adding these shards has
> ground the cluster to a halt. Are there known issues with CentOS 6 and
> Mongo 2.0? Should I downgrade to CentOS 5?
>
> Thanks,
>
> -Max
>

> >> On Tue, Nov 29, 2011 at 3:00 PM, Max Michaels <maxmicha...@gmail.com> wrote:
> >>> Another note on this is that the connection count is roughly the same
> >>> as it is on the servers that aren't having issues.
>

> >>> For more options, visit this group athttp://groups.google.com/group/mongodb-user?hl=en.

[Max Michaels]

ahhh. i think this is the issue.

so i had added this to my /etc/seucrity/limits.conf BUT. in CentOS 6,
there is a line in /etc/security/limits.d/90-nproc.conf that limits
total procs soft limit to 1024. This file doesn't exist in CentOS 5
and that's where the problem was....

Thanks!

[Tyler Brock]

Nice, thank you for reporting what the problem was after you
discovered it.

-Tyler