Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
mongodb-user
Home
+ new post
About this group
Join this group
mongos crashes in concurrent read/write load
Options
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
   4 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Yulias Stolin  
View profile  
 More options Aug 7 2012, 6:25 am
From: Yulias Stolin <sjulia...@gmail.com>
Date: Tue, 7 Aug 2012 03:25:53 -0700 (PDT)
Local: Tues, Aug 7 2012 6:25 am
Subject: mongos crashes in concurrent read/write load
Print | Individual message | Show original | Report this message | Find messages by this author
I'm using mongodb version 2.0.6. I've configured it with sharding :
3 shards (each replica set master+secondary+arbiter)
1 config sevice
1 mongos

I'm running a jmeter test with 500 concurrent threads that invokes
each some read operation on mongo.
In a batch process in single thread I'm running 10 million upserts.
I see when my collection became about 5 millions the mongos crashes.

This is the log I see in it's .log file

Tue Aug  7 12:49:53 [mongosMain] can't create new thread, closing
connection
Tue Aug  7 12:49:53 [conn143663] DBException in process: could not
initialize cursor across all shards because :
boost::thread_resource_error @
shard2/172.16.40.202:10010,172.16.40.32:10010
Tue Aug  7 12:49:53 [conn143663] DBException in process: could not
initialize cursor across all shards because :
boost::thread_resource_error @
shard2/172.16.40.202:10010,172.16.40.32:10010
Tue Aug  7 12:49:53 [mongosMain] pthread_create failed: errno:11
Resource temporarily unavailable
Tue Aug  7 12:49:53 [mongosMain] can't create new thread, closing
connection
Tue Aug  7 12:49:53 [conn143663] ERROR: Uncaught std::exception:
boost::thread_resource_error, terminating
Tue Aug  7 12:49:53 [conn143663] dbexit:  rc:100
Logstream::get called in uninitialized state
Tue Aug  7 12:49:53 [mongosMain] dbexit:  rc:0
Received signal 6
Backtrace: 0x54e5b5 0x7fb8a7d46c20 0x7fb8a7d46ba5 0x7fb8a7d4a6b0
0x7fb8a7d3fa71 0x5e9f04 0x7fb8a883d971 0x7fb8a7df9f3d
/opt/mongodb-linux-x86_64-2.0.6/bin/
mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x54e5b5]
/lib/libc.so.6(+0x33c20)[0x7fb8a7d46c20]
/lib/libc.so.6(gsignal+0x35)[0x7fb8a7d46ba5]

What does it mean?
BR,
Yulia Stolin


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Jenna  
View profile  
 More options Aug 7 2012, 11:00 am
From: Jenna <jenna.deboisbl...@10gen.com>
Date: Tue, 7 Aug 2012 08:00:51 -0700 (PDT)
Local: Tues, Aug 7 2012 11:00 am
Subject: Re: mongos crashes in concurrent read/write load
Print | Individual message | Show original | Report this message | Find messages by this author
Hello Yulia,
I believe this error message is thrown when the operating system
reaches a limit on the number of connections it can accept.  There is
a ticket in JIRA related to handling this error:
https://jira.mongodb.org/browse/SERVER-4978

Do you think this is a possible explanation? Can you post your current
ulimit (ulimit -n to get the current setting). It's possible to change
the default number of allowed connections if you haven't already:
http://www.mongodb.org/display/DOCS/Too+Many+Open+Files

I would also like to point out that if you have a single config
server, I would strongly recommend adding two additional config
servers.  These servers store information about where your data exists
in a sharded cluster, and as such, it's very important to ensure that
this data is highly durable.

On Aug 7, 6:25 am, Yulias Stolin <sjulia...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Yulias Stolin  
View profile  
 More options Aug 8 2012, 7:08 am
From: Yulias Stolin <sjulia...@gmail.com>
Date: Wed, 8 Aug 2012 04:08:38 -0700 (PDT)
Local: Wed, Aug 8 2012 7:08 am
Subject: Re: mongos crashes in concurrent read/write load
Print | Individual message | Show original | Report this message | Find messages by this author

Hello,

I'm also think that this is a connection issue, but I do not understand why
it happens?
I'm running mongo2.0.6 on ubuntu

>  ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 131072
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
I'm using mongo driver  2.8.0

The test I've run was in jmetter with 500 concurrent readers. In addition I
have one single thread writer who's loading the 10M documents.
This is only test environment, thus I've configured 1 config server, I know
that I need to run with 3 (I wanted to simplify the test meanwhile)

I do not understand what is happening with the connections? maybe there is
some bug in mongos/mongod/java driver.

This is another snapshot of crash log.

Tue Aug  7 16:14:40 [mongosMain] can't create new thread, closing connection

...

read more »


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
William Z  
View profile  
 More options Aug 23 2012, 2:29 pm
From: William Z <william.z...@10gen.com>
Date: Thu, 23 Aug 2012 11:29:05 -0700 (PDT)
Local: Thurs, Aug 23 2012 2:29 pm
Subject: Re: mongos crashes in concurrent read/write load
Print | Individual message | Show original | Report this message | Find messages by this author

Hi Yulias!

Your log file contains the error message:

> Tue Aug  7 16:14:40 [mongosMain] pthread_create failed: errno:11 Resource

temporarily unavailable

This indicates that your OS was unable to create a new thread within the
'mongos' process.  Each incoming connection to 'mongos' requires a new
thread, and the operating system is returning the EAGAIN error.

This means that this is an OS configuration problem.

Looking at the documentation for pthread_create(), I note that the OS can
return EAGAIN from pthread_create() if there aren't enough resources for
the individual process, or if there aren't enough resources system-wide.

(Ref:
http://www.kernel.org/doc/man-pages/online/pages/man3/pthread_create....)

The output from ulimit -a indicates that you have no limit on the per-user
number of processes.  You can check the number of system-wide processes by
looking in the file /proc/sys/kernel/threads-max .  You can raise this
limit temporarily using the 'sysctl' command: you can change it permanently
by editing /etc/sysctl.conf and setting the 'kernel.max-threads' value.

Let me know if this resolves your problem or if you have further
questions.  Have a great day!

 -William


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2013 Google