Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
mongodb-user
Home
+ new post
About this group
Join this group
MongoDB is segfault-ing on production
Options
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
   16 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
igotux igotux  
View profile  
 More options Nov 5 2012, 6:05 am
From: igotux igotux <igo...@gmail.com>
Date: Mon, 5 Nov 2012 16:35:02 +0530
Local: Mon, Nov 5 2012 6:05 am
Subject: MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

Hi,

We are running mongodb on master->slave model and for last 2 days, we are
seeing that, mongo is segfault-ing and crashes. The logs showing this error.
~~~~~~~~~~~~~~~~~~~
Mon Nov  5 08:45:50 Invalid access at address: 0xfffffd6ea926981c

Mon Nov  5 08:45:50 Got signal: 11 (Segmentation Fault).

Mon Nov  5 08:45:50 Backtrace:

Logstream::get called in uninitialized state
Mon Nov  5 08:45:50 [conn11] getmore local.oplog.$main query: { ts: { $gte:
new Date(5806693594035126273) } } cursorid:55082034726758 nreturned:2
reslen:9034 7939ms
Mon Nov  5 08:45:50 Invalid access at address: 0x4

Mon Nov  5 08:45:50 Got signal: 11 (Segmentation Fault).

Mon Nov  5 08:45:50 Backtrace:

Mon Nov  5 08:45:50 Invalid access at address: 0xffffffffffffffe8

Mon Nov  5 08:45:50 Got signal: 11 (Segmentation Fault).

Mon Nov  5 08:45:50 Backtrace:

Logstream::get called in uninitialized state
Mon Nov  5 08:45:50 ERROR: Client::~Client _context should be null but is
not; client:conn
Logstream::get called in uninitialized state
Mon Nov  5 08:45:50 ERROR: Client::shutdown not called: conn
~~~~~~~~~~~~~~~~~~~

Everytime, mongo seg faults, we are restarting the mongo server now. Please
let us know how can we fix this.

Thanks in advance,
/Aroop


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sam Millman  
View profile  
 More options Nov 5 2012, 7:17 am
From: Sam Millman <sam.mill...@gmail.com>
Date: Mon, 5 Nov 2012 12:17:06 +0000
Local: Mon, Nov 5 2012 7:17 am
Subject: Re: [mongodb-user] MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

I must admit I no expert when it comes to memory access errors however:
"Mon Nov  5 08:45:50 Invalid access at address: 0xfffffd6ea926981c" looks
very troubling and could be a sign of bad hardware. I would run a full
memory test on your machines.

Do your machines also crash and stall? Do other applications/programs have
difficulties? What MongoDB version is this?

On 5 November 2012 11:05, igotux igotux <igo...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
igotux igotux  
View profile  
 More options Nov 5 2012, 8:16 am
From: igotux igotux <igo...@gmail.com>
Date: Mon, 5 Nov 2012 18:46:08 +0530
Local: Mon, Nov 5 2012 8:16 am
Subject: Re: [mongodb-user] MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

Thanks Sam. The server works / responds fine, when this mongo crash
happens. It runs on smartOS though. Hence i assume this is not a hardware
issue. Please have a look into the more detailed logs here :-
http://pastie.org/private/9ijbeuduujuv3cdgry26a

These lines "[conn11] getmore local.oplog.$main query: { ts: { $gte: new
Date(5806693594035126273) } } cursorid:55082034726758 nreturned:2
reslen:187655"
looks very suspecious.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
gregor  
View profile  
 More options Nov 5 2012, 1:16 pm
From: gregor <gre...@10gen.com>
Date: Mon, 5 Nov 2012 10:16:23 -0800 (PST)
Local: Mon, Nov 5 2012 1:16 pm
Subject: Re: [mongodb-user] MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

Which version of mongodb is this?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
igotux igotux  
View profile  
 More options Nov 6 2012, 12:44 am
From: igotux igotux <igo...@gmail.com>
Date: Tue, 6 Nov 2012 11:14:19 +0530
Local: Tues, Nov 6 2012 12:44 am
Subject: Re: [mongodb-user] MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

version: 2.0.7


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Navin Kumar (@vain)  
View profile  
 More options Nov 6 2012, 3:35 am
From: "Navin Kumar (@vain)" <nav...@gmail.com>
Date: Tue, 6 Nov 2012 00:35:48 -0800 (PST)
Local: Tues, Nov 6 2012 3:35 am
Subject: MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

Do you have a low ulimit set? Set it to >4000 and you'll be fine


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sam Millman  
View profile  
 More options Nov 6 2012, 4:38 am
From: Sam Millman <sam.mill...@gmail.com>
Date: Tue, 6 Nov 2012 09:38:36 +0000
Local: Tues, Nov 6 2012 4:38 am
Subject: Re: [mongodb-user] MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

I believe ulimits usage is still bugged and doesn't function with MongoDB.
It may have been fixed but I am unsure.

On 6 November 2012 08:35, Navin Kumar (@vain) <nav...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
gregor  
View profile  
 More options Nov 6 2012, 5:05 am
From: gregor <gre...@10gen.com>
Date: Tue, 6 Nov 2012 02:05:51 -0800 (PST)
Local: Tues, Nov 6 2012 5:05 am
Subject: Re: [mongodb-user] MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

ulimit settings are working - recommended settings are here
http://docs.mongodb.org/manual/administration/ulimit/#recommended-set...

but ulimit settings wouldn't cause a segfault like this - it's difficult to
see what the problem is without a backtrace - how often is it crashing?
If you could increase the log level it might give us more to go on. You can
do this at command line with

-vvvvvv

(5 v's give maximum log level)

or at run time with

use admin
db.runCommand({setParameter : 1}, {logLevel : 5})

to set log level 5. Log level 5 will produce a *lot* of logging though - so
you might run out of disk space. If you prefer to set a lower log level
then this would still be helpful.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
igotux igotux  
View profile  
 More options Nov 6 2012, 8:09 pm
From: igotux igotux <igo...@gmail.com>
Date: Wed, 7 Nov 2012 06:39:11 +0530
Local: Tues, Nov 6 2012 8:09 pm
Subject: Re: [mongodb-user] MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

Have increased the log level now.

> use admin

switched to db admin
> db.runCommand({setParameter : 1}, {logLevel : 5})

{
        "errmsg" : "no option found to set, use help:true to see options ",
        "ok" : 0
}

bye

We had seg fault on last Sunday and Monday.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
igotux igotux  
View profile  
 More options Nov 6 2012, 8:34 pm
From: igotux igotux <igo...@gmail.com>
Date: Wed, 7 Nov 2012 07:03:54 +0530
Local: Tues, Nov 6 2012 8:33 pm
Subject: Re: [mongodb-user] MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

And seeing lot of messages on the logs like this for same Date var :-
http://pastie.org/private/zycdirobenu8ancq734m3w


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
gregor  
View profile  
 More options Nov 8 2012, 9:41 am
From: gregor <gre...@10gen.com>
Date: Thu, 8 Nov 2012 06:41:46 -0800 (PST)
Local: Thurs, Nov 8 2012 9:41 am
Subject: Re: [mongodb-user] MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

Sorry my bad. Try

use admin
db.runCommand({setParameter : 1, logLevel : 5})

And what we really need it to get more logging around the segfault. If this
is only happening a couple of times a week then logLevel 5 will be too much

try level 2 maybe.

db.runCommand({setParameter : 1, logLevel : 2})


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Dwight Merriman  
View profile  
 More options Nov 8 2012, 2:41 pm
From: Dwight Merriman <dwi...@10gen.com>
Date: Thu, 8 Nov 2012 11:41:01 -0800 (PST)
Local: Thurs, Nov 8 2012 2:41 pm
Subject: Re: MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

so it is crashing.
what version?
what operating system?
32 or 64 bit?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tux  
View profile  
 More options Nov 15 2012, 11:36 am
From: Tux <igo...@gmail.com>
Date: Thu, 15 Nov 2012 22:06:28 +0530
Local: Thurs, Nov 15 2012 11:36 am
Subject: Re: [mongodb-user] Re: MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

On 09/11/12 1:11 AM, Dwight Merriman wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
gregor  
View profile  
 More options Nov 16 2012, 11:38 am
From: gregor <gre...@10gen.com>
Date: Fri, 16 Nov 2012 08:38:02 -0800 (PST)
Local: Fri, Nov 16 2012 11:38 am
Subject: Re: [mongodb-user] Re: MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

Can you paste the 20 lines of logging before the segfault?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Tux  
View profile  
 More options Nov 16 2012, 11:44 pm
From: Tux <igo...@gmail.com>
Date: Sat, 17 Nov 2012 10:14:15 +0530
Local: Fri, Nov 16 2012 11:44 pm
Subject: Re: [mongodb-user] Re: MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

On 16/11/12 10:08 PM, gregor wrote:

> Can you paste the 20 lines of logging before the segfault?

Here it is :- http://pastie.org/private/2zfwekobitzncjlyy05bgw

Thanks for your help. We are having segfaults every day with the same
error. Any helps/inputs will be much much appreciated.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
gregor  
View profile  
 More options Nov 19 2012, 3:13 am
From: gregor <gre...@10gen.com>
Date: Mon, 19 Nov 2012 00:13:26 -0800 (PST)
Local: Mon, Nov 19 2012 3:13 am
Subject: Re: [mongodb-user] Re: MongoDB is segfault-ing on production
Print | Individual message | Show original | Report this message | Find messages by this author

Can you increase the log level

use admin
db.runCommand({setParameter : 1, logLevel : 2})

and when it segfaults again, post some of the log and the segfault again
please.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2013 Google