MongoDB is segfault-ing on production

[igotux igotux]

Hi,

We are running mongodb on master->slave model and for last 2 days, we are seeing that, mongo is segfault-ing and crashes. The logs showing this error.

~~~~~~~~~~~~~~~~~~~

Mon Nov  5 08:45:50 Invalid access at address: 0xfffffd6ea926981c

Mon Nov  5 08:45:50 Got signal: 11 (Segmentation Fault).

Mon Nov  5 08:45:50 Backtrace:

Logstream::get called in uninitialized state

Mon Nov  5 08:45:50 [conn11] getmore local.oplog.$main query: { ts: { $gte: new Date(5806693594035126273) } } cursorid:55082034726758 nreturned:2 reslen:9034 7939ms

Mon Nov  5 08:45:50 Invalid access at address: 0x4

Mon Nov  5 08:45:50 Got signal: 11 (Segmentation Fault).

Mon Nov  5 08:45:50 Backtrace:

Mon Nov  5 08:45:50 Invalid access at address: 0xffffffffffffffe8

Mon Nov  5 08:45:50 Got signal: 11 (Segmentation Fault).

Mon Nov  5 08:45:50 Backtrace:

Logstream::get called in uninitialized state

Mon Nov  5 08:45:50 ERROR: Client::~Client _context should be null but is not; client:conn

Logstream::get called in uninitialized state

Mon Nov  5 08:45:50 ERROR: Client::shutdown not called: conn

~~~~~~~~~~~~~~~~~~~

Everytime, mongo seg faults, we are restarting the mongo server now. Please let us know how can we fix this.

Thanks in advance,

/Aroop

[Sam Millman]

I must admit I no expert when it comes to memory access errors however: "Mon Nov  5 08:45:50 Invalid access at address: 0xfffffd6ea926981c" looks very troubling and could be a sign of bad hardware. I would run a full memory test on your machines.

Do your machines also crash and stall? Do other applications/programs have difficulties? What MongoDB version is this?

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb

[igotux igotux]

Thanks Sam. The server works / responds fine, when this mongo crash happens. It runs on smartOS though. Hence i assume this is not a hardware issue. Please have a look into the more detailed logs here :- http://pastie.org/private/9ijbeuduujuv3cdgry26a 

These lines "[conn11] getmore local.oplog.$main query: { ts: { $gte: new Date(5806693594035126273) } } cursorid:55082034726758 nreturned:2 reslen:187655" looks very suspecious.

[gregor]

Which version of mongodb is this?

[igotux igotux]

version: 2.0.7

[Navin Kumar (@vain)]

Do you have a low ulimit set? Set it to >4000 and you'll be fine

[Sam Millman]

I believe ulimits usage is still bugged and doesn't function with MongoDB. It may have been fixed but I am unsure.

On 6 November 2012 08:35, Navin Kumar (@vain) <nav...@gmail.com> wrote:

Do you have a low ulimit set? Set it to >4000 and you'll be fine

[gregor]

ulimit settings are working - recommended settings are here http://docs.mongodb.org/manual/administration/ulimit/#recommended-settings

but ulimit settings wouldn't cause a segfault like this - it's difficult to see what the problem is without a backtrace - how often is it crashing? 

If you could increase the log level it might give us more to go on. You can do this at command line with 

-vvvvvv 

(5 v's give maximum log level)

or at run time with

use admin

db.runCommand({setParameter : 1}, {logLevel : 5})

to set log level 5. Log level 5 will produce a *lot* of logging though - so you might run out of disk space. If you prefer to set a lower log level then this would still be helpful. 

[igotux igotux]

Have increased the log level now.

> use admin

switched to db admin

> db.runCommand({setParameter : 1}, {logLevel : 5})

{

        "errmsg" : "no option found to set, use help:true to see options ",

        "ok" : 0

}

> 

bye

We had seg fault on last Sunday and Monday. 

[igotux igotux]

And seeing lot of messages on the logs like this for same Date var :- http://pastie.org/private/zycdirobenu8ancq734m3w

[gregor]

Sorry my bad. Try

use admin

db.runCommand({setParameter : 1, logLevel : 5})

And what we really need it to get more logging around the segfault. If this is only happening a couple of times a week then logLevel 5 will be too much

try level 2 maybe.

db.runCommand({setParameter : 1, logLevel : 2})

[Dwight Merriman]

so it is crashing.

what version?

what operating system?

32 or 64 bit?

[Tux]

On 09/11/12 1:11 AM, Dwight Merriman wrote:

so it is crashing.

what version?

2.0.7

what operating system?

SmartOs

32 or 64 bit?

i86pc

--

[gregor]

Can you paste the 20 lines of logging before the segfault?

[Tux]

On 16/11/12 10:08 PM, gregor wrote:

Can you paste the 20 lines of logging before the segfault?

Here it is :- http://pastie.org/private/2zfwekobitzncjlyy05bgw

Thanks for your help. We are having segfaults every day with the same error. Any helps/inputs will be much much appreciated.

[gregor]

Can you increase the log level 

use admin

db.runCommand({setParameter : 1, logLevel : 2})

and when it segfaults again, post some of the log and the segfault again please.