Segmentation fault when running out of diskspace

[Guillaume Forget]

Hello,

We're using Mongo 2.0.7 on Linux x86 64 bits. We were preparing for a test by loading data into mongo using upserts from a single client to a single mongod. Suddenly mongod crashed with a segmentation fault. After reviewing the logs the error is likely due to mongod running out of diskspace. Yet I wonder why the mongo server didn't switch to a read-only mode instead of crashing? Isn't there such mode?

Also, had we been running a replica set with at least one secondary with a lot more disk space, would we have been able to continue inserting data?

Thank you for any insight into this somewhat minor issue.

Gui

[Stephen Steneker]

We're using Mongo 2.0.7 on Linux x86 64 bits. We were preparing for a test by loading data into mongo using upserts from a single client to a single mongod. Suddenly mongod crashed with a segmentation fault. After reviewing the logs the error is likely due to mongod running out of diskspace. Yet I wonder why the mongo server didn't switch to a read-only mode instead of crashing? Isn't there such mode?

 

Hi Guillaume,

Can you post the specific segfault / stacktrace message .. it should have more context on what caused the crash.

 

Also, had we been running a replica set with at least one secondary with a lot more disk space, would we have been able to continue inserting data?

Yes .. with a node down in a replica set, you would be able to continue inserting data assuming that:

 - not including the secondary which runs of disk space and is "down", you still have a majority of the nodes in the replica set available

 - any "write concerns" you require in your application code are still satisfied:
 http://docs.mongodb.org/manual/applications/replication/#replica-set-write-concern 

Cheers,

Stephen

[Guillaume Forget]

Hello Stephen,

Thank you for your response. We had verbose mode on in the logs so there's a lot of stuff I filtered out. Hopefully I got all the meaningful information:

Thu Sep 13 11:18:13 [FileAllocator] error failed to allocate new file: /mysqlroot/mongodb/community/community.5 size: 2146435072 errno:28 No space left on device

Thu Sep 13 11:18:13 [FileAllocator]     will try again in 10 seconds

Thu Sep 13 11:18:13 Backtrace:

0xa9609a 0x3ac52302f0 0x3ac5230285 0x3ac5231d30 0x75befd 0x777b65 0x777d9e 0x762cdc 0x76352d 0x76384d 0x76412b 0xaabdb0 0x3ac5e0677d 0x3ac52d325d 

 /usr/local/bin/mongod(_ZN5mongo10abruptQuitEi+0x3aa) [0xa9609a]

 /lib64/libc.so.6 [0x3ac52302f0]

 /lib64/libc.so.6(gsignal+0x35) [0x3ac5230285]

 /lib64/libc.so.6(abort+0x110) [0x3ac5231d30]

 /usr/local/bin/mongod(_ZN5mongo7LogFile17synchronousAppendEPKvm+0x12d) [0x75befd]

 /usr/local/bin/mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKNS_14AlignedBuilderE+0x1e5) [0x777b65]

 /usr/local/bin/mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_14AlignedBuilderE+0x4e) [0x777d9e]

 /usr/local/bin/mongod(_ZN5mongo3dur28_groupCommitWithLimitedLocksEv+0x24c) [0x762cdc]

 /usr/local/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x1d) [0x76352d]

 /usr/local/bin/mongod [0x76384d]

 /usr/local/bin/mongod(_ZN5mongo3dur9durThreadEv+0x10b) [0x76412b]

 /usr/local/bin/mongod(thread_proxy+0x80) [0xaabdb0]

 /lib64/libpthread.so.0 [0x3ac5e0677d]

 /lib64/libc.so.6(clone+0x6d) [0x3ac52d325d]

Logstream::get called in uninitialized state

Thu Sep 13 11:18:13 Invalid access at address: 0x4

Thu Sep 13 11:18:13 Got signal: 11 (Segmentation fault).

Thu Sep 13 11:18:13 Backtrace:

0xa9609a 0xa9678c 0x3ac5e0ebe0 0x54f952 0x55a864 0x55b554 0x8e07e5 0x8e13c8 0x96026e 0x96501d 0x88ce24 0x88e7cf 0xaa0b38 0x638767 0x3ac5e0677d 0x3ac52d325d 

 /usr/local/bin/mongod(_ZN5mongo10abruptQuitEi+0x3aa) [0xa9609a]

 /usr/local/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x22c) [0xa9678c]

 /lib64/libpthread.so.0 [0x3ac5e0ebe0]

 /usr/local/bin/mongod(_ZN5mongo10FieldRangeC1ERKNS_11BSONElementEbbb+0x1a2) [0x54f952]

 /usr/local/bin/mongod(_ZN5mongo13FieldRangeSet17processQueryFieldERKNS_11BSONElementEb+0x84) [0x55a864]

 /usr/local/bin/mongod(_ZN5mongo13FieldRangeSetC1EPKcRKNS_7BSONObjEbb+0x194) [0x55b554]

 /usr/local/bin/mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKNS_11BSONElementEbS5_S5_bb+0x1f5) [0x8e07e5]

 /usr/local/bin/mongod(_ZN5mongo11MultiCursorC1EPKcRKNS_7BSONObjES5_N5boost10shared_ptrINS0_8CursorOpEEEbb+0x138) [0x8e13c8]

 /usr/local/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugEPNS_11RemoveSaverEb+0x38e) [0x96026e]

 /usr/local/bin/mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugEb+0x13d) [0x96501d]

 /usr/local/bin/mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x474) [0x88ce24]

 /usr/local/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x116f) [0x88e7cf]

 /usr/local/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) [0xaa0b38]

 /usr/local/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x638767]

 /lib64/libpthread.so.0 [0x3ac5e0677d]

 /lib64/libc.so.6(clone+0x6d) [0x3ac52d325d]

Logstream::get called in uninitialized state

Thu Sep 13 11:18:13 [conn79] ERROR: Client::~Client _context should be null but is not; client:conn

Logstream::get called in uninitialized state

Thu Sep 13 11:18:13 [conn79] ERROR: Client::shutdown not called: conn

[Stephen Steneker]

Thank you for your response. We had verbose mode on in the logs so there's a lot of stuff I filtered out. Hopefully I got all the meaningful information:

Hi Guillaume,

Apologies for the delay in following up .. missed the reply on this.

I've added your stacktrace to the related server ticket.  Would be helpful if you are able to Watch this issue for any feedback:

 https://jira.mongodb.org/browse/SERVER-4038

MongoDB should handle this more gracefully than a segfault.

Cheers,

Stephen