Hello Stephen,
Thank you for your response. We had verbose mode on in the logs so there's
a lot of stuff I filtered out. Hopefully I got all the meaningful
information:
Thu Sep 13 11:18:13 [FileAllocator] error failed to allocate new file:
/mysqlroot/mongodb/community/community.5 size: 2146435072 errno:28 No space
left on device
Thu Sep 13 11:18:13 [FileAllocator] will try again in 10 seconds
Thu Sep 13 11:18:13 Backtrace:
0xa9609a 0x3ac52302f0 0x3ac5230285 0x3ac5231d30 0x75befd 0x777b65 0x777d9e
0x762cdc 0x76352d 0x76384d 0x76412b 0xaabdb0 0x3ac5e0677d 0x3ac52d325d
/usr/local/bin/mongod(_ZN5mongo10abruptQuitEi+0x3aa) [0xa9609a]
/lib64/libc.so.6 [0x3ac52302f0]
/lib64/libc.so.6(gsignal+0x35) [0x3ac5230285]
/lib64/libc.so.6(abort+0x110) [0x3ac5231d30]
/usr/local/bin/mongod(_ZN5mongo7LogFile17synchronousAppendEPKvm+0x12d)
[0x75befd]
/usr/local/bin/mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKN S_14AlignedBuilderE+0x1e5)
[0x777b65]
/usr/local/bin/mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_1 4AlignedBuilderE+0x4e)
[0x777d9e]
/usr/local/bin/mongod(_ZN5mongo3dur28_groupCommitWithLimitedLocksEv+0x24c)
[0x762cdc]
/usr/local/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x1d)
[0x76352d]
/usr/local/bin/mongod [0x76384d]
/usr/local/bin/mongod(_ZN5mongo3dur9durThreadEv+0x10b) [0x76412b]
/usr/local/bin/mongod(thread_proxy+0x80) [0xaabdb0]
/lib64/libpthread.so.0 [0x3ac5e0677d]
/lib64/libc.so.6(clone+0x6d) [0x3ac52d325d]
Logstream::get called in uninitialized state
Thu Sep 13 11:18:13 Invalid access at address: 0x4
Thu Sep 13 11:18:13 Got signal: 11 (Segmentation fault).
Thu Sep 13 11:18:13 Backtrace:
0xa9609a 0xa9678c 0x3ac5e0ebe0 0x54f952 0x55a864 0x55b554 0x8e07e5 0x8e13c8
0x96026e 0x96501d 0x88ce24 0x88e7cf 0xaa0b38 0x638767 0x3ac5e0677d
0x3ac52d325d
/usr/local/bin/mongod(_ZN5mongo10abruptQuitEi+0x3aa) [0xa9609a]
/usr/local/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x22 c)
[0xa9678c]
/lib64/libpthread.so.0 [0x3ac5e0ebe0]
/usr/local/bin/mongod(_ZN5mongo10FieldRangeC1ERKNS_11BSONElementEbbb+0x1a2)
[0x54f952]
/usr/local/bin/mongod(_ZN5mongo13FieldRangeSet17processQueryFieldERKNS_11BS ONElementEb+0x84)
[0x55a864]
/usr/local/bin/mongod(_ZN5mongo13FieldRangeSetC1EPKcRKNS_7BSONObjEbb+0x194)
[0x55b554]
/usr/local/bin/mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKN S_11BSONElementEbS5_S5_bb+0x1f5)
[0x8e07e5]
/usr/local/bin/mongod(_ZN5mongo11MultiCursorC1EPKcRKNS_7BSONObjES5_N5boost1 0shared_ptrINS0_8CursorOpEEEbb+0x138)
[0x8e13c8]
/usr/local/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES2_bbbRNS _7OpDebugEPNS_11RemoveSaverEb+0x38e)
[0x96026e]
/usr/local/bin/mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES2_bbbRNS_7 OpDebugEb+0x13d)
[0x96501d]
/usr/local/bin/mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x 474)
[0x88ce24]
/usr/local/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResp onseERKNS_11HostAndPortE+0x116f)
[0x88e7cf]
/usr/local/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_ 21AbstractMessagingPortEPNS_9LastErrorE+0x78)
[0xaa0b38]
/usr/local/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)
[0x638767]
/lib64/libpthread.so.0 [0x3ac5e0677d]
/lib64/libc.so.6(clone+0x6d) [0x3ac52d325d]
Logstream::get called in uninitialized state
Thu Sep 13 11:18:13 [conn79] ERROR: Client::~Client _context should be null
but is not; client:conn
Logstream::get called in uninitialized state
Thu Sep 13 11:18:13 [conn79] ERROR: Client::shutdown not called: conn
On Thursday, September 13, 2012 5:55:57 PM UTC-7, Stephen Steneker wrote:
> We're using Mongo 2.0.7 on Linux x86 64 bits. We were preparing for a test
>> by loading data into mongo using upserts from a single client to a single
>> mongod. Suddenly mongod crashed with a segmentation fault. After reviewing
>> the logs the error is likely due to mongod running out of diskspace. Yet I
>> wonder why the mongo server didn't switch to a read-only mode instead of
>> crashing? Isn't there such mode?
> Hi Guillaume,
> Can you post the specific segfault / stacktrace message .. it should have
> more context on what caused the crash.
>> Also, had we been running a replica set with at least one secondary with
>> a lot more disk space, would we have been able to continue inserting data?
> Yes .. with a node down in a replica set, you would be able to continue
> inserting data assuming that:
> - not including the secondary which runs of disk space and is "down", you
> still have a majority of the nodes in the replica set available
> - any "write concerns" you require in your application code are still
> satisfied:
> http://docs.mongodb.org/manual/applications/replication/#replica-set-...
> Cheers,
> Stephen
