Re: [mongodb-user] Database unclean shutdown

[Sam Millman]

This is also on the stack: http://stackoverflow.com/questions/11968775/mongodb-unclean-shutdown-fail-to-repair

On 15 August 2012 11:12, Ferry <ferry....@aitindo.com> wrote:

I was running MongoDB on Amazon AWS, until one day it ran out of space. Then, the database is completely inaccessible,and I tried to shutdown the database. Having restarted the server and installed larger disk, the MongoDB can't be started.

I tried to repair the database, and the error is shown like below:

Wed Aug 15 10:08:04 [initandlisten] MongoDB starting : pid=1447 port=27017 dbpath=/var/lib/mongodb/ 32-bit 

** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data

**       see http://blog.mongodb.org/post/137788967/32-bit-limitations

**       with --dur, the limit is lower

Wed Aug 15 10:08:04 [initandlisten] db version v1.8.3, pdfile version 4.5
Wed Aug 15 10:08:04 [initandlisten] git version: c206d77e94bc3b65c76681df5a6b605f68a2de05
Wed Aug 15 10:08:04 [initandlisten] build sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37
Wed Aug 15 10:08:04 [initandlisten] ****
Wed Aug 15 10:08:04 [initandlisten] ****
Wed Aug 15 10:08:04 [initandlisten] need to upgrade database md with pdfile version 559903.1345021735, new version: 4.5
Wed Aug 15 10:08:04 [initandlisten] starting upgrade
Wed Aug 15 10:08:04 [initandlisten]  md repairDatabase md
Wed Aug 15 10:08:04 [initandlisten]  md.system.namespaces Assertion failure isOk() db/pdfile.h 259
0x817c27b 0x819121e 0x83cf525 0x83d6bec 0x83f5420 0x82a4fb7 0x83f8680 0x83fcece 0x8401358 0x8401e29 0x8402abd 0x828f755 0x83b3d67 0x83b931c 0x83bb3df 0x81ce25f 0x81ae714 0x8380f31 0x83831ba 0x83d4faf 
 mongod(_ZN5mongo12sayDbContextEPKc+0xcb) [0x817c27b]
 mongod(_ZN5mongo8assertedEPKcS1_j+0x12e) [0x819121e]
 mongod(_ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE+0x7a5) [0x83cf525]
 mongod(_ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE+0x13c) [0x83d6bec]
 mongod(_ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi+0x830) [0x83f5420]
 mongod(_ZN5mongo11UserQueryOp5_initEv+0x437) [0x82a4fb7]
 mongod(_ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE+0x110) [0x83f8680]
 mongod(_ZN5mongo12QueryPlanSet6Runner3runEv+0x2ee) [0x83fcece]
 mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x2e8) [0x8401358]
 mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x59) [0x8401e29]
 mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x2d) [0x8402abd]
 mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1575) [0x828f755]
 mongod() [0x83b3d67]
 mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x70c) [0x83b931c]
 mongod(_ZN5mongo14DBDirectClient4callERNS_7MessageES2_bPSs+0x7f) [0x83bb3df]
 mongod(_ZN5mongo14DBClientCursor4initEv+0x15f) [0x81ce25f]
 mongod(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0x2a4) [0x81ae714]
 mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x451) [0x8380f31]
 mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x5a) [0x83831ba]
 mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x38f) [0x83d4faf]
Wed Aug 15 10:08:04 [initandlisten] assertion 0 assertion db/pdfile.h:259 ns:md.system.namespaces query:{}
Wed Aug 15 10:08:04 [initandlisten] Assertion: 10290:bad system.namespaces object { $err: "assertion db/pdfile.h:259" }
0x819045e 0x8381181 0x83831ba 0x83d4faf 0x852a30c 0x852b217 0x852e733 0x852f59c 0x85366bb 0xb74a4ce7 0x810b551 
 mongod(_ZN5mongo11msgassertedEiPKc+0x15e) [0x819045e]
 mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x6a1) [0x8381181]
 mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x5a) [0x83831ba]
 mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x38f) [0x83d4faf]
 mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x6c) [0x852a30c]
 mongod() [0x852b217]
 mongod(_ZN5mongo14_initAndListenEiPKc+0x4b3) [0x852e733]
 mongod(_ZN5mongo13initAndListenEiPKc+0x2c) [0x852f59c]
 mongod(main+0x6bdb) [0x85366bb]
 /lib/libc.so.6(__libc_start_main+0xe7) [0xb74a4ce7]
 mongod() [0x810b551]
Wed Aug 15 10:08:04 [initandlisten] exception in initAndListen std::exception: bad system.namespaces object { $err: "assertion db/pdfile.h:259" }, terminating
Wed Aug 15 10:08:04 dbexit: 
Wed Aug 15 10:08:04 [initandlisten] shutdown: going to close listening sockets...
Wed Aug 15 10:08:04 [initandlisten] shutdown: going to flush diaglog...
Wed Aug 15 10:08:04 [initandlisten] shutdown: going to close sockets...
Wed Aug 15 10:08:04 [initandlisten] shutdown: waiting for fs preallocator...
Wed Aug 15 10:08:04 [initandlisten] shutdown: closing all files...
Wed Aug 15 10:08:04 closeAllFiles() finished
Wed Aug 15 10:08:04 [initandlisten] shutdown: removing fs lock...
Wed Aug 15 10:08:04 dbexit: really exiting now

I have all *.number and *.ns file completely. But it seems the ns file is corrupted. Any way I could repair the database?

Thanks 

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb

[markh]

Hi Ferry,

I answered this on Stack Overflow (I saw it there first).

Mark