> MongoDb does not startup anymore.

> root@ip-10-144-73-149:/ebs/trivian/scripts# sudo mongod
> --dbpath=/ebs/trivian/mongodb/data --quiet --nohttpinterface --profile=2
> --slowms=100 --journal --cpu --fork --bind_ip 127.0.0.1,10.144.73.149
> --logpath=/ebs/trivian/mongodb/logs/mongodb.log
> root@ip-10-144-73-149:/ebs/trivian/scripts# forked process: 5054
> all output going to: /ebs/trivian/mongodb/logs/mongodb.log

> root@ip-10-144-73-149:/ebs/trivian/scripts# cat
> /ebs/trivian/mongodb/logs/mongodb.log
> Tue Jul  3 11:50:14 [initandlisten] MongoDB starting : pid=5054 port=27017
> dbpath=/ebs/trivian/mongodb/data 64-bit host=ip-10-144-73-149
> Tue Jul  3 11:50:14 [initandlisten] db version v2.0.3, pdfile version 4.5
> Tue Jul  3 11:50:14 [initandlisten] git version:
> 05bb8aa793660af8fce7e36b510ad48c27439697
> Tue Jul  3 11:50:14 [initandlisten] build info: Linux ip-10-110-9-236
> 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64
> BOOST_LIB_VERSION=1_41
> Tue Jul  3 11:50:14 [initandlisten] options: { bind_ip:
> "127.0.0.1,10.144.73.149", cpu: true, dbpath: "/ebs/trivian/mongodb/data",
> fork: true, journal: true, logpath:
> "/ebs/trivian/mongodb/logs/mongodb.log", nohttpinterface: true, profile: 2,
> quiet: true, slowms: 100 }
> Tue Jul  3 11:50:14 [initandlisten] journal
> dir=/ebs/trivian/mongodb/data/journal
> Tue Jul  3 11:50:14 [initandlisten] recover : no journal files present, no
> recovery needed
> Tue Jul  3 11:50:14 [initandlisten] profile: warning ns
> local.system.profile does not exist
> Tue Jul  3 11:50:14 [initandlisten] ****
> Tue Jul  3 11:50:14 [initandlisten] ****
> Tue Jul  3 11:50:14 [initandlisten] need to upgrade database trivian with
> pdfile version 5.67108864, new version: 4.5
> Tue Jul  3 11:50:14 [initandlisten]      Not upgrading, exiting
> Tue Jul  3 11:50:14 [initandlisten]      run --upgrade to upgrade dbs,
> then start again
> Tue Jul  3 11:50:14 [initandlisten] ****
> Tue Jul  3 11:50:14 dbexit:
> Tue Jul  3 11:50:14 [initandlisten] shutdown: going to close listening
> sockets...
> Tue Jul  3 11:50:14 [initandlisten] shutdown: going to flush diaglog...
> Tue Jul  3 11:50:14 [initandlisten] shutdown: going to close sockets...
> Tue Jul  3 11:50:14 [initandlisten] shutdown: waiting for fs
> preallocator...
> Tue Jul  3 11:50:14 [initandlisten] shutdown: lock for final commit...
> Tue Jul  3 11:50:14 [initandlisten] shutdown: final commit...
> Tue Jul  3 11:50:14 [initandlisten] shutdown: closing all files...
> Tue Jul  3 11:50:14 [initandlisten] closeAllFiles() finished
> Tue Jul  3 11:50:14 [initandlisten] journalCleanup...
> Tue Jul  3 11:50:14 [initandlisten] removeJournalFiles
> Tue Jul  3 11:50:14 [initandlisten] shutdown: removing fs lock...
> Tue Jul  3 11:50:14 dbexit: really exiting now
> Logstream::get called in uninitialized state
> Tue Jul  3 11:50:14 ERROR: Client::~Client _context should be null but is
> not; client:initandlisten
> root@ip-10-144-73-149:/ebs/trivian/scripts# ps -ef | grep mongo
> root      4990  4793  0 11:28 pts/2    00:00:00 sudo mongo
> root      4991  4990  0 11:28 pts/2    00:00:00 mongo
> root      5062  4436  0 11:50 pts/0    00:00:00 grep --color=auto mongo

> On Tuesday, 3 July 2012 14:47:46 UTC+3, Markku wrote:

>> We are using large ec2 instance and been importing (.update()) open
>> street map places to mongodb. Suddenly mongodb hanged. Only find().count()
>> operation seems to work. Everything else gives error.

>> > db.repairDatabase();
>> {
>>         "errmsg" : "clone failed for trivian with error: query failed
>> trivian.system.namespaces",
>>         "ok" : 0
>> }

>> > db.places.validate()
>> {
>>         "ns" : "trivian.places",
>>         "errmsg" : "exception: assertion db/pdfile.h:300",
>>         "code" : 0,
>>         "ok" : 0,
>>         "valid" : false
>> }

>> > db.places.find().count()
>> 6706729

>> root@ip-10-144-73-149:/ebs/trivian/scripts# sudo mongo --version
>> MongoDB shell version: 2.0.3

>> /ebs/trivian/mongodb/logs/mongodb.log

>> Tue Jul  3 11:24:45 [snapshotthread] cpu: elapsed:4000  writelock: 0%
>> Tue Jul  3 11:24:46 [conn17]  trivian.system.namespaces Assertion failure
>> isOk() db/pdfile.h 300
>> 0x57a636 0x5854eb 0x8a7b2a 0x8a7ed6 0x8cf301 0x94c093 0x8cbc7a 0x8cc093
>> 0x8cc59c 0x8d5d2e 0x8dd7be 0x8df1d3 0x8e04a7 0x943a49 0x8869d7 0x88df49
>> 0xaa37d6 0x637497 0x7fe1702cfe9a 0x7fe16f7ed4bd
>>  mongod(_ZN5mongo12sayDbContextEPKc+0x96) [0x57a636]
>>  mongod(_ZN5mongo8assertedEPKcS1_j+0xfb) [0x5854eb]
>>  mongod(_ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE+0x5fa) [0x8a7b2a]
>>  mongod(_ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE+0xe6)
>> [0x8a7ed6]
>>  mongod(_ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi+0x3a1) [0x8cf301]
>>  mongod(_ZN5mongo11UserQueryOp5_initEv+0x763) [0x94c093]
>>  mongod(_ZN5mongo7QueryOp4initEv+0x11a) [0x8cbc7a]
>>  mongod(_ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE+0x23)
>> [0x8cc093]
>>  mongod(_ZN5mongo12QueryPlanSet6Runner4initEv+0x2ec) [0x8cc59c]
>>  mongod(_ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv+0x1e)
>> [0x8d5d2e]
>>  mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x11e) [0x8dd7be]
>>  mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x523)
>> [0x8df1d3]
>>  mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x17) [0x8e04a7]
>>  mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x a79)
>> [0x943a49]
>>  mongod() [0x8869d7]
>>  mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11Hos tAndPortE+0x559)
>> [0x88df49]
>>  mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessa gingPortEPNS_9LastErrorE+0x76)
>> [0xaa37d6]
>>  mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x637497]
>>  /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fe1702cfe9a]
>>  /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe16f7ed4bd]
>> Tue Jul  3 11:24:46 [conn17] assertion 0 assertion db/pdfile.h:300
>> ns:trivian.system.namespaces query:{}
>> Tue Jul  3 11:24:46 [conn17] profile: warning ns trivian.system.profile
>> does not exist
>> Tue Jul  3 11:24:49 [snapshotthread] cpu: elapsed:4000  writelock: 0%
>> Tue Jul  3 11:24:51 [conn16] Assertion: 10320:BSONElement: bad type 96
>> 0x584432 0x5072bd 0x502112 0x502329 0x8d0da5 0x8d3fdb 0x8db769 0x8ddac7
>> 0x8e0968 0x943692 0x8869d7 0x88df49 0xaa37d6 0x637497 0x7fe1702cfe9a
>> 0x7fe16f7ed4bd
>>  mongod(_ZN5mongo11msgassertedEiPKc+0x112) [0x584432]
>>  mongod(_ZNK5mongo11BSONElement4sizeEv+0x1bd) [0x5072bd]
>>  mongod(_ZN5mongo20anyElementNamesMatchERKNS_7BSONObjES2_+0x92) [0x502112]
>>  mongod(_ZNK5mongo9IndexSpec12_suitabilityERKNS_7BSONObjES3_+0x19)
>> [0x502329]
>>  mongod(_ZN5mongo16QueryUtilIndexed11indexUsefulERKNS_17FieldRangeSetPairEPN S_16NamespaceDetailsEiRKNS_7BSONObjE+0x165)
>> [0x8d0da5]
>>  mongod(_ZN5mongo12QueryPlanSet13addOtherPlansEb+0x1fb) [0x8d3fdb]
>>  mongod(_ZN5mongo12QueryPlanSet4initEv+0x14b9) [0x8db769]
>>  mongod(_ZN5mongo12QueryPlanSetC1EPKcSt8auto_ptrINS_17FieldRangeSetPairEES5_ RKNS_7BSONObjES8_bPKNS_11BSONElementEbS8_S8_bb+0x1f7)
>> [0x8ddac7]
>>  mongod(_ZN5mongo16MultiPlanScannerC1EPKcRKNS_7BSONObjES5_PKNS_11BSONElement EbS5_S5_bb+0x2c8)
>> [0x8e0968]
>>  mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x 6c2)
>> [0x943692]
>>  mongod() [0x8869d7]
>>  mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11Hos tAndPortE+0x559)
>> [0x88df49]
>>  mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessa gingPortEPNS_9LastErrorE+0x76)
>> [0xaa37d6]
>>  mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x637497]
>>  /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fe1702cfe9a]
>>  /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe16f7ed4bd]

>> root@ip-10-144-73-149:/ebs/trivian/scripts# ls -la
>> /ebs/trivian/mongodb/data
>> total 4242472
>> drwxr-xr-x 4 root root       4096 Jul  3 11:35 .
>> drwxr-xr-x 4 root root       4096 Jul  2 11:38 ..
>> drwxr-xr-x 2 root root       4096 Jul  3 11:36 journal
>> -rw------- 1 root root   67108864 Jul  2 11:40 local.0
>> -rw------- 1 root root   16777216 Jul  2 11:40 local.ns
>> -rwxr-xr-x 1 root root          5 Jul  3 09:52 mongod.lock
>> -rw------- 1 root root   16777216 Jul  3 10:05 test.ns
>> drwxr-xr-x 2 root root       4096 Jul  3 10:09 _tmp
>> -rw------- 1 root root   67108759 Jul  3 11:22 trivian.0
>> -rw------- 1 root root  134217728 Jul  3 10:17 trivian.1
>> -rw------- 1 root root  268435456 Jul  3 10:17 trivian.2
>> -rw------- 1 root root  536870676 Jul  3 11:23 trivian.3
>> -rw------- 1 root root 1073741824 Jul  3 10:17 trivian.4
>> -rw------- 1 root root 2146435072 Jul  3 10:09 trivian.5
>> -rw------- 1 root root   16777216 Jul  3 11:37 trivian.ns

>> -Markku

>> On Saturday, 9 June 2012 00:23:34 UTC+3, Eliot Horowitz wrote:

>>> Can you send the full mongo log?

>>> On Fri, Jun 8, 2012 at 6:28 AM, Fathi Ghroul
>>> <fathi.ghr...@feelandclic.com> wrote:
>>> > I tried but it not improve.

>>> > 2012/6/8 Nat <nat.lu...@gmail.com>

>>> >> It could be that database is corrupted. Try to repairDatabase() to
>>> see
>>> >> whether it helps or not.
>>> >> -----Original Message-----
>>> >> From: Fathi Ghroul <fathi.ghr...@feelandclic.com>
>>> >> Sender: mongodb-user@googlegroups.com
>>> >> Date: Fri, 8 Jun 2012 02:51:37
>>> >> To: mongodb-user<mongodb-user@googlegroups.com>
>>> >> Reply-To: mongodb-user@googlegroups.com
>>> >> Subject: [mongodb-user] JS Error: uncaught exception: error: { "$err"
>>> :
>>> >>  "assertion db/pdfile.h:223" }

>>> >> Hello,

>>> >> I've just started to learn MongoDb using RockMongo. I added a
>>> database
>>> >> and I'm receiving the following message whenever I'm trying to access
>>> >> the database through Rockmongo.

>>> >> Execute failed:db assertion failure
>>> >> function (){ return db.getCollectionNames(); }

>>> >> Thanks,
>>> >> Fathi

>>> >> --
>>> >> You received this message because you are subscribed to the Google
>>> >> Groups "mongodb-user" group.
>>> >> To post to this group, send email to mongodb-user@googlegroups.com
>>> >> To unsubscribe from this group, send email to
>>> >> mongodb-user+unsubscribe@googlegroups.com
>>> >> See also the IRC channel -- freenode.net#mongodb

>>> >> --
>>> >> You received this message because you are subscribed to the Google
>>> >> Groups "mongodb-user" group.
>>> >> To post to this group, send email to mongodb-user@googlegroups.com
>>> >> To unsubscribe from this group, send email to
>>> >> mongodb-user+unsubscribe@googlegroups.com
>>> >> See also the IRC channel -- freenode.net#mongodb

>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups "mongodb-user" group.
>>> > To post to this group, send email to mongodb-user@googlegroups.com
>>> > To unsubscribe from this group, send email to
>>> > mongodb-user+unsubscribe@googlegroups.com
>>> > See also the IRC channel -- freenode.net#mongodb