Re: MongoDB corrupt db?

[Asya Kamsky]

Alex,

Sorry you ran into this problem.

What version of 2.4 is this?  Is the data directory on local storage?

It looks like the problem may be in indexes - do you have the most recent copy of the (corrupt) db?  You can try dropping indexes and seeing if that resolves the problem.  If you are determining that you lost documents by querying for them (and the index is corrupt) then the documents might very well still be there.

Did you try running validate(true) on the DB?

http://docs.mongodb.org/manual/reference/command/validate/

Asya

On Tuesday, May 7, 2013 6:25:13 PM UTC-7, alex wrote:

Hi,

I am using mongodb 2.4 in standalone mode monitored with mms and I had some problems about durability

First I got some errors messages in my log : (seems to be during a background indexing)

Tue May  7 18:11:08.473 [conn5685] build index prod.users { did: 1 }

Tue May  7 18:11:08.479 [conn5685] Assertion: 14045:missing Extra

0xdced21 0xd903eb 0x8d7567 0xaae541 0xabac7a 0xabed32 0xac0a9f 0x9ed451 0x9eeeee 0x9f56de 0x6f1640 0xdbb53e 0x7f9992569e9a 0x7f999187ccbd 

 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdced21]

 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd903eb]

 /usr/bin/mongod(_ZN5mongo16NamespaceDetails3idxEib+0x1e7) [0x8d7567]

 /usr/bin/mongod(_ZN5mongo21IndexBuildsInProgress6removeEPKci+0x81) [0xaae541]

 /usr/bin/mongod(_ZN5mongo16insert_makeIndexEPNS_16NamespaceDetailsERKSsRKNS_7DiskLocEb+0x97a) [0xabac7a]

 /usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbbPb+0x7d2) [0xabed32]

 /usr/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEbb+0x4f) [0xac0a9f]

 /usr/bin/mongod(_ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE+0xd1) [0x9ed451]

 /usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0xa1e) [0x9eeeee]

 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xb0e) [0x9f56de]

 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x90) [0x6f1640]

 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x42e) [0xdbb53e]

 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f9992569e9a]

 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f999187ccbd]

Tue May  7 18:11:08.482 [conn5685] insert prod.system.indexes keyUpdates:0 exception: missing Extra code:14045 locks(micros) w:9604 9ms

then several messages about corrupt db? :

Tue May  7 18:12:15.371 [initandlisten] connection accepted from 127.0.0.1:52003 #5692 (6 connections now open)

Tue May  7 18:12:21.874 [conn5690] getFile(): n=-2

Tue May  7 18:12:21.874 [conn5690] Assertion: 10295:getFile(): bad file number value (corrupt db?): run repair

0xdced21 0xd903eb 0x8cc113 0xb82b72 0xa5ddae 0xa8834b 0xa8d007 0x9f0143 0x9f56ee 0x6f1640 0xdbb53e 0x7f9992569e9a 0x7f999187ccbd 

 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdced21]

 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd903eb]

 /usr/bin/mongod(_ZN5mongo8Database7getFileEiib+0x393) [0x8cc113]

 /usr/bin/mongod(_ZNK5mongo7DiskLoc3recEv+0x42) [0xb82b72]

 /usr/bin/mongod(_ZN5mongo25NamespaceDetailsTransient16computeIndexKeysEv+0x28e) [0xa5ddae]

 /usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x38b) [0xa8834b]

 /usr/bin/mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0xa8d007]

 /usr/bin/mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x4d3) [0x9f0143]

 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xb1e) [0x9f56ee]

 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x90) [0x6f1640]

 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x42e) [0xdbb53e]

 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f9992569e9a]

 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f999187ccbd]

and all my write queries failed : 

Tue May  7 18:12:21.878 [conn5690] update prod.users query: { ... } update: { $set: { ... } } keyUpdates:0 exception: getFile(): bad file number value (corrupt db?): run repair code:10295 locks(micros) w:3622 3ms

I didn't found a message about "Unclean shutdown detected." in my log.

I tried to repair database with mongo shell:

use prod

db.repairDatabase()

everything seemed ok, but I lost a lot of my prod.users data 

As I do a backup each hour, I tried to mongorestore my backup.

All my backups created after the first corrupted message failed during mongorestore with : 

Error creating index 14045 err: "missing Extra"

I finally succeeded to restore an older backup, created before the first corrupt message.

Using standalone mode, how can I prevent this behavior.

I thought journaling and a mongodump each hour will be efficient to protect me against this, but it seems not to be the right option.

And I am looking for info about my first error message : Assertion: 14045:missing Extra.

Thanks.

[alex]

Hi,

I am using mongodb 2.4.2.

I reproduced the same problem, and it was due to a bad index creation: due to the creation of a unique index on a non unique attribute.

After deleting all indexes on my collection,  I was able to write again on my db.

Then I removed the unique attribute and I was able to create my index on my did attribute without any "corrupt db?" message.

[Daniel Hodgin]

The corrupt DB issue I was having on a development replica set I also traced back to having something to do with indexes and the point in time that we are creating indexes in an initialization script for sample data.

I've been testing our latest code for a few days to see if the corruption is fixed and we haven't experienced it since changing the order that we create indexes in our initialization script.

I will have to check and see if we were inserting some data that violated an index before we created the index as we were asynchronously building indexes at the same time as inserting sample data.

My errors were the same as above.

This is very helpful. I will check our initialization script tomorrow.

[Yasuki Okumura]

FYI,

This bug(https://jira.mongodb.org/browse/SERVER-12481) cause this problem and has fixed in 2.4.10.

The above link says it happen only when using unique index but I think it happen when you attempt to create 10th index and the creating index fails by any reason such as limit of key length.

I could reproduce it in 2.4.9 by following:

mongos> db.c.ensureIndex({a:1});

mongos> db.c.ensureIndex({b:1});

mongos> db.c.ensureIndex({c:1});

mongos> db.c.ensureIndex({d:1});

mongos> db.c.ensureIndex({e:1});

mongos> db.c.ensureIndex({f:1});

mongos> db.c.ensureIndex({g:1});

mongos> db.c.ensureIndex({h:1});

mongos> db.c.insert({i:0});

mongos> db.c.insert({i:0});

mongos> db.c.ensureIndex({i:1}, {name: Array(126).join("a")});

{

        "singleShard" : "vep-db/192.168.3.102:27017,192.168.3.103:27017",

        "err" : "missing Extra",

        "code" : 14045,

        "n" : 0,

        "lastOp" : {

                "t" : 1400667719,

                "i" : 2

        },

        "connectionId" : 498391,

        "ok" : 1

}

mongos> db.c.insert({});

getFile(): bad file number value (corrupt db?): run repair

2013年5月10日金曜日 9時28分24秒 UTC+9 Daniel Hodgin:

[Asya Kamsky]

You are correct that it's possible to get to this situation via other failures, but I want to clarify that the data itself is *not* corrupt - it's only the index catalog that gets effected.

I've made a note that the jira description/note should be amended to include a more complete description of failure scenarios and their effect.

Asya

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/24d97902-8780-4ab6-b07a-32df1bd204a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.