Missing indexes after mongorestore [was: databases for QA environment]

[Brian Zambrano]

After a mongodump and mongorestore, I'm missing two indexes on one of my collections:

Here is our production DB:

> db.system.indexes.find()
{ "name" : "_id_", "ns" : "eventbrite.orders", "key" : { "_id" : 1 } }
{ "ns" : "eventbrite.orders", "name" : "_uid_-1", "key" : { "_uid" : -1 } }
{ "ns" : "eventbrite.orders", "name" : "_eid_-1", "key" : { "_eid" : -1 } }
{ "name" : "_id_", "ns" : "eventbrite.recommendable_events", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "eventbrite.non_recommendable_events", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "eventbrite.users", "key" : { "_id" : 1 } }
{ "ns" : "eventbrite.users", "name" : "fb._id_1", "key" : { "fb._id" : 1 } }
{ "ns" : "eventbrite.users", "name" : "events.curr_ids_1", "key" : { "events.curr_ids" : 1 } }

And our QA DB, after a mongorestore:

> db.system.indexes.find()
{ "name" : "_id_", "ns" : "eventbrite.orders", "key" : { "_id" : 1 } }
{ "ns" : "eventbrite.orders", "name" : "_uid_-1", "key" : { "_uid" : -1 } }
{ "ns" : "eventbrite.orders", "name" : "_eid_-1", "key" : { "_eid" : -1 } }
{ "name" : "_id_", "ns" : "eventbrite.recommendable_events", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "eventbrite.non_recommendable_events", "key" : { "_id" : 1 } }
{ "name" : "_id_", "ns" : "eventbrite.users", "key" : { "_id" : 1 } }

So, the last two indexes are not there in the "users" collection.  I've inspected system.indexes.bson after mongodump finished, usind od, and the indexes appear to be there.

BZ

On Tue, Sep 28, 2010 at 6:57 AM, Eliot Horowitz <elioth...@gmail.com> wrote:

That's quite bizarre.
Can you send the output of db.system.indexes.find() and tell us which
make it and which don't?

On Tue, Sep 28, 2010 at 2:19 AM, Brian Zambrano <bri...@gmail.com> wrote:
> Mongodump has been working just fine, and *does* bring over our indexes.
> However, we have one collection where the indexes are strangely missing.
> The only difference I can think of between this collection (with missing
> indexes) and the others is that the indexes were created from the mongo
> shell, where the indexes which survive the dump/restore were created from
> PyMongo.
>
> I'm not even sure how to investigate this.  Has anyone else seen this?
>

[roger]

It looks like you need :
http://jira.mongodb.org/browse/SERVER-808
Roger

[Eliot Horowitz]

Is there anything in the log?
Can you try running mongorestore just on the system.indexes.bson

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>

[Brian Zambrano]

When running:

mongorestore --db eventbrite system.indexes.bson

all seems to go well, the indexes are there as expected.

I didn't see anything abnormal in the logs from our original mongorestore, which simply runs pointed to a directory which has all of the bson files (for each DB):

mongorestore --drop -v  2010-09-28_00h01m.Tuesday/

Since we're not doing this with the --collection option, I don't think we're hitting http://jira.mongodb.org/browse/SERVER-808.

[Eliot Horowitz]

What version are you running?
Can you send the mongo log and the restore log?
Can't reproduce this locally.

No - you wouldn't be hitting SERVER-808

[Brian Zambrano]

Indexes which exist on the mongodump machine:

> db.users.getIndexes()             
[
        {

                "name" : "_id_",
                "ns" : "eventbrite.users",
                "key" : {
                        "_id" : 1
                }

        },
        {

                "ns" : "eventbrite.users",
                "name" : "events.curr_ids_1",
                "key" : {
                        "events.curr_ids" : 1
                }

        },
        {

                "ns" : "eventbrite.users",
                "name" : "fb._id_1",
                "key" : {
                        "fb._id" : 1
                }
        }

]

After the mongorestore, on a different machine:

> db.users.getIndexes()
[
        {

                "name" : "_id_",
                "ns" : "eventbrite.users",
                "key" : {
                        "_id" : 1
                }
        }

]

On Mon, Oct 18, 2010 at 9:38 PM, Eliot Horowitz <elioth...@gmail.com> wrote:

what indexes are missing?
can you send db.users.getIndexes() before and after.
Can we try to keep on-list as well.

On Mon, Oct 18, 2010 at 7:32 PM, Brian Zambrano <bri...@gmail.com> wrote:
> Hi Eliot,
>
> Following up on this a bit late, but this still seems to be happening, where
> one of our collections is missing some indexes after a
> mongodump/mongorestore.
>
> We're using 1.6.3.
>
> The relevant DB/collection is "eventbrite.users".  Looking at this, I think
> the relevant lines start around 1779:
>
> 1779 Mon Oct 18 14:53:10 [conn6] building new index on { fb._id: 1 } for
> eventbrite.users
> 1780 Mon Oct 18 14:53:10 [conn6] done for 0 records 0.008secs
> 1781 Mon Oct 18 14:53:10 [conn6] CMD: drop eventbrite.users
> 1782 Mon Oct 18 14:53:10 [conn6] building new index on { _id: 1 } for
> eventbrite.users
> 1783 Mon Oct 18 14:53:10 [conn6] done for 0 records 0secs
> 1784 Mon Oct 18 14:53:11 [conn6] insert eventbrite.users 160ms
>
> It looks like indexes are added, and then the data is inserted...not sure if
> that is by design or not?
>
> Any help appreciated....thanks!
>
> BZ
> bri...@eventbrite.com
>
>
>
> On Tue, Sep 28, 2010 at 1:46 PM, Eliot Horowitz <elioth...@gmail.com>

[Eliot Horowitz]

Ok - I think i know what's happening.
Looks like system.indexes is getting loaded first - then the collections.
Since you have --drop - when it gets dropped, the indexes go away.
system.indexes is supposed to be last - not sure why its not working.
added a case: http://jira.mongodb.org/browse/SERVER-1975

[Brian Zambrano]

I thought the order of things looked a bit funny....thanks for the help!  We'll keep track of that issue and look forward to seeing it get fixed.

BZ