listDatabases slow in 2.8.0rc5

[Travis Redman]

I'm evaluating 2.8.0rc5 with one of our largest replica sets (1.63M collections/indexes). I built a test replica using initial sync, and it appears to have worked (counts on source and destination collections all match up, can read data back) but I can't get a list of DBs. "show dbs" or db.runCommand({listDatabases: 1}) both hang, and I can't tell that anything is going on. "show collections" and "db.getCollectionNames()" seem to work fine. Below is a snippet of db.currentOp() showing a listDatabases command running for over 6 hours. I attempted to ctrl+c and kill the op but it continues to run.

I found https://jira.mongodb.org/browse/SERVER-16353, which describes my problem, but it's supposedly fixed as of rc3. Is this something new?

{

                "desc" : "conn32583",

                "threadId" : "0x72e3948a0",

                "connectionId" : 32583,

                "opid" : 27153911,

                "active" : true,

                "secs_running" : 23699,

                "microsecs_running" : NumberLong("23699290754"),

                "op" : "query",

                "ns" : "admin.$cmd",

                "query" : {

                        "listDatabases" : 1

                },

                "client" : "127.0.0.1:58167",

                "killPending" : true,

                "numYields" : 0,

                "locks" : {

                        "Global" : "r",

                        "Database" : "r"

                },

                "waitingForLock" : false,

                "lockStats" : {

[Travis Redman]

The log seems to indicate that this thread is repeatedly looking up metadata for various collections. Example:

2015-01-20T00:43:23.774+0000 D STORAGE  [conn32583] looking up metadata for: <db>.<collection> @ RecordId(181676)

[Asya Kamsky]

Hi Travis:

How many databases are there in this replica set?

Asya

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/abe6ea74-ac71-4f73-9674-d1d0bbb2d1c0%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Travis Redman]

There are 142.

Thanks,

Travis

[Travis Redman]

Hi Asya,

Were you able to find any information about this? It looks like having a large number of collections/indexes also severely impacts startup time. I have logs showing startup taking 7 hours. This is also happening on 3.0rc6. During startup, it looks the time is spent walking all of the collection files:

2015-01-28T01:08:02.608+0000 D STORAGE  [initandlisten] looking up metadata for: <db>.<collection1> @ RecordId(349191)

2015-01-28T01:08:02.644+0000 D STORAGE  [initandlisten] doing scan of collection <db>.<collection1> to get info

2015-01-28T01:08:02.644+0000 D STORAGE  [initandlisten] looking up metadata for: <db>.<collection2> @ RecordId(349247)

2015-01-28T01:08:02.687+0000 D STORAGE  [initandlisten] looking up metadata for: <db>.<collection3> @ RecordId(349250)

2015-01-28T01:08:02.724+0000 D STORAGE  [initandlisten] looking up metadata for: <db>.<collection4> @ RecordId(349190)

2015-01-28T01:08:02.760+0000 D STORAGE  [initandlisten] doing scan of collection <db>.<collection4> to get info

Thanks,

Travis