Frequent listDatabase calls for no known reason

[Chad Kreimendahl]

[Created SERVER-27666 for this]

We're seeing a potentially large and looming performance issue. Given that we have hundreds of databases with thousand of collections, the listDatabase call is rather expensive when using wiredTiger. Fortunately, they fixed a bug in it some time ago, that took it from ~10 seconds to ~1 second. 

However, there appears to be something related to replicasets that is causing the listDatabases command to run, without known reason. Even though we've got the fastest performing solid state nvme drives that exist, iterating over all of the files to get sizes takes considerable time (2 seconds, on average).

1. Why would replica members constantly be asking for "listDatabases". It appears each member asks about once per minute.
2. Assuming a reason for 1, why would it need to *know* the size of the databases, or other info grabbed by listDatabase.
• Could you not make a more simplified version of "listDatabases" that just iterated over the names of databases, and returned near-instantly? And would be used for this purpose in the replicaset communications?
3. As related to this: SERVER-17078:
1. Chad Kreimendahl added a comment - Feb 08 2015 08:36:48 PM GMT-0600

   Is it possible to get the in-memory size vs on-disk size? Would they correlate well? Is size necessary when listing databases?

{ 

    "desc" : "conn2640548", 

    "threadId" : "140220760643328", 

    "connectionId" : NumberInt(2640548), 

    "client" : "replicaMember03s-IP-Address:37135", 

    "active" : true, 

    "opid" : NumberInt(1655614729), 

    "secs_running" : NumberInt(1), 

    "microsecs_running" : NumberLong(1029143), 

    "op" : "command", 

    "ns" : "admin.$cmd", 

    "query" : {

        "listDatabases" : NumberInt(1)

    }, 

    "numYields" : NumberInt(0), 

    "locks" : {

        "Global" : "r", 

        "Database" : "r"

    }, 

    "waitingForLock" : false, 

    "lockStats" : {

        "Global" : {

            "acquireCount" : {

                "r" : NumberLong(98)

            }

        }, 

        "Database" : {

            "acquireCount" : {

                "r" : NumberLong(49)

            }

        }

    }

}

{ 

    "desc" : "conn2640550", 

    "threadId" : "140221014251264", 

    "connectionId" : NumberInt(2640550), 

    "client" : "replicaMember04s-IP-Address:37589", 

    "active" : true, 

    "opid" : NumberInt(1655614735), 

    "secs_running" : NumberInt(0), 

    "microsecs_running" : NumberLong(840545), 

    "op" : "command", 

    "ns" : "admin.$cmd", 

    "query" : {

        "listDatabases" : NumberInt(1)

    }, 

    "numYields" : NumberInt(0), 

    "locks" : {

        "Global" : "r", 

        "Database" : "r"

    }, 

    "waitingForLock" : false, 

    "lockStats" : {

        "Global" : {

            "acquireCount" : {

                "r" : NumberLong(72)

            }

        }, 

        "Database" : {

            "acquireCount" : {

                "r" : NumberLong(36)

            }

        }

    }

}

[Dan Pasette]

Hi Chad,

The only place the replication needs to call listDatabases is during initial sync.  Are you using the mongodb monitoring app?  Is there an agent on that host?  That may be the source of the command.  If yes, you can configure monitoring to not retrieve collection and database stats in the settings panel.

Dan

--
You received this message because you are subscribed to the Google Groups "mongodb-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev+unsubscribe@googlegroups.com.
To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-dev/bfcad05d-0c98-4010-bc16-f5e57ddc0e35%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chad Kreimendahl]

We're not using the mongodb monitoring app. We do have serverdensity running to monitor the databases. However, I disabled serverdensity on all of the hosts in the cluster, and the commands still continued. When I checked the connections, they were never coming from localhost (which is the only way that serverdensity was configured), but were always coming from one of the *other* members of the replicaset.

So quick answer: no... not using your monitoring app, and not the monitoring app we do use.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev...@googlegroups.com.

[Dan Pasette]

I've looked at the code and there aren't calls in repl for this at a regular frequency, and I don't know what else would be calling listDatabases.  If you open a SERVER ticket we can look deeper into the issue.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev+unsubscribe@googlegroups.com.

To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-dev/d96c5ade-1584-4d6f-8d14-1a91ef789bb7%40googlegroups.com.