Script to count number of Collections & Records for each Replica Set member

505 views
Skip to first unread message

A. Jalil @AJ

unread,
Sep 26, 2015, 11:28:30 PM9/26/15
to mongodb-user
Hello,

I have migrated 3 nodes successfully to AWS, on the status below I don't see any lags which is good, but how do I validate each member so I can compare it with the older member to make sure all data is in sync .
I would like to check the number of Collections in each member and the number of records in each collection, but I don't want to go thru each collection and count, I might have over 500 collections.. I was wondering if there is script I can use to generate such a report.. 

I know I can run the commands below, but that only lists all collections.  I am trying to get COUNTS of collections instead, and I don't want to go thru one collection a the time.. If my database is called HR, I want count of all HR collections & count of all records of each HR collection by using a script hopefully..

Something similar to this report:

HR DB
Number of collections    =>  500

Collections
payroll                          =>  200,000  Doc
employees                    =>  150,000 Doc
Dept                             =>  50  Doc
NewHire                        =>  10,000  Doc 


db.collectionName.find()


db.getCollectionNames()


show collections





rs0:PRIMARY> rs.status()
{
        "set" : "rs0",
        "date" : ISODate("2015-09-27T01:26:18Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 8,
                        "name" : "server-1-aws.com:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 3092,
                        "optime" : Timestamp(1443314263, 1),
                        "optimeDate" : ISODate("2015-09-27T00:37:43Z"),
                        "lastHeartbeat" : ISODate("2015-09-27T01:26:18Z"),
                        "lastHeartbeatRecv" : ISODate("2015-09-27T01:26:17Z"),
                        "pingMs" : 1,
                        "syncingTo" : "server-2-aws.com:27017"
                },
                {
                        "_id" : 9,
                        "name" : "server-2-aws.com:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 448311,
                        "optime" : Timestamp(1443314263, 1),
                        "optimeDate" : ISODate("2015-09-27T00:37:43Z"),
                        "electionTime" : Timestamp(1443313454, 1),
                        "electionDate" : ISODate("2015-09-27T00:24:14Z"),
                        "self" : true
                },
                {
                        "_id" : 10,
                        "name" : "server-3-aws.com:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 2915,
                        "optime" : Timestamp(1443314263, 1),
                        "optimeDate" : ISODate("2015-09-27T00:37:43Z"),
                        "lastHeartbeat" : ISODate("2015-09-27T01:26:18Z"),
                        "lastHeartbeatRecv" : ISODate("2015-09-27T01:26:17Z"),
                        "pingMs" : 1,
                        "syncingTo" : "server-2-aws.com:27017"
                }
        ],
        "ok" : 1
}


Thank you !
@AJ

Stephen Steneker

unread,
Sep 27, 2015, 8:08:17 PM9/27/15
to mongodb-user

On Sunday, 27 September 2015 13:28:30 UTC+10, A. Jalil @AJ wrote:

I have migrated 3 nodes successfully to AWS, on the status below I don’t see any lags which is good, but how do I validate each member so I can compare it with the older member to make sure all data is in sync .
I would like to check the number of Collections in each member and the number of records in each collection, but I don’t want to go thru each collection and count, I might have over 500 collections.. I was wondering if there is script I can use to generate such a report..

Hi AJ,

To iterate all collections or databases you can write forEach() loops in the mongo shell.

Rather than counting each collection, it is probably easier to compare the overall stats for all databases (which includes number of collections & objects):

db.adminCommand("listDatabases").databases.forEach(
    function (dblist) {
        dbcheck = db.getSiblingDB(dblist.name);
        // Could optionally iterate dbcheck.collectionNames() to get per-collection counts or stats
        printjson(dbcheck.stats());
    }
)

You could add an inner loop to iterate or count all collections if you prefer to compare at that level.

For easier comparison I would save the JavaScript code in a file (eg alldbstats.js) and redirect the output for each member of the replica set:

mongo --quiet alldbstats.js node1:27017 > node1.json

… and then use a tool like diff to compare the JSON files (or jsondiffpatch if you want to get fancy).

You could also do all of the comparison by iterating the replica set in the mongo shell, but that’s probably overkill. For more information see: Write Scripts for the mongo Shell.

Regards,
Stephen

​

A. Jalil @AJ

unread,
Sep 27, 2015, 11:04:48 PM9/27/15
to mongodb-user
Thank you so much Stephen ! This is very helpful indeed !


Reply all
Reply to author
Forward
0 new messages