Monitoring for elections in ReplicaSet

[rohit reddy]

Hello,
I have a mongo cluster with two shards and three replica set members each. We are currently running a POC, so our servers are all Virtual Machines and we use mongo Community edition.
Because of the substantial amount of load we put on these VM's, we have observed that there will be a change in the primary of the ReplicaSet members quite often. In other words, as we push these VM's to their maximum network and I/O capabilities, sometimes these the primary node does not respond, causing an election -- and one of the secondary becomes primary. (there will be some rollback data as well in the old primary).

We want to monitor these elections and changes in the primary of a ReplicaSet. To do that currently we need to login to the individual server of a Shard and initiate rs.status command to identify the primary member of Shard.  This works when we have only two shards, as we need to login to maximum of two servers to check which one is the primary.

However, we want to know if we can get the same information from QueryRouter (mongos).  Typically we login to mongos to get all our data, however we are unable to run any rs.**** (replication helper methods) from mongos. Is there any way where we can list the primary members of all the Shard Replica Sets while connecting through mongos?


Thanks
Rohit Reddy

[Kevin Adistambha]

Hi Rohit

Although the mongos process monitors the status of the replica set on each shard, I don’t believe there is a user-accessible method to extract this information from mongos.

Instead of using rs.status(), you might be able to use db.isMaster() instead, which is a more lightweight command. You can also run the command without authentication, which avoids having to authenticate to run rs.status() and keeping the connection open to avoid authentication cost.

Best regards
Kevin

​

[rohit reddy]

Thank you Kevin for your reponse.

Is there a plan to include a user-accessible method to extract replicaset information from mongos?

Thanks
Rohit

​

[Kevin Adistambha]

Hi Rohit

It appears that the command connPoolStats does provide you with this information. You can run this command from a mongos instance, and the shard’s primary would have the field ismaster: true.

For example:

> db.runCommand({connPoolStats:1})
...
"replicaSets": {
    "shard01": {
      "hosts": [
        {
          "addr": "localhost:27018",
          "ok": true,
          "ismaster": true,
          "hidden": false,
          "secondary": false,
          "pingTimeMillis": 0
        },
        {
          "addr": "localhost:27019",
          "ok": true,
          "ismaster": false,
          "hidden": false,
          "secondary": true,
          "pingTimeMillis": 0
        },
        {
          "addr": "localhost:27020",
          "ok": true,
          "ismaster": false,
          "hidden": false,
          "secondary": true,
          "pingTimeMillis": 0
        }
      ]
    },
    "shard02": {
      "hosts": [
        {
          "addr": "localhost:27021",
          "ok": true,
          "ismaster": true,
          "hidden": false,
          "secondary": false,
          "pingTimeMillis": 0
        },
        {
          "addr": "localhost:27022",
          "ok": true,
          "ismaster": false,
          "hidden": false,
          "secondary": true,
          "pingTimeMillis": 0
        },
        {
          "addr": "localhost:27023",
          "ok": true,
          "ismaster": false,
          "hidden": false,
          "secondary": true,
          "pingTimeMillis": 0
        }
      ]
    },
...

From the example output above, the command shows that in shard01 the current primary is localhost:27018, and in shard02 the current primary is localhost:27021.

Best regards
Kevin

​

[rohit reddy]

Kevin,

thanks a lot for finding me this information. This is very helpful.

Regards,

Rohit

[rohit reddy]

Hello Kevin,

I am trying to write a small script in native Mongo Shell which lists all the mongod processes and their respective roles.

I was able to write below script, but i am only able to iterate through array of hosts inside a nested document.

############################Code to iterate through members of one Shard ######################################

db.runCommand({connPoolStats:1}).replicaSets.shard01.hosts.forEach(function(p) {
    if(p.ismaster == true){
        printjson('PRIMARY   of the replicaSet ShardA is :' +p.addr)
    }
    else if(p.secondary == true) {
        printjson('Secondary of the replicaSet ShardA is :' +p.addr)
    }
});

############################################################

I am unable to think of way to iterate through a nested document. Is there is an easy way of doing this inside the mongoShell?

Any help would be greatly appreciated.

Thanks

Rohit Reddy

[Kevin Adistambha]

Hi Rohit

I believe you can use Javascript’s for..in statement to iterate through sub-documents. As an example, in a sharded cluster with 2 shards, 3 node replica set per shard, and a single node replica set as the config server:

connPool = db.runCommand({connPoolStats:1}).replicaSets;
for (var rs in connPool) {
  print(rs);
  replSet = connPool[rs].hosts;
  for (var hs in replSet) {
    print(' - ' + replSet[hs].addr + ' : ' + replSet[hs].ismaster);
  }
}

Running this in the mongo shell yields this output:

shard02
 - localhost:27021 : true
 - localhost:27022 : false
 - localhost:27023 : false
configRepl
 - localhost:27024 : true
shard01
 - localhost:27018 : true
 - localhost:27019 : false
 - localhost:27020 : false

This shows that localhost:27018 is the primary of shard01 and localhost:27021 is the primary of shard02. This was tested in the mongo shell of MongoDB 3.6.5.

Best regards
Kevin

​