MongoDB Failover is not working using YCSB

[mahen...@evolvussolutions.com]

Hi Team,

Just few days back onwards, we started doing some performance testing on MongoDB. Everything is working as expect failover. We have tried following options.

Could you please let us know if any other parameters needs to send via ycsb command line for automatically perform write/read operations while primary is down.

1st option

We have taken 2 AWS MongoDB instances one primary and one secondary. We have created replicaset for validating failover case.

If Primary is down it should automatically elect secondary as primary and perform the operations. However it is not happening from YCSB tool with command line. Below is the command line which we are using.

./bin/ycsb load mongodb -P workloads/workloada -p recordcount=750000 -threads 20 -p mongodb.url=mongodb://username:pass...@mongodbinstance1.com:27017,mongodbinstance2.com:27017/dbname?replicaSet=rep1&autoReconnect=true

2nd option

We have taken 3 machines and created replicaset. Once primary is down, behind electing secondary is happening, however writing to db is not happening.

Here is the command which we have used while validating in local machines:

./bin/ycsb load mongodb-async -s -P workloads/workloada -p recordcount=750000 -threads 20 -p mongodb.url=mongodb://username:password@machineoneIP:27017,machinetwoIP:27017,machinethreeIP:27017/mongotag?replicaSet=rep1&autoReconnect=true&&serverSelectionTimeoutMS=10&serverSelectionTryOnce=true&readPreference=primary&w=2 

We have got following errors:

1. com.mongodb.MongoCommandException: Command failed with error 11600: 'interrupted at shutdown' on server 192.168.1.118:27017. The full response is { "operationTime" : { "$timestamp" : { "t" : 1534227554, "i" : 2423 } }, "ok" : 0.0, "errmsg" : "interrupted at shutdown", "code" : 11600, "codeName" : "InterruptedAtShutdown", "$clusterTime" : { "clusterTime" : { "$timestamp" : { "t" : 1534227554, "i" : 2423 } }, "signature" : { "hash" : { "$binary" : "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "$type" : "00" }, "keyId" : { "$numberLong" : "0" } } } }

at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:164)

 2. com.mongodb.MongoWriteException: Not primary while writing to mongotag.usertablea com.mongodb.MongoCollectionImpl.executeSingleWriteRequest(MongoCollectionImpl.java:1033)

INFO: Closed connection [connectionId{localValue:5, serverValue:34}] to 192.168.1.60:27017 because there was a socket exception raised on another connection from this pool.

Exception while trying bulk insert with 0

com.mongodb.MongoNotPrimaryException: The server is not the primary and did not execute the operation

at com.mongodb.connection.ProtocolHelper.createSpecialException(ProtocolHelper.java:223)

Thank you,

Mahendra

DISCLAIMER “This email message, including any of its attachments, is intended solely for the addressee(s) and may contain privileged information. If you are not the addressee or you have received this email message in error, please notify the sender and delete the email, destroying any hard copies of the original e-mail message, and the sender will remove your details from their database. You are not authorized to read, copy, disseminate, distribute or use this e-mail message or any attachment to it in any manner. This e-mail message does not contain financial instructions or commitments of any kind. Any views expressed in this message are those of the individual sender and do not necessarily reflect the views of Evolvus Solutions Pvt. Ltd. or any other related subsidiaries, entities or persons.”

Carbon footprint – please do not print this e-mail unless you really need to.

[lian....@mongodb.com]

Hi Mahendra,

With a two node member replica set (one primary and one secondary), there is no automatic failover if the primary goes down. The way automatic failover works in MongoDB is that a replica set needs a qualified majority to successfully elect a new primary. Alternatively, introduce an arbiter into the replica set. Arbiters do not hold data, their purpose is to participate in elections in order to break ties in the election process. If a replica set has an even number of members, recommend to add an arbiter. The minimum recommended replica set deployment for production system is a three member replica set, a primary and two secondaries. These sets provide redundancy and fault tolerance.

For more information see:

Replica Set Elections

Replica Set Deployment Architectures

The primary is the only member in the replica set that receives write operations from client applications.

The secondary node does not accept write operations from clients, but only accepts replication writes from the primary. Clients can read data from secondary members. See Read Preference for more information on how clients direct read operations to replica sets.

You can set the readPreference in the connection string for the clients to read from the secondary (instead of the primary). For example:

mongodb.url=mongodb://host.example.com/?readPreference=secondary

In your case, as the primary is down and the secondary has not become primary, it is not possible to continue the client application writes to a secondary member (which is in process of election and has not become primary) in the replica set. This is confirmed by the errors seen from the ycsb command output:

com: mongodb.MongoNotPrimaryException: The server is not the primary and did not execute the operation.

Regards,

Lian

​