Mongo 3.0 lot of operations waiting for lock.

27 views

Skip to first unread message

vaibhav singh

unread,

Nov 7, 2017, 11:37:59 AM11/7/17

to mongodb-user

Hi,

We have a mongodb cluster version 2.6 . We want to upgrade to latest version of mongo. As we cant jump to the version 3.4 directly. We added a 3.0 secondary in 2.6 replicaset and sent the traffic to this new 3.0 secondary from one of the 2.6 secondary using flashback . We see sudden drop in QPS and during that low QPS period replication threads hold locks.

We didnt see this locking issue when we did the same test on 3.0 standalone node. We used mongooplog tool to keep this standalone node in sync with 2.6 cluster and used flashback to send traffic from one of the secondary node of 2.6 cluster to 3.0 standalone node. We tested on standalone 3.2 and 3.4 as well. But we didnt see this locking issue.

Kevin Adistambha

unread,

Nov 16, 2017, 12:36:42 AM11/16/17

to mongodb-user

We added a 3.0 secondary in 2.6 replicaset and sent the traffic to this new 3.0 secondary from one of the 2.6 secondary using flashback . We see sudden drop in QPS and during that low QPS period replication threads hold locks.

If I understand correctly, you are adding a 3.0 secondary in an existing replica set, and you are measuring the 3.0 secondary performance while it was still acting as a secondary. If my understanding is correct, this is expected, since a secondary applies the oplog in batches. While a batch is being applied, it is possible to see inconsistent documents, thus the lock is necessary to avoid this.

We didnt see this locking issue when we did the same test on 3.0 standalone node…. We tested on standalone 3.2 and 3.4 as well. But we didnt see this locking issue

This is because you are testing a different thing. In this case, the nodes are not a secondary trying to replicate writes that’s happening to the primary as quickly as possible.

If you need to do a performance testing, I would suggest you to create a separate 3.0/3.2/3.4 replica set, and do load testing on the set as a whole, instead of on a single secondary attached to an existing replica set. This would be closer to how the set is deployed in the future, so I imagine the result would also be more realistic.