Increasing open session cursor count in 3.0.4

[Mike Bartlett]

Hi folks,

Yesterday we experienced a very unusual situation where the CPU on our 3.0.4 (WiredTiger/Snappy) raced up to 100%+ for a sustained period of time. Stepping the box down resulted in a new election and the new primary's CPU utilisation was just fine.

Upon investigation we noticed that the underlying disc read operations increased substantially, so looks like Mongo was constantly going to disc for some reason. We are pretty positive memory utilisation and oplog size are just fine.

One thing we did notice, and are wondering if it's related, is the following:

TroupeReplicaSet:PRIMARY> db.serverStatus().wiredTiger.session

{ "open cursor count" : 32797, "open session count" : 144 }

TroupeReplicaSet:PRIMARY> db.serverStatus().wiredTiger.session

{ "open cursor count" : 32809, "open session count" : 144 }

This number just continues to increase upwards. Could it be the cause and any idea why that would be creeping up continuously? 

Cheers

Mike

[Alexander Gorrod]

Hi Mike,

The growing cursor count is generally a consequence of MongoDB caching WiredTiger cursors. It does not necessarily indicate a bug in MongoDB. We would need more information to diagnose the symptoms you are reporting.

There is currently a ticket in JIRA that is discussing symptoms similar to those you report:

https://jira.mongodb.org/browse/SERVER-19485

I suggest watching that thread. If it happens again gathering a full db.serverStatus while the instance is demonstrating the behavior would be useful.

[Mike Bartlett]

Thanks Alexander,

I've added a pretty verbose and detailed comment to https://jira.mongodb.org/browse/SERVER-19485

Happy to talk more if you want to get in touch any other way.

Cheers

Mike

[Alexander Gorrod]

We will respond in the JIRA ticket - watch there for updates.

- Alex