mongos crash with no log information

[Zhang Lian]

Hi, yesterday, our mongo cluster suddenly crashed, with stack trace as below. 

Our application feature: frequent update and delete, with create collections in one database automatically by application self.

2017-01-19T00:51:36.321+0800 I SHARDING [Balancer] distributed lock with ts: 587f9d18e078be11593c7930' unlocked.

2017-01-19T00:51:43.319+0800 I -        [conn23249] Invariant failure numDeleted == 1 src/mongo/s/query/cluster_cursor_manager.cpp 472

2017-01-19T00:51:43.319+0800 I -        [conn23249]

***aborting after invariant() failure

2017-01-19T00:51:43.323+0800 F -        [conn23249] Got signal: 6 (Aborted).

 0xc42e82 0xc41da9 0xc425b2 0x366160f710 0x3661232625 0x3661233e05 0xbd7b4b 0xb794f8 0xb79779 0xb79a11 0xb7bf42 0xb32506 0xb84cb4 0xb8597d 0xb9418d 0xb842f6 0x672ea5 0xbfd2cd 0x36616079d1 0x36612e89dd

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"400000","o":"842E82","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"841DA9"},{"b":"400000","o":"8425B2"},{"b":"3661600000","o":"F710"},{"b":"3661200000","o":"32625","s":"gsignal"},{"b":"3661200000","o":"33E05","s":"abort"},{"b":"400000","o":"7D7B4B","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"7794F8","s":"_ZN5mongo20ClusterCursorManager19detachCursor_inlockERKNS_15NamespaceStringEx"},{"b":"400000","o":"779779","s":"_ZN5mongo20ClusterCursorManager13checkInCursorESt10unique_ptrINS_19ClusterClientCursorESt14default_deleteIS2_EERKNS_15NamespaceStringExNS0_11CursorStateE"},{"b":"400000","o":"779A11","s":"_ZN5mongo20ClusterCursorManager12PinnedCursor12returnCursorENS0_11CursorStateE"},{"b":"400000","o":"77BF42","s":"_ZN5mongo11ClusterFind10runGetMoreEPNS_16OperationContextERKNS_14GetMoreRequestE"},{"b":"400000","o":"732506"},{"b":"400000","o":"784CB4","s":"_ZN5mongo7Command22execCommandClientBasicEPNS_16OperationContextEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderE"},{"b":"400000","o":"78597D","s":"_ZN5mongo7Command20runAgainstRegisteredEPNS_16OperationContextEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi"},{"b":"400000","o":"79418D","s":"_ZN5mongo8Strategy15clientCommandOpEPNS_16OperationContextERNS_7RequestE"},{"b":"400000","o":"7842F6","s":"_ZN5mongo7Request7processEPNS_16OperationContextEi"},{"b":"400000","o":"272EA5","s":"_ZN5mongo21ShardedMessageHan

[Kevin Adistambha]

Hi,

An invariant failure indicates an issue that originates in the server code. We’ve opened a ticket for this in JIRA so that our sharding engineers can have a look: https://jira.mongodb.org/browse/SERVER-27796

Could you post additional details in the ticket please:

• Your MongoDB version, and your driver version (if applicable)
• The complete stack trace
• Your deployment topology (how many mongos, how many shards, is each shard a replica set, etc.)
• The output of sh.status()
• What are the sequence of events that lead to this failure?

It would also be helpful if you can provide a description of your use case.

Best regards,
Kevin

​