Mongos are failing to Calculate Config Difference

[Mohamed Abada]

Greetings, 

I hope you're doing well.

I am currently facing the below error and I will appreciate it if someone can help or advise how can I get it fixed :

mongos> db.stats()

2017-07-24T09:12:10.064+0000 E QUERY    Error: error: {

"$err" : "error loading initial database config information :: caused by :: could not calculate config difference for ns test.new on IP:27019,IP:27019,IP:27019 :: caused by :: can't find shard for: shard2",

"code" : 13129

}

    at Error (<anonymous>)

    at DBQuery.next (src/mongo/shell/query.js:259:15)

    at DBCollection.findOne (src/mongo/shell/collection.js:189:22)

    at DB.runCommand (src/mongo/shell/db.js:58:41)

    at DB.stats (src/mongo/shell/db.js:30:17)

    at (shell):1:4 at src/mongo/shell/query.js:259

What I am was trying to do is to downscale the cluster and remove one shard, draining completed successfully and all chunks got moved to the primary shard, however after I issued the second shard remove command db.runCommand( { removeShard: "shard2" } ) I noticed that the shard got removed however one chunk (shard key) was moved back to shard2 befor it got removed ( I have no idea how or why).

mongos> db.chunks.find({shard:"shard2"})

{ "_id" : "test.new-_id_MinKey", "lastmod" : Timestamp(4, 0), "lastmodEpoch" : ObjectId("5641e15eab20f8848d81a6e2"), "ns" : "qs_place.places", "min" : { "_id" : { "$minKey" : 1 } }, "max" : { "_id" : "1446560142605673436" }, "shard" : "shard2" }

mongos> 

sh.status Output :

{  "_id" : "test",  "partitioned" : true,  "primary" : "shard1" }

test.new

shard key: { "_id" : "hashed" }

chunks:

shard1 1

shard2 1

{ "_id" : { "$minKey" : 1 } } -->> { "_id" : "1446560142605673436" } on : shard2 Timestamp(4, 0) 

{ "_id" : "1446560142605673436" } -->> { "_id" : { "$maxKey" : 1 } } on : shard1 Timestamp(4, 1) 

I tried db.adminCommand("flushRouterConfig") but it didn't help.

Any idea what caused this issue and how can I get it resolved. ?

Best Regards,

Mohamed Abada

[Mohamed Abada]

I will appreciate it if someone can help with this issue.

Best Regards,

Mohamed Abada 

[Mohamed Abada]

Any help here ?

[Kevin Adistambha]

Hi,

I noticed that you have filed a SERVER ticket for this particular issue: https://jira.mongodb.org/browse/SERVER-30517, and work is ongoing in that ticket to help you resolve this situation. So far it was determined that it’s likely that you ran into SERVER-21896, which was fixed in MongoDB 3.2.3.

For completeness, you mentioned that the MongoDB version you are running is 3.0.7, which was released two years ago. Once this issue is resolved, I would recommend that you upgrade to at least the newest version in the 3.2 series, which is currently 3.2.16, since there have been many improvements that were made in many areas in the 3.2 series over the 3.0 series. It is of course much better if you can upgrade to the latest MongoDB, which is currently 3.4.6.

Having said that, please note that there is no supported upgrade path from 3.0.7 directly to 3.4.6. Rather, the supported upgrade path is to incrementally upgrade from 3.0 → 3.2 → 3.4.

• Upgrade MongoDB to 3.2
• Upgrade a Sharded Cluster to 3.4

Best regards,
Kevin

​