Unable to Query Sharded Collections after Primary Resync and Failover

[ky...@zenreach.com]

Yesterday, my team resynced a secondary and elected it as the new primary.

Both resyncing and the election went without a hitch. However, we are now unable to query our sharded collections. When we try to do so, we receive error messages similar to the following:

mongos> db.<collection>.count()

2015-04-09T10:42:17.733-0700 E QUERY    Error: count failed: {

	"code" : 10429,

	"ok" : 0,

	"errmsg" : "exception: setShardVersion failed host: <host>:27017 { oldVersion: Timestamp 0|0, oldVersionEpoch: ObjectId('000000000000000000000000'), ns: \"<collection>\", version: Timestamp 331000|1, versionEpoch: ObjectId('542d6fa157164e15b2c91c50'), globalVersion: Timestamp 0|0, globalVersionEpoch: ObjectId('000000000000000000000000'), reloadConfig: true, ok: 0.0, errmsg: \"could not refresh metadata for <collection> with requested shard version 331|1||542d6fa157164e15b2c91c50, stored shard version is 0|0...\", $gleStats: { lastOpTime: Timestamp 0|0, electionId: ObjectId('5525e871bb20e9bc47b8c66a') } }"

}

    at Error (<anonymous>)

    at DBQuery.count (src/mongo/shell/query.js:326:11)

    at DBCollection.count (src/mongo/shell/collection.js:1046:27)

    at (shell):1:21 at src/mongo/shell/query.js:326

mongos>

I've logged into the individual shards and queried the collection manually. Everything appears to be fine; there doesn't seem to be any underlying data inconsistency.

Has anyone seen this issue before?

[Edward Goslin]

Did you find a solution for this?

I'm getting, 

Error: error: {

        "$err" : "setShardVersion failed shard: shard0000:vm-808c-9118.nam.nsroot.net:32017 { oldVersion: Timestamp 0|0, oldVersionEpoch: ObjectId('000000000000000000000000'), ns: \"CEEMEA.aggregate_eod\", version: Timestamp 1000|0, versionEpoch: ObjectId('55e8574ebccbfbd113d64be7'), globalVersion: Timestamp 0|0, globalVersionEpoch: ObjectId('000000000000000000000000'), reloadConfig: true, ok: 0.0, errmsg: \"could not refresh metadata for CEEMEA.aggregate_eod with requested shard version 1|0||55e8574ebccbfbd113d64be7, stored shard version is 0|0||000000000...\" }",

        "code" : 10429,

        "shard" : "shard0000"

}

my status is

--- Sharding Status ---

  sharding version: {

        "_id" : 1,

        "minCompatibleVersion" : 5,

        "currentVersion" : 6,

        "clusterId" : ObjectId("55e7fec1bccbfbd113d63a7f")

}

  shards:

        {  "_id" : "shard0000",  "host" : "vm-808c-9118.nam.nsroot.net:32017" }

        {  "_id" : "shard0001",  "host" : "vm-f37b-4a11.nam.nsroot.net:32017" }

        {  "_id" : "shard0002",  "host" : "vm-082e-c369.nam.nsroot.net:32017" }

  balancer:

        Currently enabled:  yes

        Currently running:  no

        Failed balancer rounds in last 5 attempts:  0

        Migration Results for the last 24 hours:

                No recent migrations

  databases:

        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }

        {  "_id" : "CEEMEA",  "partitioned" : true,  "primary" : "shard0000" }

                CEEMEA.aggregate_eod

                        shard key: { "metaData.producedByTopology" : 1 }

                        chunks:

                                shard0000       1

                        { "metaData.producedByTopology" : { "$minKey" : 1 } } -->> { "metaData.producedByTopology" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0)

                CEEMEA.aggregate_live

                        shard key: { "metaData.producedByTopology" : 1 }

                        chunks:

                                shard0000       1

                        { "metaData.producedByTopology" : { "$minKey" : 1 } } -->> { "metaData.producedByTopology" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0)

                CEEMEA.aggregate_sod

                        shard key: { "metaData.producedByTopology" : 1 }

                        chunks:

                                shard0000       1

                        { "metaData.producedByTopology" : { "$minKey" : 1 } } -->> { "metaData.producedByTopology" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0)

                CEEMEA.result_eod

                        shard key: { "metaData.producedByTopology" : 1 }

                        chunks:

                                shard0000       1

                        { "metaData.producedByTopology" : { "$minKey" : 1 } } -->> { "metaData.producedByTopology" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0)

                CEEMEA.result_live

                        shard key: { "metaData.producedByTopology" : 1 }

                        chunks:

                                shard0000       1

                        { "metaData.producedByTopology" : { "$minKey" : 1 } } -->> { "metaData.producedByTopology" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0)

                CEEMEA.result_sod

                        shard key: { "metaData.producedByTopology" : 1 }

                        chunks:

                                shard0000       1

                        { "metaData.producedByTopology" : { "$minKey" : 1 } } -->> { "metaData.producedByTopology" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0)

        {  "_id" : "test",  "partitioned" : false,  "primary" : "shard0001" }

        {  "_id" : "CEEMA",  "partitioned" : false,  "primary" : "shard0001" }

        {  "_id" : "system",  "partitioned" : false,  "primary" : "shard0001" }

[ChCow]

Same issue but does't solved...

[Kevin Adistambha]

Hi ChCow

Same issue but does’t solved…

Please note that you’re replying to a thread that is more than 2 years old. Lots of things have changed since then, and what appear to be the same error could be caused by entirely different things.

In order to get more help, please create a new thread and post:

• Your MongoDB version and your OS version
• Details of your topology (how many shards, whether each shard is a replica set, whether any host runs more than one mongod process)
• Output of sh.status()
• What exactly is the error you see, and whether the error is consistent or not
• What are you trying to do that leads to this error
• Relevant snippets from the mongod logs
• Any repair attempts
• Whether your deployment follows the Production Notes closely, and any divergences to the recommended settings
• More information that will help

Best regards
Kevin

​

[ChCow]

mongodb:v3.4.9

error logs from mongodb c++ source code: https://github.com/mongodb/mongo/blob/367d31e1da549c460ae710a8cc280f4c235ab24f/src/mongo/s/client/shard_registry.cpp#L384

Mongos throw error when i add new node to a shard cluster and all the enableShardCollection can't query (ExceededTimeLimit).

Marking host config.app.com as failed :: caused by :: ExceededTimeLimit: Operation timed out, request was RemoteCommand 871 -- target:config.app.com db:config expDate:2017-10-21T13:16:38.250+0000 cmd:{ find: "shards", readConcern: { level: "majority", afterOpTime: { ts: Timestamp 1508586527000|1, t: 24 } }, maxTimeMS: 30000 }

2017-10-21T13:16:38.250+0000 I SHARDING [shard registry reload] Operation timed out :: caused by :: ExceededTimeLimit: Operation timed out, request was RemoteCommand 871 -- target:config.app.com db:config expDate:2017-10-21T13:16:38.250+0000 cmd:{ find: "shards", readConcern: { level: "majority", afterOpTime: { ts: Timestamp 1508586527000|1, t: 24 } }, maxTimeMS: 30000 }

2017-10-21T13:16:38.250+0000 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 50 could not get updated shard list from config server due to Operation timed out, request was RemoteCommand 871 -- target:config.app.com db:config expDate:2017-10-21T13:16:38.250+0000 cmd:{ find: "shards", readConcern: { level: "majority", afterOpTime: { ts: Timestamp 1508586527000|1, t: 24 } }, maxTimeMS: 30000 }; will retry after 30s

[Kevin Adistambha]

Hi ChCow

Thanks for the update.

Could you create a new thread with the answers to the rest of my questions?

Best regards
Kevin

​