Can't enable sharding - shard version not ok: version mismatch detected for db.products

[Alexander]

Hi.

We have 3.2.9v mongodb sharded cluster - 12 mongos, 3 configservers, 5 shards. All shards are 3 node replicasets.
We have php application wich used php-pecl-mongodb-1.2.2-1.el6.remi.7.0.x86_64.

When balancer stopped application works fine. When balancer running we have error:

youla-front13 [2017-05-17 14:52:48] app.591c39907fb181.77657805.CRITICAL:
MongoException: [youla.products] shard version not ok: version mismatch detected for youla.products, stored major version 833 does not match received 829 ( ns : youla.products, received : 829|1||58488e8f21348926412f73b3, wanted : 833|1||58488e8f21348926412f73b3, send )
(uncaught exception) at /home/sites/youla.io/.www/vendor/alcaeus/mongo-php-adapter/lib/Alcaeus/MongoDbAdapter/ExceptionConverter.php line 83 {"exception":"[object]
(MongoException(code: 13388): [youla.products] shard version not ok: version mismatch detected for youla.products, stored major version 833 does not match received 829
( ns : youla.products, received : 829|1||58488e8f21348926412f73b3, wanted : 833|1||58488e8f21348926412f73b3, send )
at /home/sites/youla.io/.www/vendor/alcaeus/mongo-php-adapter/lib/Alcaeus/MongoDbAdapter/ExceptionConverter.php:83,
MongoDB\\Driver\\Exception\\RuntimeException(code: 13388):
[youla.products] shard version not ok: version mismatch detected for youla.products, stored major version 833 does not match received 829 ( ns : youla.products, received : 829|1||58488e8f21348926412f73b3, wanted : 833|1||58488e8f21348926412f73b3, send )
at /home/sites/youla.io/.www/vendor/mongodb/mongodb/src/Operation/Find.php:180)"} []

And i can see error on shard primary node with db.setLogLevel(1, "command" ) option:

2017-05-17T14:52:21.841+0300 I COMMAND  [conn1260458] command youla.products command: find { find: "products", filter: { _id: ObjectId('5916f3e662e1c63a2322a105') }, limit: 1, batchSize: 0, shardVersion: [ Timestamp 829000|1, ObjectId('58488e8f21348926412f73b3') ] } keyUpdates:0 writeConflicts:0 exception: [youla.products] shard version not ok: version mismatch detected for youla.products, stored major version 833 does not match received 829 ( ns : youla.products, received : 829|1||58488e8f21348926412f73b3, wanted : 833|1||58488e8f21348926412f73b3, send ) code:13388 numYields:0 reslen:455 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_command 0ms
2017-05-17T14:52:21.857+0300 D COMMAND  [conn1260513] assertion while executing command 'find' on database 'youla' with arguments '{ find: "products", filter: { is_deleted: false, owner_id: ObjectId('577397afd53f3da598553ccb'), is_blocked: false, is_sold: false, is_archived: false }, projection: { $sortKey: { $meta: "sortKey" } }, sort: { _id: -1 }, limit: 40, batchSize: 0, shardVersion: [ Timestamp 829000|1, ObjectId('58488e8f21348926412f73b3') ] }' and metadata '{ $ssm: { $secondaryOk: 1 }, configsvr: { opTime: { ts: Timestamp 1495021940000|4, t: 11 } } }': 13388 [youla.products] shard version not ok: version mismatch detected for youla.products, stored major version 833 does not match received 829 ( ns : youla.products, received : 829|1||58488e8f21348926412f73b3, wanted : 833|1||58488e8f21348926412f73b3, send )

At the same time i can see querries on secondary node with read preference secondary and thay executed without error.
Please give us some advice why this is happen and how to overcome that?

mongos> sh.status()
--- Sharding Status ---
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5847d90ad088c314272d627e")
}
  shards:
    {  "_id" : "classified",  "host" : "classified/youla-db1.p:27017,youla-db2.p:27017,youla-db3.p:27017" }
    {  "_id" : "shard1",  "host" : "shard1/youla-db4.p:27021,youla-db5.p:27021,youla-db6.p:27021" }
    {  "_id" : "shard2",  "host" : "shard2/youla-db7.p:27022,youla-db8.p:27022,youla-db9.p:27022" }
    {  "_id" : "shard3",  "host" : "shard3/youla-db10.p:27023,youla-db11.p:27023,youla-db12.p:27023" }
    {  "_id" : "shard4",  "host" : "shard4/youla-db13.p:27024,youla-db14.p:27024,youla-db15.p:27024" }
  active mongoses:
    "3.2.9" : 12
  balancer:
    Currently enabled:  no
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours:
        2564 : Success
        9121 : Failed with error 'aborted', from classified to shard2
  databases:
    {  "_id" : "nagios",  "primary" : "classified",  "partitioned" : false }
    {  "_id" : "test",  "primary" : "classified",  "partitioned" : false }
    {  "_id" : "youla",  "primary" : "classified",  "partitioned" : true }

     youla.products
            shard key: { "_id" : "hashed" }
            unique: false
            balancing: true
            chunks:
                classified    430
                shard1    430
                shard2    430
                shard3    431
                shard4    10
            too many chunks to print, use verbose if you want to force print

[Alexander]

Add some info:

When i do sh.stopBalancer() or sh.disableBalancer( youla.products ) i can see messages:

2017-05-17T16:24:18.763+0300 I SHARDING [RangeDeleter] waiting for open cursors before removing range [{ _id: -5915624687736732463 }, { _id: -5899691929036541355 }) in youla.products, elapsed secs: 5523, cursor ids: [103250336044, 103383258360, 103686320180, 103953388284, 104015056969, 104018951996, 104105725613, 104387400467, 104508893536, 105735771469, 105923411925, 106132348939, 106187726785, 106293444945, 106373209653, 106747676914, 106751432597, 106865765892, 106888475680, 106901487800, 107347117987]

typical load:

mongostat --port 27022
insert query update delete getmore command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn    set repl                      time
    18  2953    312     *0    2782  1233|0     0.7   80.6       0 22.7G 15.3G   0|0   0|0 2.10m  5.26m  857 shard2  PRI 2017-05-17T16:26:50+03:00
    20  2596    251     *0    2383  1047|0     0.5   80.1       0 22.7G 15.3G   1|0   0|0 1.82m  4.65m  857 shard2  PRI 2017-05-17T16:26:51+03:00
    25  3041    321      1    2966  1162|0     0.6   80.8       0 22.7G 15.3G   0|0   0|0 2.12m  5.13m  857 shard2  PRI 2017-05-17T16:26:52+03:00

среда, 17 мая 2017 г., 16:09:20 UTC+3 пользователь Alexander написал:

[Alexander]

Fixed.
https://jira.mongodb.org/browse/SERVER-28019
Update linked library.


среда, 17 мая 2017 г., 16:27:16 UTC+3 пользователь Alexander написал: