shard version not ok in Client::Context: your version is too old

[Xavier Ho]

Hello,

Whenever I try to run a map-reduce query on my sharded databases (I have 3 of them), this error comes up every so often.

It seems to be consistently reproduceable whenever there is a heavy writing to the database, or more probably, when shards are moving across shards.

failed: mongod mr failed: { assertion: "[test.coll] shard version not ok in Client::Context: your version is too old ns: test.coll global: 703|0 client: 701|1", assertionCode: 13388, errmsg: "db assertion failure", ok: 0.0 }

I'm using MongoDB 2.0.0 with PyMongo 2.0.1.  This error happens with both map-reduce over the Python driver, and using the mongo interactive shell.

Has anyone experienced similar problems?

[Eliot Horowitz]

How long are the map/reduces taking?
Is there more info in the mongod log?

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/DiIThAbd5s8J.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>

[Xavier Ho]

On Wednesday, 26 October 2011 15:01:36 UTC+10, Eliot Horowitz wrote:

How long are the map/reduces taking?

Between 0.05 and 2 seconds. Median is 0.5 seconds.

I just reproduced the problem again on a collection with 2 million documents. 

 

Is there more info in the mongod log?

I see something like this:

Wed Oct 26 23:54:23 [conn5] insert test.coll 0ms

Wed Oct 26 23:54:23 [conn5] connection meta data too old - will retry ns:(test.coll) op:(insert) your version is too old  ns: test.coll global: 22|0 client: 21|5

Wed Oct 26 23:54:23 [conn5] got write with an old config - writing back ns: test.coll

Wed Oct 26 23:54:23 [conn5] op: insert len: 259 ns: test.coll{ <data omitted> }

Wed Oct 26 23:54:23 [conn5] insert test.coll 0ms

Wed Oct 26 23:54:23 [conn5] connection meta data too old - will retry ns:(test.coll) op:(insert) your version is too old  ns: test.coll global: 22|0 client: 21|5

 

Currently there are two processes writing to the database.

Also found a section that mentions map reduce:

Wed Oct 26 13:49:21 [Balancer] ChunkManager: time to load chunks for test.coll: 2ms sequenceNumbe: 548 version: 179|1

Wed Oct 26 13:49:21 [Balancer] *** end of balancing round

Wed Oct 26 13:49:21 [Balancer] distributed lock 'balancer/<ip>.lan:27017:1319588025:1804289383' unlocked. 

Wed Oct 26 13:49:21 [conn6] setShardVersion  shard0000 <ip>:27018  test.coll  { setShardVersion: "test.coll", configdb: "<ip>:27019", version: Timestamp 179000|1, serverID: <objectid>, shard: "shard0000", shardHost: "<ip>:27018" } 0x7f1fb0001920

Wed Oct 26 13:49:21 [conn6] setShardVersion success: { oldVersion: Timestamp 178000|5, ok: 1.0 }

Wed Oct 26 13:49:21 [conn6] retrying insert because of StaleConfigException: 9996 ns: test.coll for insert object: { <data omitted> }

Wed Oct 26 13:49:21 [conn6] sequence number - old: 548 new: 548

Wed Oct 26 13:49:21 [conn129] ERROR: sharded m/r failed on shard: <ip>:27018 error: { assertion: "[test.coll] shard version not ok in Client::Context: your version is too old  ns: cybersafehouse.logs global: 179|0 client: 178|3", assertionCode: 13388, errmsg: "db assertion failure", ok: 0.0 }

Wed Oct 26 13:49:26 [Balancer] about to acquire distributed lock 'balancer/<ip>:27017:1319588025:1804289383:

[Greg Studer]

Working on this issue - can follow SERVER-4158 - workaround is to
retry mapreduce after running flushRouterConfig

[Xavier Ho]

Cheers Greg.  Will do.

Xav