Balancer Not able to move chunks
90 views
Skip to first unread message
Ryan Fetz
Nov 5, 2012, 9:27:00 AM11/5/12
to mongod...@googlegroups.com
The balancer keeps erroring in the mongos log, stating it cannot move chunks.
We have 2 repl sets (3 nodes per set). Each set is a shard. The repl set names are rs0 ad rs1. We have 3 cfg servers, and 4 mongos'es.
Here is a snippet of the log:
Mon Nov 05 09:25:01 [Balancer] moving chunk ns: valkyrie.documents moving ( ns:v
alkyrie.documents at: rs0:rs0/DEMO-THEME01:27017,demo-db01:27017,demo-evr02:2701
7 lastmod: 1|1596||000000000000000000000000 min: { _id: MinKey } max: { _id: Obj
ectId('507dcc1c68e2eb468c2d2c8f') }) rs0:rs0/DEMO-THEME01:27017,demo-db01:27017,
demo-evr02:27017 -> rs1:rs1/demo-evr01:10004,demo-evr02:10004,demo-theme01:10004
Mon Nov 05 09:25:02 [Balancer] moveChunk result: { cause: { errmsg: "migrate alr
eady in progress", ok: 0.0 }, errmsg: "moveChunk failed to engage TO-shard in th
e data transfer: migrate already in progress", ok: 0.0 }
Mon Nov 05 09:25:02 [Balancer] balancer move failed: { cause: { errmsg: "migrate
already in progress", ok: 0.0 }, errmsg: "moveChunk failed to engage TO-shard i
n the data transfer: migrate already in progress", ok: 0.0 } from: rs0 to: rs1 c
hunk: min: { _id: MinKey } max: { _id: MinKey }
Mon Nov 05 09:25:02 [Balancer] moving chunk ns: valkyrie.serviceJobs moving ( ns
:valkyrie.serviceJobs at: rs0:rs0/DEMO-THEME01:27017,demo-db01:27017,demo-evr02:
27017 lastmod: 1|0||000000000000000000000000 min: { created: MinKey } max: { cre
ated: new Date(1351802286079) }) rs0:rs0/DEMO-THEME01:27017,demo-db01:27017,demo
-evr02:27017 -> rs1:rs1/demo-evr01:10004,demo-evr02:10004,demo-theme01:10004
Mon Nov 05 09:25:03 [Balancer] moveChunk result: { cause: { errmsg: "migrate alr
eady in progress", ok: 0.0 }, errmsg: "moveChunk failed to engage TO-shard in th
e data transfer: migrate already in progress", ok: 0.0 }
Mon Nov 05 09:25:03 [Balancer] balancer move failed: { cause: { errmsg: "migrate
already in progress", ok: 0.0 }, errmsg: "moveChunk failed to engage TO-shard i
n the data transfer: migrate already in progress", ok: 0.0 } from: rs0 to: rs1 c
hunk: min: { created: MinKey } max: { created: MinKey }
Mon Nov 05 09:25:03 [Balancer] moving chunk ns: valkyrie.documentTokens moving (
ns:valkyrie.documentTokens at: rs0:rs0/DEMO-THEME01:27017,demo-db01:27017,demo-
evr02:27017 lastmod: 1|16375||000000000000000000000000 min: { docId: MinKey } ma
x: { docId: ObjectId('507dcc1c68e2eb468c2d2c8f') }) rs0:rs0/DEMO-THEME01:27017,d
emo-db01:27017,demo-evr02:27017 -> rs1:rs1/demo-evr01:10004,demo-evr02:10004,dem
o-theme01:10004
Mon Nov 05 09:25:04 [Balancer] moveChunk result: { cause: { errmsg: "migrate alr
eady in progress", ok: 0.0 }, errmsg: "moveChunk failed to engage TO-shard in th
e data transfer: migrate already in progress", ok: 0.0 }
Mon Nov 05 09:25:04 [Balancer] balancer move failed: { cause: { errmsg: "migrate
already in progress", ok: 0.0 }, errmsg: "moveChunk failed to engage TO-shard i
n the data transfer: migrate already in progress", ok: 0.0 } from: rs0 to: rs1 c
hunk: min: { docId: MinKey } max: { docId: MinKey }
Mon Nov 05 09:25:05 [Balancer] distributed lock 'balancer/DEMO-EVR02:27018:13517
82403:41' unlocked.
The collections in the valkyrie database are not even close to balanced.
mongos> db.stats()
{
"raw" : {
"rs0/DEMO-THEME01:27017,demo-db01:27017,demo-evr02:27017" : {
"db" : "valkyrie",
"collections" : 28,
"objects" : 7740284,
"avgObjSize" : 2597.480543091184,
"dataSize" : 20105237088,
"storageSize" : 23921823680,
"numExtents" : 128,
"indexes" : 72,
"indexSize" : 1506117312,
"fileSize" : 85791342592,
"nsSizeMB" : 16,
"ok" : 1
},
"rs1/demo-evr01:10004,demo-evr02:10004,demo-theme01:10004" : {
"db" : "valkyrie",
"collections" : 80,
"objects" : 322,
"avgObjSize" : 83.98757763975155,
"dataSize" : 27044,
"storageSize" : 13991936,
"numExtents" : 90,
"indexes" : 81,
"indexSize" : 662256,
"fileSize" : 19251855360,
"nsSizeMB" : 16,
"ok" : 1
}
},
"objects" : 7740606,
"avgObjSize" : 2597.3759847743186,
"dataSize" : 20105264132,
"storageSize" : 23935815616,
"numExtents" : 218,
"indexes" : 153,
"indexSize" : 1506779568,
"fileSize" : 105043197952,
"ok" : 1
}
mongos>
gregor
Nov 5, 2012, 11:00:17 AM11/5/12
to mongod...@googlegroups.com
Can you do
use config
db.changelog.find({}).sort({time: -1})
and paste the output please?
Also run
sh.status()
gregor
Nov 8, 2012, 11:08:49 AM11/8/12
to mongod...@googlegroups.com
Are you mixing 2.0.x and 2.2?
Reply all
Reply to author
Forward
0 new messages