Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Message from discussion Balancing does not work

Received: by 10.58.254.39 with SMTP id af7mr1167239ved.41.1346326632171;
        Thu, 30 Aug 2012 04:37:12 -0700 (PDT)
X-BeenThere: mongodb-user@googlegroups.com
Received: by 10.52.35.202 with SMTP id k10ls2132371vdj.0.gmail; Thu, 30 Aug
 2012 04:37:03 -0700 (PDT)
Received: by 10.52.180.202 with SMTP id dq10mr711063vdc.17.1346326623066;
        Thu, 30 Aug 2012 04:37:03 -0700 (PDT)
Date: Thu, 30 Aug 2012 04:37:02 -0700 (PDT)
From: Daniel Schlegel <firmas...@gmail.com>
To: mongodb-user@googlegroups.com
Message-Id: <e2e68c38-90b1-4419-9a24-a3e558e6e191@googlegroups.com>
In-Reply-To: <55bcdddf-9c03-44c7-9b0c-25468f3523d4@googlegroups.com>
References: <f6418ef6-d1ea-401b-be60-e624e8d7ff2f@googlegroups.com>
 <55bcdddf-9c03-44c7-9b0c-25468f3523d4@googlegroups.com>
Subject: Re: Balancing does not work
MIME-Version: 1.0
Content-Type: multipart/mixed; 
	boundary="----=_Part_262_24152655.1346326622716"

------=_Part_262_24152655.1346326622716
Content-Type: multipart/alternative; 
	boundary="----=_Part_263_25723838.1346326622716"

------=_Part_263_25723838.1346326622716
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Hi Adam
Here's the output:

mongos> db.locks.find();

{ "_id" : "admin-movePrimary", "process" : 
"Web1:27069:1343571761:1804289383", "state" : 0, "ts" : 
ObjectId("501555d74439248d85dc8867"), "when" : 
ISODate("2012-07-29T15:25:11.099Z"), "who" : 
"Web1:27069:1343571761:1804289383:conn122:1714636915", "why" : "Moving 
primary shard of admin" }

{ "_id" : "example_production-movePrimary", "process" : 
"Web1:27069:1343571761:1804289383", "state" : 0, "ts" : 
ObjectId("501553614439248d85dc885a"), "when" : 
ISODate("2012-07-29T15:14:41.616Z"), "who" : 
"Web1:27069:1343571761:1804289383:conn1:1681692777", "why" : "Moving 
primary shard of example_production" }

{ "_id" : "example_production_vanity-movePrimary", "process" : 
"Web1:27069:1343571761:1804289383", "state" : 0, "ts" : 
ObjectId("501552fb4439248d85dc8855"), "when" : 
ISODate("2012-07-29T15:12:59.598Z"), "who" : 
"Web1:27069:1343571761:1804289383:conn1:1681692777", "why" : "Moving 
primary shard of example_production_vanity" }

{ "_id" : "balancer", "process" : "web1:27069:1346283357:314909341", 
"state" : 2, "ts" : ObjectId("503f4f2a3c113ffbd8e4a7e9"), "when" : 
ISODate("2012-08-30T11:31:54.320Z"), "who" : 
"web1:27069:1346283357:314909341:Balancer:1842493053", "why" : "doing 
balance round" }

{ "_id" : "example_production.people", "process" : 
"mongo11:20011:1346282264:758785138", "state" : 0, "ts" : 
ObjectId("503f4f2ce69a6c2009e22331"), "when" : 
ISODate("2012-08-30T11:31:56.182Z"), "who" : 
"mongo11:20011:1346282264:758785138:conn37208:1670912857", "why" : 
"migrate-{ _id: MinKey }" }

{ "_id" : "example_production.new_coll", "process" : 
"web1:27069:1346283357:314909341", "state" : 0, "ts" : 
ObjectId("503f36eb3c113ffbd8e4a6a1"), "when" : 
ISODate("2012-08-30T09:48:27.208Z"), "who" : 
"web1:27069:1346283357:314909341:conn37665:149759223", "why" : "drop" }

{ "_id" : "example_production_vanity.metrics", "process" : 
"web1:27069:1346283357:314909341", "state" : 0, "ts" : 
ObjectId("503f37833c113ffbd8e4a6aa"), "when" : 
ISODate("2012-08-30T09:50:59.474Z"), "who" : 
"web1:27069:1346283357:314909341:conn37665:149759223", "why" : "drop" }


Thanks!

Dani

Am Donnerstag, 30. August 2012 12:32:06 UTC+2 schrieb Adam C:
>
> Dani,
>
> You might have a stale lock lying around, or the mongos may have a stale 
> view of things - can you do a couple of things for me:
>
> 1. Bounce (restart) all of your mongos
> 2. Once the bounce is complete, log into the mongos and run:
>
> use config;
> db.locks.find();
>
> And post the results here.
>
> Thanks,
>
> Adam
>
> On Thursday, August 30, 2012 11:20:08 AM UTC+1, Daniel Schlegel wrote:
>>
>> Hello
>> i've upgraded to mongo 2.2.0 yesterday. all went like charme and i could 
>> shard my collection.
>> In the documentation i read:
>> Both splits and migrates are performed automatically.
>>
>> But unfortunately this doesn't work on our setup. 
>> I get error messages like these:
>>
>> Thu Aug 30 12:16:42 [Balancer]  ns: production.people going to move { 
>> _id: "production.people-_id_MinKey", lastmod: Timestamp 1000|0, 
>> lastmodEpoch: ObjectId('503e9d5ef940d75c2de07f8e'), ns: 
>> "production.people", min: { _id: MinKey }, max: { _id: 304836 }, shard: 
>> "s1" } from: s1 to: s2 tag []
>>
>> Thu Aug 30 12:16:42 [Balancer] moving chunk ns: production.people moving 
>> ( ns:production.people at: s1:s1/mongo11.example.com:20011,
>> mongo12.example.com:20012 lastmod: 1|0||000000000000000000000000 min: { 
>> _id: MinKey } max: { _id: 304836 }) s1:s1/mongo11.example.com:20011,
>> mongo12.example.com:20012 -> s2:s2/mongo21.example.com:20021,
>> mongo22.example.com:20022
>>
>> Thu Aug 30 12:16:43 [Balancer] moveChunk result: { cause: { errmsg: 
>> "migrate already in progress", ok: 0.0 }, errmsg: "moveChunk failed to 
>> engage TO-shard in the data transfer: migrate already in progress", ok: 0.0 
>> }
>>
>> Thu Aug 30 12:16:43 [Balancer] balancer move failed: { cause: { errmsg: 
>> "migrate already in progress", ok: 0.0 }, errmsg: "moveChunk failed to 
>> engage TO-shard in the data transfer: migrate already in progress", ok: 0.0 
>> } from: s1 to: s2 chunk:  min: { _id: MinKey } max: { _id: MinKey }
>>
>> After 10h of waiting there is still all data on the first shard and the 
>> second is empty.
>> I think about manually splitting but as it's not so easy to find the 
>> right splitpoint in our database i've not done this yet.
>>
>> Thanks for your Help
>> Dani
>>
>
------=_Part_263_25723838.1346326622716
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<div>Hi Adam</div><div>Here's the output:</div><div><br></div><div><p style=
=3D"margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: =
Helvetica; ">mongos&gt; db.locks.find();</p>
<p style=3D"margin-bottom: 0px; font-size: 12px; line-height: normal; font-=
family: Helvetica; ">{ "_id" : "admin-movePrimary", "process" : "Web1:27069=
:1343571761:1804289383", "state" : 0, "ts" : ObjectId("501555d74439248d85dc=
8867"), "when" : ISODate("2012-07-29T15:25:11.099Z"), "who" : "Web1:27069:1=
343571761:1804289383:conn122:1714636915", "why" : "Moving primary shard of =
admin" }</p>
<p style=3D"margin-bottom: 0px; font-size: 12px; line-height: normal; font-=
family: Helvetica; ">{ "_id" : "example_production-movePrimary", "process" =
: "Web1:27069:1343571761:1804289383", "state" : 0, "ts" : ObjectId("5015536=
14439248d85dc885a"), "when" : ISODate("2012-07-29T15:14:41.616Z"), "who" : =
"Web1:27069:1343571761:1804289383:conn1:1681692777", "why" : "Moving primar=
y shard of example_production" }</p>
<p style=3D"margin-bottom: 0px; font-size: 12px; line-height: normal; font-=
family: Helvetica; ">{ "_id" : "example_production_vanity-movePrimary", "pr=
ocess" : "Web1:27069:1343571761:1804289383", "state" : 0, "ts" : ObjectId("=
501552fb4439248d85dc8855"), "when" : ISODate("2012-07-29T15:12:59.598Z"), "=
who" : "Web1:27069:1343571761:1804289383:conn1:1681692777", "why" : "Moving=
 primary shard of example_production_vanity" }</p>
<p style=3D"margin-bottom: 0px; font-size: 12px; line-height: normal; font-=
family: Helvetica; ">{ "_id" : "balancer", "process" : "web1:27069:13462833=
57:314909341", "state" : 2, "ts" : ObjectId("503f4f2a3c113ffbd8e4a7e9"), "w=
hen" : ISODate("2012-08-30T11:31:54.320Z"), "who" : "web1:27069:1346283357:=
314909341:Balancer:1842493053", "why" : "doing balance round" }</p>
<p style=3D"margin-bottom: 0px; font-size: 12px; line-height: normal; font-=
family: Helvetica; ">{ "_id" : "example_production.people", "process" : "mo=
ngo11:20011:1346282264:758785138", "state" : 0, "ts" : ObjectId("503f4f2ce6=
9a6c2009e22331"), "when" : ISODate("2012-08-30T11:31:56.182Z"), "who" : "mo=
ngo11:20011:1346282264:758785138:conn37208:1670912857", "why" : "migrate-{ =
_id: MinKey }" }</p>
<p style=3D"margin-bottom: 0px; font-size: 12px; line-height: normal; font-=
family: Helvetica; ">{ "_id" : "example_production.new_coll", "process" : "=
web1:27069:1346283357:314909341", "state" : 0, "ts" : ObjectId("503f36eb3c1=
13ffbd8e4a6a1"), "when" : ISODate("2012-08-30T09:48:27.208Z"), "who" : "web=
1:27069:1346283357:314909341:conn37665:149759223", "why" : "drop" }</p>
<p style=3D"margin-bottom: 0px; font-size: 12px; line-height: normal; font-=
family: Helvetica; ">{ "_id" : "example_production_vanity.metrics", "proces=
s" : "web1:27069:1346283357:314909341", "state" : 0, "ts" : ObjectId("503f3=
7833c113ffbd8e4a6aa"), "when" : ISODate("2012-08-30T09:50:59.474Z"), "who" =
: "web1:27069:1346283357:314909341:conn37665:149759223", "why" : "drop" }</=
p><p style=3D"margin-bottom: 0px; font-size: 12px; line-height: normal; fon=
t-family: Helvetica; "><br></p><p style=3D"margin-bottom: 0px; font-size: 1=
2px; line-height: normal; font-family: Helvetica; ">Thanks!</p><p style=3D"=
margin-bottom: 0px; font-size: 12px; line-height: normal; font-family: Helv=
etica; ">Dani</p></div><br>Am Donnerstag, 30. August 2012 12:32:06 UTC+2 sc=
hrieb Adam C:<blockquote class=3D"gmail_quote" style=3D"margin: 0;margin-le=
ft: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;">Dani,<div><br></d=
iv><div>You might have a stale lock lying around, or the mongos may have a =
stale view of things - can you do a couple of things for me:</div><div><br>=
</div><div>1. Bounce (restart) all of your mongos<br>2. Once the bounce is =
complete, log into the mongos and run:</div><div><br></div><div>use config;=
</div><div>db.locks.find();</div><div><br></div><div>And post the results h=
ere.</div><div><br></div><div>Thanks,</div><div><br>Adam<br><br>On Thursday=
, August 30, 2012 11:20:08 AM UTC+1, Daniel Schlegel wrote:<blockquote clas=
s=3D"gmail_quote" style=3D"margin:0;margin-left:0.8ex;border-left:1px #ccc =
solid;padding-left:1ex"><span style=3D"color:rgb(76,58,44);font-family:helv=
etica,arial,sans-serif;line-height:21.316667556762695px;text-align:-webkit-=
left;background-color:rgb(243,244,235)">Hello</span><div><span style=3D"col=
or:rgb(76,58,44);font-family:helvetica,arial,sans-serif;line-height:21.3166=
67556762695px;text-align:-webkit-left;background-color:rgb(243,244,235)">i'=
ve upgraded to mongo 2.2.0 yesterday. all went like charme and i could shar=
d my collection.</span></div><div><span style=3D"color:rgb(76,58,44);font-f=
amily:helvetica,arial,sans-serif;line-height:21.316667556762695px;text-alig=
n:-webkit-left;background-color:rgb(243,244,235)">In the documentation i re=
ad:</span></div><div><span style=3D"color:rgb(76,58,44);font-family:helveti=
ca,arial,sans-serif;line-height:21.316667556762695px;text-align:-webkit-lef=
t;background-color:rgb(243,244,235)">Both splits and migrates are performed=
 automatically.</span><br></div><div><span style=3D"color:rgb(76,58,44);fon=
t-family:helvetica,arial,sans-serif;line-height:21.316667556762695px;text-a=
lign:-webkit-left;background-color:rgb(243,244,235)"><br></span></div><div>=
<span style=3D"color:rgb(76,58,44);font-family:helvetica,arial,sans-serif;l=
ine-height:21.316667556762695px;text-align:-webkit-left;background-color:rg=
b(243,244,235)">But unfortunately this doesn't work on our setup.&nbsp;</sp=
an></div><div><span style=3D"color:rgb(76,58,44);font-family:helvetica,aria=
l,sans-serif;line-height:21.316667556762695px;text-align:-webkit-left;backg=
round-color:rgb(243,244,235)">I get error messages like these:</span></div>=
<div><span style=3D"color:rgb(76,58,44);font-family:helvetica,arial,sans-se=
rif;line-height:21.316667556762695px;text-align:-webkit-left;background-col=
or:rgb(243,244,235)"><br></span></div><div><p style=3D"margin-bottom:0px;fo=
nt-size:12px;line-height:normal;font-family:Helvetica">Thu Aug 30 12:16:42 =
[Balancer]&nbsp; ns: production.people going to move { _id: "production.peo=
ple-_id_MinKey"<wbr>, lastmod: Timestamp 1000|0, lastmodEpoch: ObjectId('<w=
br>503e9d5ef940d75c2de07f8e'), ns: "production.people", min: { _id: MinKey =
}, max: { _id: 304836 }, shard: "s1" } from: s1 to: s2 tag []</p>
<p style=3D"margin-bottom:0px;font-size:12px;line-height:normal;font-family=
:Helvetica">Thu Aug 30 12:16:42 [Balancer] moving chunk ns: production.peop=
le moving ( ns:production.people at: s1:s1/<a href=3D"http://mongo11.exampl=
e.com:20011" target=3D"_blank">mongo11.example.com:<wbr>20011</a>,<a href=
=3D"http://mongo12.example.com:20012" target=3D"_blank">mongo12.example.com=
:<wbr>20012</a> lastmod: 1|0||000000000000000000000000 min: { _id: MinKey }=
 max: { _id: 304836 }) s1:s1/<a href=3D"http://mongo11.example.com:20011" t=
arget=3D"_blank">mongo11.example.com:<wbr>20011</a>,<a href=3D"http://mongo=
12.example.com:20012" target=3D"_blank">mongo12.example.com:<wbr>20012</a> =
-&gt; s2:s2/<a href=3D"http://mongo21.example.com:20021" target=3D"_blank">=
mongo21.example.com:<wbr>20021</a>,<a href=3D"http://mongo22.example.com:20=
022" target=3D"_blank">mongo22.example.com:<wbr>20022</a></p>
<p style=3D"margin-bottom:0px;font-size:12px;line-height:normal;font-family=
:Helvetica">Thu Aug 30 12:16:43 [Balancer] moveChunk result: { cause: { err=
msg: "migrate already in progress", ok: 0.0 }, errmsg: "moveChunk failed to=
 engage TO-shard in the data transfer: migrate already in progress", ok: 0.=
0 }</p>
<p style=3D"margin-bottom:0px;font-size:12px;line-height:normal;font-family=
:Helvetica">Thu Aug 30 12:16:43 [Balancer] balancer move failed: { cause: {=
 errmsg: "migrate already in progress", ok: 0.0 }, errmsg: "moveChunk faile=
d to engage TO-shard in the data transfer: migrate already in progress", ok=
: 0.0 } from: s1 to: s2 chunk:&nbsp; min: { _id: MinKey } max: { _id: MinKe=
y }</p></div><div><br></div><div><span style=3D"color:rgb(76,58,44);font-fa=
mily:helvetica,arial,sans-serif;line-height:21px;text-align:-webkit-left;ba=
ckground-color:rgb(243,244,235)">After 10h of waiting there is still all da=
ta on the first shard and the second is empty.</span><br></div><div style=
=3D"text-align:-webkit-left"><font color=3D"#4c3a2c" face=3D"helvetica, ari=
al, sans-serif"><span style=3D"line-height:21px">I think about manually spl=
itting but as it's not so easy to find the right splitpoint in our database=
 i've not done this yet.</span></font></div><div style=3D"text-align:-webki=
t-left"><font color=3D"#4c3a2c" face=3D"helvetica, arial, sans-serif"><span=
 style=3D"line-height:21px"><br></span></font></div><div style=3D"text-alig=
n:-webkit-left"><font color=3D"#4c3a2c" face=3D"helvetica, arial, sans-seri=
f"><span style=3D"line-height:21px">Thanks for your Help</span></font></div=
><div style=3D"text-align:-webkit-left"><font color=3D"#4c3a2c" face=3D"hel=
vetica, arial, sans-serif"><span style=3D"line-height:21px">Dani</span></fo=
nt></div></blockquote></div></blockquote>
------=_Part_263_25723838.1346326622716--

------=_Part_262_24152655.1346326622716--