Date: Sat, 13 Oct 2012 07:20:40 -0700 (PDT)
From: Dwight Merriman <dwi...@10gen.com>
To: mongodb-user@googlegroups.com
Message-Id: <e720d02b-3394-494f-8652-fac5ae633e5a@googlegroups.com>
In-Reply-To: <d35731c0-1ed4-4d78-be37-1acd7392ebdc@googlegroups.com>
References: <55b22bb4-64e6-41c7-b851-48f2495f2ae2@googlegroups.com>
 <2bf13525-5faf-4cab-bf13-5f79512e76eb@googlegroups.com>
 <d35731c0-1ed4-4d78-be37-1acd7392ebdc@googlegroups.com>
Subject: Re: OOM on secondary re-sync, bgsync buffer consumes all ram and
 swap
MIME-Version: 1.0
Content-Type: multipart/mixed; 
	boundary="----=_Part_172_11171558.1350138040343"

------=_Part_172_11171558.1350138040343
Content-Type: multipart/alternative; 
	boundary="----=_Part_173_23331767.1350138040343"

------=_Part_173_23331767.1350138040343
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

please run this in the shell and post the result?

db.printReplicationInfo()


On Friday, October 12, 2012 2:35:10 PM UTC-4, Brent Miller wrote:
>
> All machines are 64bit, and yeah, we've configured the oplog manually to 
> 32G (oplogSize = 32768) This was due to the fact that when we first started 
> testing, we only partitioned out 256G of disk for Mongo, and found that the 
> oplog ended up being too small to add new secondaries. (It might be worth 
> noting that we have a fairly write-heavy load and we're making a fair 
> number of updates to documents that are on the order of a megabyte.)
>
> In order to keep the replica set up, I've had to demote one of the 
> secondaries to be an arbiter and just disabled the other, so I don't know 
> if rs.status() would be any help at the moment. I can start a re-sync later 
> on today if it will be helpful. However, once we get the secondaries 
> through their inital sync, their oplog will trail the primary anywhere from 
> 1 - 60 seconds.
>
>
> Thanks,
> Brent
>
>
>
> On Thursday, October 11, 2012 7:50:09 PM UTC-7, Dwight Merriman wrote:
>>
>> are you running 64 bit or 32 bit?
>> did you configure --oplogsize on the command line (the default behaves 
>> well usually so the question is mainly if it is non-default)?
>> if you can post rs.status() that would be helpful.
>>
>>
>> On Thursday, October 11, 2012 4:10:52 PM UTC-4, Brent Miller wrote:
>>>
>>> We're running into this issue where we cannot re-sync our secondaries 
>>> after taking them off line for more than 10 minutes or so. When we take try 
>>> to bring a secondary back on-line it will consume all ram and swap and then 
>>> crash.
>>>
>>> When we start mongod with -vv, the log files on the crashed secondary is 
>>> filled with:
>>>
>>> Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171004922 bytes
>>> Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171152019 bytes
>>> Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171257666 bytes
>>> Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171402953 bytes
>>> Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171534548 bytes
>>> Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171644793 bytes
>>> Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171751196 bytes
>>> Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171896415 bytes
>>> Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5172027296 bytes
>>> Thu Oct 11 12:38:23 [rsBackgroundSync] bgsync buffer has 5172138255 bytes
>>>
>>> The primary is running 2.2.0, Ubuntu 12.04 with 48G ram. We have two 
>>> secondaries that are running 2.2.0, Ubuntu 12.04, but only have 4G of ram. 
>>> (We see the same behavior when running 2.2.1rc0 too.)
>>>
>>> Is there a reason why the bgsync buffer is growing so large? Is there a 
>>> way to minimize this?
>>>
>>> The rs.config() is as follows:
>>> {
>>> "_id" : "apm",
>>> "version" : 27,
>>> "members" : [
>>> {
>>> "_id" : 0,
>>> "host" : "apc:27017",
>>> "priority" : 2
>>> },
>>> {
>>> "_id" : 3,
>>> "host" : "apm-dev:27017"
>>> },
>>> {
>>> "_id" : 4,
>>> "host" : "apm:27017"
>>> }
>>> ]
>>> }
>>>
>>>
>>> Thanks,
>>> Brent
>>>
>>>
------=_Part_173_23331767.1350138040343
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

please run this in the shell and post the result?<div><br></div><div><div>d=
b.printReplicationInfo()</div><div><br></div><br>On Friday, October 12, 201=
2 2:35:10 PM UTC-4, Brent Miller wrote:<blockquote class=3D"gmail_quote" st=
yle=3D"margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-lef=
t: 1ex;">All machines are 64bit, and yeah, we've configured the oplog manua=
lly to 32G (oplogSize =3D 32768) This was due to the fact that when we firs=
t started testing, we only partitioned out 256G of disk for Mongo, and foun=
d that the oplog ended up being too small to add new secondaries.&nbsp;(It =
might be worth noting that we have a fairly write-heavy load and we're maki=
ng a fair number of updates to documents that are on the order of a megabyt=
e.)<div><br></div><div>In order to keep the replica set up, I've had to dem=
ote one of the secondaries to be an arbiter and just disabled the other, so=
 I don't know if rs.status() would be any help at the moment. I can start a=
 re-sync later on today if it will be helpful. However, once we get the sec=
ondaries through their inital sync, their oplog will trail the primary anyw=
here from 1 - 60 seconds.</div><div><br></div><div><br></div><div>Thanks,</=
div><div>Brent<br><div><br></div><div><br></div><div><br>On Thursday, Octob=
er 11, 2012 7:50:09 PM UTC-7, Dwight Merriman wrote:<blockquote class=3D"gm=
ail_quote" style=3D"margin:0;margin-left:0.8ex;border-left:1px #ccc solid;p=
adding-left:1ex">are you running 64 bit or 32 bit?<div>did you configure --=
oplogsize on the command line (the default behaves well usually so the ques=
tion is mainly if it is non-default)?</div><div>if you can post rs.status()=
 that would be helpful.</div><div><br><br>On Thursday, October 11, 2012 4:1=
0:52 PM UTC-4, Brent Miller wrote:<blockquote class=3D"gmail_quote" style=
=3D"margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex"=
>We're running into this issue where we cannot re-sync our secondaries afte=
r taking them off line for more than 10 minutes or so. When we take try to =
bring a secondary back on-line it will consume all ram and swap and then cr=
ash.<div><br></div><div>When we start mongod with -vv, the log files on the=
 crashed secondary is filled with:</div><div><br></div><div><div>Thu Oct 11=
 12:38:22 [rsBackgroundSync] bgsync buffer has 5171004922 bytes</div><div>T=
hu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171152019 bytes</d=
iv><div>Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171257666=
 bytes</div><div>Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5=
171402953 bytes</div><div>Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buf=
fer has 5171534548 bytes</div><div>Thu Oct 11 12:38:22 [rsBackgroundSync] b=
gsync buffer has 5171644793 bytes</div><div>Thu Oct 11 12:38:22 [rsBackgrou=
ndSync] bgsync buffer has 5171751196 bytes</div><div>Thu Oct 11 12:38:22 [r=
sBackgroundSync] bgsync buffer has 5171896415 bytes</div><div>Thu Oct 11 12=
:38:22 [rsBackgroundSync] bgsync buffer has 5172027296 bytes</div><div>Thu =
Oct 11 12:38:23 [rsBackgroundSync] bgsync buffer has 5172138255 bytes</div>=
</div><div><br></div><div>The primary is running 2.2.0, Ubuntu 12.04 with 4=
8G ram. We have two secondaries that are running 2.2.0, Ubuntu 12.04, but o=
nly have 4G of ram. (We see the same behavior when running 2.2.1rc0 too.)</=
div><div><br></div><div>Is there a reason why the bgsync buffer is growing =
so large? Is there a way to minimize this?</div><div><br></div><div>The rs.=
config() is as follows:</div><div><div>{</div><div><span style=3D"white-spa=
ce:pre">=09</span>"_id" : "apm",</div><div><span style=3D"white-space:pre">=
=09</span>"version" : 27,</div><div><span style=3D"white-space:pre">=09</sp=
an>"members" : [</div><div><span style=3D"white-space:pre">=09=09</span>{</=
div><div><span style=3D"white-space:pre">=09=09=09</span>"_id" : 0,</div><d=
iv><span style=3D"white-space:pre">=09=09=09</span>"host" : "apc:27017",</d=
iv><div><span style=3D"white-space:pre">=09=09=09</span>"priority" : 2</div=
><div><span style=3D"white-space:pre">=09=09</span>},</div><div><span style=
=3D"white-space:pre">=09=09</span>{</div><div><span style=3D"white-space:pr=
e">=09=09=09</span>"_id" : 3,</div><div><span style=3D"white-space:pre">=09=
=09=09</span>"host" : "apm-dev:27017"</div><div><span style=3D"white-space:=
pre">=09=09</span>},</div><div><span style=3D"white-space:pre">=09=09</span=
>{</div><div><span style=3D"white-space:pre">=09=09=09</span>"_id" : 4,</di=
v><div><span style=3D"white-space:pre">=09=09=09</span>"host" : "apm:27017"=
</div><div><span style=3D"white-space:pre">=09=09</span>}</div><div><span s=
tyle=3D"white-space:pre">=09</span>]</div><div>}</div></div><div><br></div>=
<div><br></div><div>Thanks,</div><div>Brent</div><div><br></div></blockquot=
e></div></blockquote></div></div></blockquote></div>
------=_Part_173_23331767.1350138040343--

------=_Part_172_11171558.1350138040343--