Details on FourSquare bug? - mongodb-user

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion Details on FourSquare bug?
Received-SPF: neutral (google.com: 209.85.214.49 is neither permitted nor denied by best guess record for domain of dwi...@10gen.com) client-ip=209.85.214.49;
MIME-Version: 1.0
In-Reply-To: <e84b952d-3f48-453c-930e-edbcf973a644@x42g2000yqx.googlegroups.com>
References: <e84b952d-3f48-453c-930e-edbcf973a644@x42g2000yqx.googlegroups.com>
Date: Tue, 5 Oct 2010 20:28:57 -0400
Message-ID: <AANLkTinVvcO-9EELUt51QMaqZ3O0tc7rSC_G-z4T851Q@mail.gmail.com>
Subject: Re: [mongodb-user] Details on FourSquare bug?
From: Dwight Merriman <dwi...@10gen.com>
To: mongodb-user@googlegroups.com
Content-Type: multipart/alternative; boundary=0016e6d975baf23ff00491e7dcc4

--0016e6d975baf23ff00491e7dcc4
Content-Type: text/plain; charset=ISO-8859-1

Basically, the issue is that if data migrates to a new shard, there is no
re-compaction yet in the old shard of the old collection.  So there could be
small empty spots throughout it which were migrated out, and if the objects
are small, there is no effective improvement in RAM caching immediately
after the migration.

A few more notes:

- If your shard key has any correlation to insertion order, I think you are
ok.
- If you add new shards very early s.t. the source shard doesn't have high
load, i think you are ok.
- If your objects are fairly large (say 4KB+), i think you are ok.
- If the above don't hold, you will need the defrag enhancement which we
will do asap.

On Tue, Oct 5, 2010 at 8:15 PM, Colin Davis <cir...@gmail.com> wrote:

> Reading through FourSquare's post-mortem (http://blog.foursquare.com/
> 2010/10/05/so-that-was-a-bummer/) it looks like they had a blowup when
> adding a new shard to their existing systems; They had overloading
> problems, leading to a pretty sizable downtime.
>
> Are there any technical details about what went wrong, and more
> specifically, how do I avoid the same fate?
> If there is some particular "gotcha" I should avoid when adding new
> shards, it would be good to know about it. ;)
>
> Any details would be appreciated.
> -Colin
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com<mongodb-user%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--0016e6d975baf23ff00491e7dcc4
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div>Basically, the issue is that if data migrates to a new shard, there is=
 no re-compaction yet in the old shard of the old collection. =A0So there c=
ould be small empty spots throughout it which were migrated out, and if the=
 objects are small, there is no effective improvement in RAM caching immedi=
ately after the migration.</div>
<div><br></div><div>A few more notes:=A0</div><div><br></div><div>- If your=
 shard key has any correlation to insertion order, I think you are ok.</div=
><div>- If you add new shards very early s.t. the source shard doesn&#39;t =
have high load, i think you are ok.</div>
<div>- If your objects are fairly large (say 4KB+), i think you are ok.</di=
v><div>- If the above don&#39;t hold, you will need the defrag enhancement =
which we will do asap.</div><div><br></div><div class=3D"gmail_quote">On Tu=
e, Oct 5, 2010 at 8:15 PM, Colin Davis <span dir=3D"ltr">&lt;<a href=3D"mai=
lto:cir...@gmail.com">cir...@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex;">Reading through FourSquare&#39;s post-morte=
m (<a href=3D"http://blog.foursquare.com/
2010/10/05/so-that-was-a-bummer/" target=3D"_blank">http://blog.foursquare.=
com/<br>
2010/10/05/so-that-was-a-bummer/</a>) it looks like they had a blowup when<=
br>
adding a new shard to their existing systems; They had overloading<br>
problems, leading to a pretty sizable downtime.<br>
<br>
Are there any technical details about what went wrong, and more<br>
specifically, how do I avoid the same fate?<br>
If there is some particular &quot;gotcha&quot; I should avoid when adding n=
ew<br>
shards, it would be good to know about it. ;)<br>
<br>
Any details would be appreciated.<br>
-Colin<br>
<font color=3D"#888888"><br>
--<br>
You received this message because you are subscribed to the Google Groups &=
quot;mongodb-user&quot; group.<br>
To post to this group, send email to <a href=3D"mailto:mongodb-user@googleg=
roups.com">mongodb-user@googlegroups.com</a>.<br>
To unsubscribe from this group, send email to <a href=3D"mailto:mongodb-use=
r%2Bunsubscribe@googlegroups.com">mongodb-user+unsubscribe@googlegroups.com=
</a>.<br>
For more options, visit this group at <a href=3D"http://groups.google.com/g=
roup/mongodb-user?hl=3Den" target=3D"_blank">http://groups.google.com/group=
/mongodb-user?hl=3Den</a>.<br>
<br>
</font></blockquote></div><br>

--0016e6d975baf23ff00491e7dcc4--
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
