Details on FourSquare bug?

[Colin Davis]

Reading through FourSquare's post-mortem (http://blog.foursquare.com/
2010/10/05/so-that-was-a-bummer/) it looks like they had a blowup when
adding a new shard to their existing systems; They had overloading
problems, leading to a pretty sizable downtime.

Are there any technical details about what went wrong, and more
specifically, how do I avoid the same fate?
If there is some particular "gotcha" I should avoid when adding new
shards, it would be good to know about it. ;)

Any details would be appreciated.
-Colin

[Dwight Merriman]

Basically, the issue is that if data migrates to a new shard, there is no re-compaction yet in the old shard of the old collection.  So there could be small empty spots throughout it which were migrated out, and if the objects are small, there is no effective improvement in RAM caching immediately after the migration.

A few more notes: 

- If your shard key has any correlation to insertion order, I think you are ok.

- If you add new shards very early s.t. the source shard doesn't have high load, i think you are ok.

- If your objects are fairly large (say 4KB+), i think you are ok.

- If the above don't hold, you will need the defrag enhancement which we will do asap.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Dwight Merriman]

i.e., if you remove 10% of the data on a shard, but that data was uniformly distributed throughout, and the objects are small, that won't give you a 10% decrease in "ram working set" as-is.

[Chase]

Will the defrag enhancement address the issue of reindexing that is
currently required in a heavy add / delete scenario?

> > mongodb-user...@googlegroups.com<mongodb-user%2Bunsubscribe@google groups.com>
> > .

[Dwight Merriman]

probably.  is there a JIRA for that?

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

[Chase]

I saw the following thread and it seems like it is a known issue with
index fragmentation and the need to reindex.

http://groups.google.com/group/mongodb-user/browse_thread/thread/6a5fc8c1d5c5c66e/d0b7e1aae3d69bd0?lnk=gst&q=reindex#d0b7e1aae3d69bd0

We ran into this in 1.4 and we are moving to 1.6 so it seems like it
maybe better already.
Thanks

[MattK]

Do we know MongoDB what version FourSquare was using?

[tmountain]

It appears that it is still an issue with 1.6. We're running in a
delete heavy environment, so I decided to run a test to see what
happens when repeatedly inserting and deleting from a collection.

simple PHP script:

<?php
$m = new Mongo();
$c = $m->test->foo;

for ($i = 0; $i < 6000000; $i++) {
$c->save(array("data" => "xxxxxxxxxxxxxxxxxxxxxxxx"));
}
?>

So, I run that to dump a bunch of junk data into a collection.
> db.stats();
{
"collections" : 3,
"objects" : 6000004,
"avgObjSize" : 59.99999000000667,
"dataSize" : 360000180,
"storageSize" : 476848640,
"numExtents" : 19,
"indexes" : 1,
"indexSize" : 296511376,
"fileSize" : 1006632960,
"ok" : 1
}

Index size is roughly 300M. Remove all the data:

> db.foo.remove();
> db.stats();
{
... snip ....
"indexSize" : 2318336,
}

2.3 megabytes still hanging around. Not a crazy amount of wasted
space, but if I run the process again:

$ php writer.php
> db.foo.remove();
>
> db.stats();
{
"indexSize" : 4653056,
}

and again...

$ php writer.php
> db.foo.remove();
> db.stats();
{
"indexSize" : 7028736,
}

I'm fairly certain this kind of reclamation issue is pretty common in
the DB world. I still have nightmares about running vacuum in the old
Postgres 7.x days.

On Oct 6, 12:06 am, Chase <chase.wolfin...@gmail.com> wrote:
> I saw the following thread and it seems like it is a known issue with
> index fragmentation and the need to reindex.
>

> http://groups.google.com/group/mongodb-user/browse_thread/thread/6a5f...

[Chase]

Is there a JIRA for the defrag or any other information on the target
release for defrag?

On Oct 5, 6:28 pm, Dwight Merriman <dwi...@10gen.com> wrote:

> > mongodb-user...@googlegroups.com<mongodb-user%2Bunsubscribe@google groups.com>
> > .