MongoDB 1.2.2 on Freebsd 8 -- bad buzz

[Peter Kieltyka]

Hey guys,

I've been monitoring my applications performance on MongoDB 1.2.2 and
FreeBSD 8 and have experienced some significant disk i/o spiking every
time the the database fsync's.

I've been talking to Eliot in #mongodb on freenode under the alias
"elux"... so Eliot if you read this maybe it'll make more sense.

My system is a Intel Xeon-Lynnfield 3460 (Quad Core 2.8 GHz -- it's
got that new hyperthreading stuff so FBSD 8 treats my system like I
have 8 cores). Not that the CPU's matters since what I'm experiencing
is a disk i/o issue.

I have a daemon (10 worker processes) that insert about 300-800
documents / sec (pretty small docs). As I monitor my io stats, I can
see every 10 seconds, my hard drive spikes to 100% usage for about 4
seconds. My entire system practically locks up and looking at
mongod.log even simple queries asking for an indexed object takes 4
seconds .. ie:

Thu Feb 18 19:50:30 update mydb.posts query: { _id:
ObjId(4b7c0ec1eaf5e3e9980020d3) } 4286ms
... the slow down occurs during the time my disk's are being hammered
(fsync)

Some details on my HD from dmesg

ada0 at ahcich0 bus 0 target 0 lun 0
ada0: <WDC WD1500ADFD-00NLR5 21.07QR9> ATA/ATAPI-7 SATA 1.x device
ada0: 150.000MB/s transfers
ada0: 143089MB (293046768 512 byte sectors: 16H 63S/T 16383C)
ada0: Native Command Queueing enabled

Thats a WD1500AD Raptor 10,000 RPM 150GB drive.

My system is setup with a new AHCI driver with NCQ enabled. It's a
SATA150 drive, but the drive seems to be manufactured in 2005 so
performance isn't that great, or something else is screwed with it.
Running diskinfo -t /dev/ada0 gives me me:

/dev/ada0
512 # sectorsize
150039945216 # mediasize in bytes (140G)
293046768 # mediasize in sectors
290721 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
WD-WMAP42306276 # Disk ident.

Seek times:
Full stroke: 250 iter in 3.413137 sec = 13.653 msec
Half stroke: 250 iter in 2.412713 sec = 9.651 msec
Quarter stroke: 500 iter in 4.413826 sec = 8.828 msec
Short forward: 400 iter in 2.023488 sec = 5.059 msec
Short backward: 400 iter in 1.647349 sec = 4.118 msec
Seq outer: 2048 iter in 0.138163 sec = 0.067 msec
Seq inner: 2048 iter in 0.145327 sec = 0.071 msec
Transfer rates:
outside: 102400 kbytes in 1.268048 sec = 80754 kbytes/sec
middle: 102400 kbytes in 1.372853 sec = 74589 kbytes/sec
inside: 102400 kbytes in 2.005116 sec = 51069 kbytes/sec

So I'm getting 80MB/sec tops .. not great, should be enough for what
I'm doing which is about 1 MB/sec of data.

I did try changing the syncdelay paramater of mongod at
60,30,15,10,1 ... and it all performs about the same. I tried setting
the syncdelay to 240, and indeed my disk i/o was fine up until the 240
delay expired and my drives got hammered -- at least this confirms the
issue has to do with syncing the data via mmaped files.

At this point, I'm a little bit disappointed that my wicked server and
mongodb is giving me performance issues and I can expect that within
1-2 months the number of inserts per second will double. I have
thought about getting new drives, but I don't think thats the problem
here. I am going to try using 1.3.2 shortly, however I'm concerned
that the issue is how FreeBSD 8 handles mmap'ed files. Usually from my
experiences FBSD outperforms everything, but maybe this is the
exception. http://ivoras.sharanet.org/freebsd/freebsd8.html tells me
that 8.0 introduced superpages (enabled by default), which should give
mmap'ed files even more juice.

If you're curious about more stats about the system or db, just ask. I
will report back about how 1.3.2 performs.

Please let me know if my numbers are consistent with I should expect
from MongoDB 1.2.2, or if this throws up a big red flag and the
problem is either with how FreeBSD handles mmaped files or MongoDB
needs to provide better support for FBSD (or warn users it doesn't
perform well under heavy load).

Thanks guys.

Peter

[Peter Kieltyka]

Tried 1.3.2 -- my app works great, even feels faster, but when it
comes to the fsync, I still get frequent 100% spikes in my disk i/o
which blocks my system.

> exception.http://ivoras.sharanet.org/freebsd/freebsd8.htmltells me

[Eliot Horowitz]

Can you run for with syncdelay=1 for a while (20 minutes maybe) and
have iostat running the entire time.
That would be the most beneficial.

Theoretically, every 1 second all dirty pages should get written to
disk, and so there should never be a big spike.

If that doesn't fix it, then there are a few options
- freebsd doesn't track dirty at the page level or something like that
- you're actually writing more data than your disks can keep up

But doing that test with 1.3.2 will be helpful.

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[Mathias Stearn]

If you only need 150GB of capacity, you may want to consider investing
in a good SSD.

[Mathias Stearn]

Also, if you are using superpages then the minimum write size is
either 2 or 4MB rather than the normal 4kB. While superpages will
(probably) significantly improve performance if your dataset fits in
ram, it will also (probably) have a strong negative effect if it
doesn't.

[Peter Kieltyka]

Hey guys,

Thanks for the feedback -- you guys have been very supportive here and
on IRC.

I took sometime to prepare a proper followup. The granular details of
the FBSD virtual memory system is beyond me, and I couldn't find
enough documentation on how superpages works. So instead I benchmarked
io, vm, and mongod over a 10 minute interval in six scenarios.

Mathias: sysctl tells me that my hw.pagesize is 4096 regardless if
superpages is on or off. But that still doesn't convince me that
something whacky isn't going on with the VM system that is causing my
system to under perform. My system's VM settings (superpages on and
off): http://gist.github.com/308894.

I tried to give you guys as much detail as I could on how my system,
application and dataset performs in various scenarios. For each test I
sampled disk i/o, mongod.log and for the last three vmstat as well.
Hopefully this is helpful.

Tests:
======

Disks in a raid1 mirror:
------------------------
1.a Superpages on -- 2.5M doc data set -- syncdelay 60: 10 minute
sample
http://gist.github.com/72342d9120cba468923c

1.b Superpages on -- 2.5M doc data set -- syncdelay 5: 10 minute
sample
http://gist.github.com/2b0e958f57bb8693e344

1.c Superpages off -- 2.5M doc data set -- syncdelay 5: 10 minute
sample
http://gist.github.com/de444d27770f5dcabac9

Single disk (also samples vmstat):
----------------------------------
2.a Superpages on -- 2.5M doc data set -- syncdelay 60: 10 minute
sample
http://gist.github.com/5f5ff923b32e378de1b0

2.b Superpages on -- 2.5M doc data set -- syncdelay 5: 10 minute
sample
https://gist.github.com/5c7ff3a09357ddd9b349

2.c Superpages off -- 2.5M doc data set -- syncdelay 5: 10 minute
sample
http://gist.github.com/ca1ca4262bdb8b98c1e9

Please let me know if this is helpful and if you'd like any more data.
Next steps for me, I have an identical machine which I'm going to
install Archlinux on and try to do benchmark 2a to see how it
compares.

By the way, I found a paper on various VM's entitled: Comparison of
Memory Management Systems of BSD, Windows and Linux. It has some good
content in it, but dates back to 2002. It ends to paper saying that
the BSD VM has greatly improved since the paper was published, but
it's still based on the original code.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97.9553&rep=rep1&type=pdf

Cheers,

Peter

[Dwight Merriman]

thanks - give us a little time to look this over, eliot is out today.

btw if you could make a simplified version of your app that is a "benchmark" that doesn't use any proprietary data (maybe self populates the db) and exhibits the problem, and post, we could run that on some different OS's and see if we see different performance.  not sure if that is easy or not for you.

Peter

[Peter Kieltyka]

Hey Dwight,

Let me know what you think of the numbers. Mathias was trying to help
me through it yesterday and we came to the conclusion that turning off
syncdelay (0) and superpages produced the best results when the data
set > ram on my FBSD 8 system. Either way, the disk i/o was still
spiking and not really a proper solution.

I wanted to see how the same application, data set and hardware would
compare running on a Linux so I installed Fedora 11 on another
identical box along with the data set and app code. Turned out it
rarely spiked .. the only time was when the db first kicked in to read
in some of the working set. Here's the iostat from it: http://gist.github.com/311633

I have to get my product out in the next two weeks and time is running
out, so I've decided to switch all of my MongoDB machines to
Archlinux. I still have FBSD 8 machines though if you ever want me to
test this same application with a new developed MongoDB branch. The
data set I'm using is really quite standard, I just have three
collections: users, posts, and comments. The application code simply
aggregates and processes the data at ~800 ops/sec. Unfortunately it's
not trivial to write a simple version of what my app is doing, I'd be
writing a full benchmark suite to make it happen. Perhaps this is
something worth doing... I'm using the Mongo ruby driver and Mongoid
by the way.

Thanks guys. I'm happy to see the app running great on Linux even with
this crappy drives. I'm going to get some new ones to make sure I give
myself some room for growth.

Cheers.

On Feb 22, 4:14 pm, Dwight Merriman <dwi...@10gen.com> wrote:
> thanks - give us a little time to look this over, eliot is out today.
>
> btw if you could make a simplified version of your app that is a "benchmark"
> that doesn't use any proprietary data (maybe self populates the db) and
> exhibits the problem, and post, we could run that on some different OS's and
> see if we see different performance.  not sure if that is easy or not for
> you.
>
> On Mon, Feb 22, 2010 at 11:56 AM, Peter Kieltyka

> <peter.kielt...@gmail.com>wrote:

> >http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97.9553&rep=...

>
> > Cheers,
>
> > Peter
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongod...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > mongodb-user...@googlegroups.com<mongodb-user%2Bunsubscribe@google groups.com>

[Dwight Merriman]

i created a ticket for this; in the meantime using Linux sounds good.  the spikes you see on Linux are benign, that is likely when we preallocate datafiles.

http://jira.mongodb.org/browse/SERVER-663

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.