random write time spikes

[Colin Marc]

I'm MongoDB cluster - it has three shards with each shard being a
replica set with three mongods.

I'm currently in the middle of developing an application which sits on
a separate server and inserts 1-100k rows as a bulk insert at a time.
The collection is sharded, and in general it's pretty amazingly fast;
It usually takes about .01 seconds per 1000 rows in the bulk insert.
But then I randomly get spikes, and it'll take huge amounts of time to
insert similar amounts of data. Here's a (sort of cleaned up) snippet
of output from my benchmarking program:

the format is: seconds to do the insert | number of rows | seconds per
1000 rows

0.044 | 3880 | 0.011
0.001 | 45 | 0.022
0.001 | 109 | 0.009
0.427 | 28009 | 0.015
0.117 | 3300 | 0.035
0.061 | 5603 | 0.011
0.023 | 2181 | 0.011
0.005 | 420 | 0.012
0.111 | 4552 | 0.024
0.001 | 44 | 0.023
0.002 | 224 | 0.009
0.092 | 2927 | 0.031
0.002 | 136 | 0.015
0.003 | 256 | 0.012
13.56 | 4593 | 2.953 spike
3.42 | 19071 | 0.179
0.303 | 6767 | 0.045
0.152 | 8387 | 0.018
0.001 | 76 | 0.013
19.18 | 7525 | 2.549 spike
0.024 | 2200 | 0.011
0.001 | 111 | 0.009
0.004 | 343 | 0.012
22.52 | 30657 | 0.735 spike
0.015 | 1305 | 0.011
0.047 | 4344 | 0.011
0.0 | 32 | 0.0
0.027 | 2457 | 0.011
0.037 | 3272 | 0.011
0.035 | 2819 | 0.012
3.596 | 503 | 7.149 huge spike or tiny spike, depending on
how you look at it

These spikes tend to be about once a minute or two, maybe more
frequent (I cut out rows from the data I just showed). It seems like
the spikes might be some sort of lock or hang rather than a general
slowdown, and I've seen 10 seconds all they way up to a full minute.

What should I be looking at? What are some possible causes? Is this
normal operation, or solvable?

[Eliot Horowitz]

What version?
Have you looked at the mongod logs for those periods?
Do you have hardware monitoring?

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>

[Colin Marc]

I'm on 2.0.2. I noticed that it was the chunk migrations that seemed
to be hanging it up.

My shard key has two parts - a coarse, incremental value, and what is
effectively a random hash. So as part of the import process, I pre-
split on the coarse part, for each incremental value, before the
import process got to it. I also manually striped the new, empty
chunks across the shards. This seems to be working wonders.

If anyone would like to see the script I used for manually striping
chunks (written in python) shoot me an email!