scalability and acceptable write lock percentages

[Jess]

All--

I'm currently running load tests against a production grade db stack
in EC2. I have 3 micro config servers, properly configured against
two replica sets comprised of large EC2 instances.

Surprisingly when I simulate just 20K user sessions, I see pretty high
global write lock percentages in mongostat. I've included sample
output from mongostat against one of the nodes in the image below.

http://i.imgur.com/lXnnx.png

It appears as if there aren't too many faults happening, and that the
r/w queues are reasonable. So, is this level of write lock percentage
necessarily problematic or not, given that requests aren't queuing
up?

When I run larger load tests, on the order of 35K user sessions, I see
the write lock percentage spike as high as 85% (and NOT while a flush
is happening, btw), and some slow queries begin to get logged.

Does anyone have an explanation, or even just some hints about what I
might want to look into?

Please note that the nature of our data is *mostly* contained in a
single collection. We aggregate atomic updates against these objects
and flush them at the end of each request. Therefore we are often
calling updates against single records, comprised of many $set, $inc
operations on embedded fields within these records. We also have a
smaller collection against which we make some inserts.

We are running mongo 1.8.1.

Thanks everyone!

[Jess]

I am also seeing io write spikes, not corresponding to flushes.... I
don't have a good explanation for why these might be happening.

Here is an iostat -x 2 snippet:

avg-cpu: %user %nice %system %iowait %steal %idle
6.37 0.00 9.48 25.48 1.19 57.48

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
xvdf 0.00 2689.50 0.00 2258.50 0.00 38972.00
17.26 112.53 48.51 0.35 79.50

avg-cpu: %user %nice %system %iowait %steal %idle
5.96 0.00 7.56 10.03 0.73 75.73

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
xvdf 0.00 919.00 0.00 1162.00 0.00 17268.00
14.86 48.29 44.11 0.31 36.50

[Richard Kreuter]

The lock percentages you're showing aren't terribly high. So long as
your total operation counters (insert/query/update/delete) are
tolerable, the lock% shouldn't be particularly worrisome.

That said, we've had numerous encounters with EBS volumes becoming
impractically slow, with no apparent proximate cause, for sustained
durations. Generally speaking, we recommend striping (or striping and
mirroring, i.e., RAID10) across EBS volumes.

Regards,
Richard

> > in EC2. =A0I have 3 micro config servers, properly configured against

> > two replica sets comprised of large EC2 instances.
> >
> > Surprisingly when I simulate just 20K user sessions, I see pretty high

> > global write lock percentages in mongostat. =A0I've included sample

> > output from mongostat against one of the nodes in the image below.
> >
> > http://i.imgur.com/lXnnx.png
> >
> > It appears as if there aren't too many faults happening, and that the

> > r/w queues are reasonable. =A0So, is this level of write lock percentage

> > necessarily problematic or not, given that requests aren't queuing
> > up?
> >
> > When I run larger load tests, on the order of 35K user sessions, I see
> > the write lock percentage spike as high as 85% (and NOT while a flush
> > is happening, btw), and some slow queries begin to get logged.
> >
> > Does anyone have an explanation, or even just some hints about what I
> > might want to look into?
> >
> > Please note that the nature of our data is *mostly* contained in a

> > single collection. =A0We aggregate atomic updates against these objects
> > and flush them at the end of each request. =A0Therefore we are often

> > calling updates against single records, comprised of many $set, $inc

> > operations on embedded fields within these records. =A0We also have a

> > smaller collection against which we make some inserts.
> >
> > We are running mongo 1.8.1.
> >
> > Thanks everyone!
>

> --=20
> You received this message because you are subscribed to the Google Groups "=
> mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user+unsubscribe@goog=
> legroups.com.
> For more options, visit this group at http://groups.google.com/group/mongod=
> b-user?hl=3Den.
>