Major Write Lock Issues without a lot of writes..

[Joe Esposito]

We've experienced a significant slowdown of our application over the past week.  I got involved with troubleshooting last evening, and am a bit confused.  While I think my problem may be caused by a missed file limit (i'm waiting for a window to restart mongo).  

The servers have 144GB of Memory, and 8x10k HW Raid10.  

The server architecture is a 3 node replica set with the application connecting via mongos (we're not sharded at this point).  All connections are happening over a private gigabit network.  

10gens MMS service shows our writelock% spike on January 29.  On this day, we were running around 90GB of indexes.  I'm not sure how to read the memory graph provided.. (10gen graphs: http://imgur.com/a/3AoMm)

So I got thinking that maybe we had too many indexes.  Knowing that this would destroy the read performance, we removed them and restarted mongo last night.  NOTHING was reading from the database, and we turned the write client back on.  The WriteLock shot right back up to 80-90% range.

Here's the data: http://pastie.org/3349778

the first Mongostat output shows Mongo with the indexes still on the database.  I also show some IO statistics.  The second mongostat output shows mongo after the clients were stopped, the indexes dropped, mongo restarted.  The writelock% goes from 0 to >85% instantly.

Anyone have any ideas on where to look? 

[Joe Esposito]

I forced the issue, and got a restart of mongo in today.  I added the line

ulimit -n 65535

into the script stanza of the mongo init (upstart) script.

I'm now seeing write locks of 40-50% which allows clients to read from the db at least.  Progress, but I'm still a bit confused.

[Scott Hernandez]

Please capture more data from mongostat and iostat -xm 2 (for 2-3
minutes). One sample of iostat is not helpful, unfortunately.

Also, what is the group name in MMS?

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

[Joe Esposito]

I truncated iostat due to pastie.org 64kbit limit, but I think this is a representative sample..

mongostat: http://pastie.org/3363694

iostat: http://pastie.org/3363691

Sure doesn't look disk related to me.  

[Scott Hernandez]

What commands are you running?

[Eliot Horowitz]

Regular server logs are also very helpful in these situations.
Are there any slow operations?
You might want to enable profiling for a bit as well.

[Joe Esposito]

I found a slow query that was doing a full table scan every time it was run.  I added an index to handle it, and my write locks fell to 0-10% depending on load.

Thanks for the advise.