Application running very slow, high write lock %, high read queue and high netIn, netOut

[Astro B]

Hi,

We are running mongodb 2.4.9 with replica set having three members (PRIMARY, SECONDARY, SECONDARY). There are more than enough RAM and CPUs on all of these mongod instances.
However, we are facing serious performance degradation in mongodb with high write lock %, high read queue and high netIn, netOut. The application stopped responding. The heavy inserts are being done on mongo PRIMARY member where the performance degradation was found.

The mongostat output are attached below. How can I track the root cause of this issue?. The high read queue (qr) was observer on PRIMARY and not on SECONDARY members. The read preference is set to secondary.



Thanks,

Astro

[Rendy Bambang Junior]

Hi,

It might caused by high number of insert query. Have you checked mongotop and see how long those insert queries run? Also check disk I/O performance using mongoperf.

Thanks,

Rendy

[Astro]

Hi Rendy,

I have tested disk performance using mongoperf and it looks like there was no issue with disk as I tested SSD with different IOPS.
The mongotop outputs the write operations being performed on that specific collections iteratively for around 867 ms. The currentOp() shows the write operation is being performed on the collection and holding write lock queuing other operations to wait to acquire a lock.
 
I am concerened about high lock percentage and high read queue resulting in app running slow.  How can I tackle this situation to reduce the lock% and high read queue.

Thanks,

Astro

[Rendy Bambang Junior]

Hi Astro,

Your write query latency is quite high, mine is about <10ms. How about your query write concern? Is it fire and forget or wait until record wrote at replicas? Is it write or update operation with unindexed condition?

If you have a very huge number of write transaction, you might have to consider sharding.

However, before you come into scalability issue, you have to investigate why it takes so long to write. High write latency lead to high db lock %.

Rendy

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/AQwx81_Xph0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/12475bba-cc03-46d9-816a-da30901dddc1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Astro]

Rendy,

The write concern is set to 1. These are update operations having index being used. Does that need sharding though total data size < 10GB. The working set fits in  RAM. Also indexes fits in memory.

Atish

[Asya Kamsky]

You don't need sharding, you need to figure out why your writes are
taking such a long time.

Can you show some examples from the logs of slow writes? If none of
the writes are taking longer than 100ms you can lower the threshold
for logging slow operations by setting slowms to something lower, or
just raise logging for all ops via
db.adminCommand({setParameter:1,logLevel:1})

Asya
P.S. write concern is not relevant to how long the writes are taking
as the write lock is only held for the duration of the write, not
while the thread is waiting for acknowledgement.

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---

> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/mongodb-user/88126bf7-5565-4530-8a5e-6821ebda7c2a%40googlegroups.com.