Wired tiger available write falls to zero

[Joe Waller]

Hi,

I'm testing an update heavy workload with WT 3.0.5 and am seeing some strange behaviour. The screenshot of my MMS console is attached. I noticed that when the workload starts the write tickets drops to 0 and doesn't recover for a few hours even though the number of updates is reduced. Page fault and queue activity appears minimal - so i am curious what is going on.

1. What controls the number of available write tickets? Does it always start at 100?

2. Why doesn't the number of write tickets return to normal once the updates reduce?

3. Cache usage drops to 0 when the update load starts - what is causing this?

4. Cache activity "writtento" doesnt appear to increase. Page fault and queue activity appear normal - what is the bottleneck here?

thanks,

Joe

[Joe Waller]

On further investigation I see that a lot of time is being spent in "database level locks" - how do we determine what is being locked at the db level?

2015-11-10T17:33:52.277+0000 I COMMAND  [conn188711] command xxx.$cmd command: update { update: "xxxx", updates: 2, writeConcern: { w: 1 }, ordered: true } keyUpdates:0 writeConf

licts:0 numYields:0 reslen:182 locks:{ Global: { acquireCount: { r: 5, w: 5 } }, Database: { acquireCount: { w: 5 }, acquireWaitCount: { w: 1 }, timeAcquiringMicros: { w: 1132890 } }, Collec

tion: { acquireCount: { w: 3 } }, oplog: { acquireCount: { w: 2 } } } 1084ms

[Joe Waller]

Hi folks,

Any thoughts on what is causing the high lock acquisition time at the database level?

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/WC8RFu57Vv0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/ddd12c67-f267-43df-9e5a-848aac2d1091%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Christian Bayer]

Hey Joe, did you ever find out what was causing this behavior? I'm running into a very similar situation:

- ~500 update ops/s

- ~70k document size

- WiredTiger Primary/secondary RS w/ Journal

- MongoDB 3.0.7

The server can sustain the load for about half an hour, then write tickets available drop to 0 and response times for update increase from 25ms to 1.2s. CPU now gets very busy, before it hovers at around 300% (3 cores fully utilized), now it jumps up to 1000% (10 CPUs). There is headroom in terms of CPU on the machine but it seems to cap there. Overall, the server is still responsive but slow.

My question: What is the reason for the write tickets being used up? Is that the level of concurrency? What does it depend on in terms of throughput? Can/Should I tune the number of write tickets for that instance? What type of resource will it cost? (I/O/CPU/Mem?) The disk doesn't seem to be the culprit, we utilize about 5% of what can be written to the disk in terms of bandwidth (~20MB/s)

Thanks,

Christian

[Ankur Raina]

Hi Christian,

What is the reason for the write tickets being used up? Is that the level of concurrency? What does it depend on in terms of throughput?

WiredTiger uses tickets to limit the number of simultaneous operations processed by the storage engine. The default value is 128. If the number of available write tickets drops to 0, new write requests would have to queue up waiting for a free ticket.

You can see available read/write tickets as below:

> db.serverStatus().wiredTiger.concurrentTransactions
{
    "write" : {
        "out" : 0,
        "available" : 128,
        "totalTickets" : 128
    },
    "read" : {
        "out" : 0,
        "available" : 128,
        "totalTickets" : 128
    }
}

Can/Should I tune the number of write tickets for that instance?

The default should generally not need to be adjusted, and is set to prevent the storage engine from being overwhelmed. Performance for WT may degrade when the number of concurrent workers greatly exceeds the number of cores. It may however be appropriate to tune the number of tickets down in some cases.

You can adjust the number of tickets by settting the wiredTigerConcurrentWriteTransactions and wiredTigerConcurrentReadTransactions parameters.

Before changing the defaults, I would consider testing an upgrade to the latest MongoDB version (currently v3.0.11) as there have been a number of bug fixes & improvements since 3.0.7.

I would also encourage may to test the latest 3.2.x production release in a development/staging environment as there have been significant improvements which are likely to improve your performance.

If you have any follow-up questions, please start a new discussion with the details relevant to your environment.

Regards,
Ankur

​