MongoDB + WiredTiger: lots of WriteConflict

[Alex Sobrino Beltrán]

Hi,

First of all, I want to apologize if this isn't the right place to write this issue to. But since I'm lost, I hope someone can point me the right direction :)

We have a pretty complex python app running over Apache Storm 0.9.2, MongoEngine 0.8.7, PyMongo 2.8, and MongoDB 3.0.2. Our Mongo setup is a sharding cluster (3 mongos, 3 mongo-config and 5 ReplicaSets) and all ReplicaSets are running with WiredTiger.

With a very high load of concurrent updates, I get a lot of exceptions:

OperationError: Update failed (exception: WriteConflict)

And looking into the mongod logs:

2015-04-22T16:24:02.100+0200 I WRITE    [conn81979] Had WriteConflict during multi update, aborting
2015-04-22T16:24:04.907+0200 I WRITE    [conn75303] Had WriteConflict during multi update, aborting
2015-04-22T16:24:18.538+0200 I WRITE    [conn81979] Had WriteConflict during multi update, aborting
2015-04-22T16:24:27.539+0200 I WRITE    [conn75526] Had WriteConflict during multi update, aborting
2015-04-22T16:24:34.413+0200 I WRITE    [conn81979] Had WriteConflict during multi update, aborting
2015-04-22T16:24:34.455+0200 I WRITE    [conn77036] Had WriteConflict during multi update, aborting
2015-04-22T16:24:46.693+0200 I WRITE    [conn81979] Had WriteConflict during multi update, aborting

What I've done is to try/except the update operation+exception and re-do it in the except. Seems to work fine almost all the time, but sometimes it fails.

I've tried to find some information about what the WriteConflict is, why it isn't working and how to solve it... But I haven't been able to U_U Any clue?

I also posted this in StackOverflow but didn't get any answer (http://stackoverflow.com/questions/29800823/mongodb-with-wiredtiger-lots-of-writeconflict)

Thanks!

[Asya Kamsky]

What is the exact version you are using - can you post the beginning of you log file from mongod - that should have all the details.   You should never be seeing writeConflicts at the application level, they are handled within mongod for you.

Asya

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/dbcbf974-1342-477b-aeba-9f64a1a7554b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Sobrino Beltrán]

Thanks for your reply Asya. Here's the info you requested:

2015-04-10T10:34:06.213+0200 I CONTROL  [initandlisten] MongoDB starting : pid=21243 port=27017 dbpath=/opt/mongodb 64-bit host=bb07
2015-04-10T10:34:06.213+0200 I CONTROL  [initandlisten] db version v3.0.2
2015-04-10T10:34:06.213+0200 I CONTROL  [initandlisten] git version: 6201872043ecbbc0a4cc169b5482dcf385fc464f
2015-04-10T10:34:06.213+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2015-04-10T10:34:06.213+0200 I CONTROL  [initandlisten] build info: Linux ip-10-229-1-2 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-04-10T10:34:06.213+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2015-04-10T10:34:06.213+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { http: { enabled: false } }, replication: { replSet: "bbrs01" }, storage: { dbPath: "/opt/mongodb", engine: "wiredTiger" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }

It's from one of the mongod processes of the sharding cluster. I also checked the mongos and mongo-config and they're running the same version:

> mongos --version
MongoS version 3.0.2 starting: pid=19084 port=27017 64-bit host=bb01 (--help for usage)
git version: 6201872043ecbbc0a4cc169b5482dcf385fc464f
OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
build sys info: Linux ip-10-229-1-2 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49

Hope it helps.

[Asya Kamsky]

Alex:

Would you please open a Jira ticket in SERVER project?   This seems like it should not happen and I couldn't find an issue that was similar to this one that was not already fixed for 3.0.   It would be helpful if you include the logs, as well as description of exactly what your application is doing when this happens.

If the logs have sensitive information we can provide a secure location to scp them to rather than attaching them to the ticket.

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/2e327d07-e2bd-4afc-849b-34fd218c940f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!

[Alex Sobrino Beltrán]

Hi Asya, thanks for your reply. I'll open a Jira ticket on monday. I have a few questions:

- Should I attach the whole logs for all the mongos/mongo-config and mongod processes?

- It would be better to upload the logs via SCP/SFTP, so should I ask for the secure location after I open the ticket?

As for the description, no problem. I will provide all the information possible.

Thanks again! :)

[Asya Kamsky]

Best would be to include a small snippet of the log showing the write conflict message and then explaining that you can upload full logs for diagnosis.

Btw, if you can reproduce this it would be great to see mongod logs at higher level of verbosity (you can run db.adminCommand({setParameter:1, verbose:1}) then reproduce the error then rerunning it with verbose:0)

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/89d11ec1-9a7a-476d-9c71-9fab91bac52b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alex Sobrino Beltrán]

Jira ticket opened: https://jira.mongodb.org/browse/SERVER-18213

Thanks for your help Asya :)