MongoDB concurrency issues - How to resolve

[hyndavi Marella]

Concurrency issues and how to resolve them.

Currently ,we are facing lot of issues in stage environment , where in because of both node asynchrosity and mongo concurrency ,

We are seeing the parallel calls being made , and because of which we could see our quantities being mismatched ,

we dont have a unique key to put forth , but even we tried putting unique index, even then , we are run a data load of 100k + ,for few of the records , we could see duplicates being inserted !!

Please let us know how to resolve them .

Should we implement any locking mechanism :

Locking mechanism in mongodb (exclusive Lock , Collection lock and document lock)

Please suggest some mechanisms to resolve

[Shivakumar T R]

Hi,

What are your use-cases? Without knowing this one cannot determine whether locking is required, and if required which mechanism is best suited.

It would help if you begin by describing what you are trying to achieve with Mongodb.

Thanks,
Shiv

[hyndavi Marella]

Hi

Our application is node js .So , since node js is asynchronous , when we are doing data loading as per the messages received from kafka cluster, since our app is node js , the calls being made concurrently and hence in the mongo db , because of concurrency , we are having duplicate records in mongodb .

For one of the objects , we tried with unique index.But , it dint help us and also , from the logs perpective ,I am unable to debug the logs as we ran for around 6lakh messages, where exactly the dupliciate issue was caused is difficult to find .
Please let us know the best practises if any.

[Shiva Ramagopal]

Hi,

>> For one of the objects , we tried with unique index

Which version of MongoDB are you using? If you have setup unique ids for the documents in the collection (assuming you mean documents when you said objects), MongoDB will report a write error in case of duplicates and your logs should indicate it. It shouldn't be difficult to scan through 600k log records to find if there are any write errors due to duplicates.

If you do end up finding duplicates, I would suspect its probably coming from Kafka. Check its delivery guarantee if it is set to At least Once in which case you can have duplicates delivered. Sometimes even an Exactly Once guarantee could cause duplicates. I'm not familiar with the details of how Kafka works.

Cheers,

Shiv

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/79c031df-e5c3-4b83-b375-23c7e8be1de4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[hyndavi Marella]

We are using mongoose 4.5.5 version

 

is there any alternative approach to unique indexes as it will have a performance impact !!

[Shiva Ramagopal]

If you can avoid duplicates in the first place the uniqueness constraint on that field is not required.

If you can't, then I would recommend that you first find out the percentage of duplicates in your data. If the percentage is small, say 10-20%, you could filter out duplicates before inserting into the collection by maintaining a dict or hashmap in your application. If the percentage is higher, you could move to a cache based on something like Redis.

Another approach you could try a solution that reduces lock contention by hashing the field and writing to different collections (bucketing). That can help since locking is at collection level. The tradeoff is of course the additional architectural complexity.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/184379c1-8054-4738-b3ea-e1a1ad7a22e0%40googlegroups.com.

[Kevin Adistambha]

Hi,

Just to add to the discussion, you may want to check out the db.collection.update() with upsert to see if it can be used to solve your problem. If the document doesn’t yet exist on the database, upsert will insert a single document instead. For example:

db.test.update({_id:123}, {$set:{field1:true}}, {upsert:true})

The update statement will attempt to update a document with _id: 123. If the document doesn’t exist, it will create one instead. Regardless of the path taken, the resulting collection will contain:

> db.test.find()
{
  "_id": 123,
  "field1": true
}

Also, there is an excellent blog post by A. Jesse Jiryu Davis that gives a detailed explanation into a similar issue: How to write resilient MongoDB applications

Best regards,
Kevin

​