Find returning duplicate documents on sorted query.

[Leif Mortenson]

Hi,

We have a very simple query that we use to get a list of documents sorted by their name.  It returns 49 records when working correctly.

db.docs.find({}, {"_id":true, "name":true}).sort({"name":1});

Normally this works fine, but every once in a while at the end of the returned list, one or more documents will be seen a second time.  So we get duplicates.  When this happens they are always at the end and appear to be sorted amongst themselves, but not as a whole.

So the "sorted list" with extra documents would be: A, B, C, D, E, F, G, H, I, C, E

I searched around and it looks like this might be being caused because we have other threads that are updating the records in the docs collection at the same time.

https://www.quora.com/How-did-mongodb-return-duplicated-but-different-documents

The suggestion is to use cursor.snapshot().

The problem is that snapshot can not be used for a sorted collection.

Our solution is to scan through the resulting query, AFTER getting the results, and strip off the duplicates at the end.  But is this the best solution?

We are currently using Mongo 2.6.10.  If you think this might have been resolved in a newer version we will look into the upgrade.  I didn't see anything in the release notes.

Even getting to this level of understanding of the problem took quite a while and it may be having unintended effects on other parts of our application.

Thanks in advance,

Leif

[Stephen Steneker]

On Tuesday, 19 April 2016 13:05:56 UTC+10, Leif Mortenson wrote:

We have a very simple query that we use to get a list of documents sorted by their name. It returns 49 records when working correctly.

db.docs.find({}, {“_id”:true, “name”:true}).sort({“name”:1});

Normally this works fine, but every once in a while at the end of the returned list, one or more documents will be seen a second time. So we get duplicates. When this happens they are always at the end and appear to be sorted amongst themselves, but not as a whole.

So the “sorted list” with extra documents would be: A, B, C, D, E, F, G, H, I, C, E

I searched around and it looks like this might be being caused because we have other threads that are updating the records in the docs collection at the same time.

…

We are currently using Mongo 2.6.10. If you think this might have been resolved in a newer version we will look into the upgrade. I didn’t see anything in the release notes.

Hi Leif,

There have been a significant number of changes since 2.6.10, so I would definitely recommend testing the upgrade to a newer release series of MongoDB (i.e. latest 3.0.x or 3.2.x) to see if this issue still occurs.

I suspect you may be encountering SERVER-19996, which was fixed prior to the MongoDB 3.2.0 release and backported to 3.0.9.

If you are able to share a gist with a real query and explain(true)results, this could help confirm the issue.

Regards,
Stephen

​