Duplicate Key Error on Upsert

[James Gregory]

Hi all,

I just got a weird error sent through from our application: it was
complaining of a duplicate key error on a collection with a unique index
on it, but the operation in question was an upsert, with the query part
matching a document by that unique key. I'd have expected that if the
key already existed that it would have just done an update, rather than
an insert.

Mongo 1.6.5, via the pymongo connector.

The exception is as follows:

DuplicateKeyError: E11000 duplicate key error index:
whereoscope.userDevices.$deviceId_1 dup key: { : "xxxxxxxx" }

And the line of code that triggered it was this:

res = self.db.userDevices.update({ 'deviceId' : deviceId }, { '$set'
: { key : deviceDetails } }, upsert=True, multiple=False, safe=True)

Some of the names above have been changed to protect the innocent. 'key'
above is one of handful of vanilla looking strings, notably not
'deviceId'. We are using replica sets, but that query was sent to the
master (we send all writes to the master, only do reads from slaves).

Any thoughts on what could be going wrong here?

Cheers,

James.

--
See where your kids are: http://whereoscope.com

[Eliot Horowitz]

Can you send a full script that reproduces if possible?
I can't reproduce simply.

Do you have any other unique indexes?

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

[James Gregory]

Not trivially. We've only see this happen once (and we would see it: our
servers email us whenever there's a runtime error), and the method in
question is called thousands of times a day. There are a bunch of other
indices on this collection, but that is the only unique one.

James.

[dan]

Did you ever get info on this? I am seeing the same thing with mongo-2.1.jar in Java:

Exception in thread "Thread-33" com.mongodb.MongoException$DuplicateKey: E11000 duplicate key error index: <db>.users.$_id_  dup key: { : "t:Niketown588" }
        at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:129)
        at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:142)
        at com.mongodb.DBApiLayer$MyCollection.update(DBApiLayer.java:256)
        at com.mongodb.DBCollection.update(DBCollection.java:114)
...

and the code is an upsert:


        users.update(userMatch, userMod, true /* upsert */, false /* multi */);

Both userMatch and userMod have _id, but that's okay I think.

[Scott Hernandez]

Are both _id fields the same, in the query + update-operations? There
is no need for the _id in the update if you specify it in the query.

[Dan Frankowski]

Is there a need for _id if the record doesn't exist? It's an upsert, it might be new or an update.

And why does it work 99.9% of the time? I can't reproduce the error, I just see it occasionally.

Dan

[Dan Frankowski]

I should answer your question: yes, "_id" is the same in both objects (the same var theId that is used in both):

        String theId = user.getDatabaseId();
        BasicDBObject userMatch = new BasicDBObject("_id", theId);

        // Upsert the basic fields
        BasicDBObject userMod = new BasicDBObject();
        userMod.put("_id", theId);
        // userMod.put more fields..

Thanks for your suggestion.

Dan

On Fri, Jan 7, 2011 at 2:42 PM, Scott Hernandez <scotthe...@gmail.com> wrote:

[dan]

Switched to mongo-java-driver-2.4.jar and I still see the same thing. I see a few a day now. There is something about mongo I don't understand.

[Nat]

Can you show what your update statement / criteria / new document look
like?

[Dan Frankowski]

Previous messages in this thread have the code (an upsert on _id), and the exception has the dup field (_id). Copied from https://groups.google.com/d/topic/mongodb-user/2W4A8QDr03s/discussion:


Exception in thread "Thread-33" com.mongodb.MongoException$

DuplicateKey: E11000 duplicate key error index: <db>.users.$_id_  dup key: { : "t:Niketown588" }

        at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:129)
        at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:142)
        at com.mongodb.DBApiLayer$MyCollection.update(DBApiLayer.java:256)
        at com.mongodb.DBCollection.update(DBCollection.java:114)
...

and the code is an upsert:

        String theId = user.getDatabaseId();
        BasicDBObject userMatch = new BasicDBObject("_id", theId);

        // Upsert the basic fields
        BasicDBObject userMod = new BasicDBObject();
        userMod.put("_id", theId);
        // userMod.put more fields..

Do you mean an example document?

> db.users.find({_id:"t:Niketown588"})
{ "_id" : "t:Niketown588", "screen_name" : "Niketown588", "user_name" : "Thomas Jones", "description" : "WWT DCSE, Host of CWT Podcast, EMCTAe-Symm, BCFD (8Gb/s), CCNA, ICSNS, and MSCE Jedi Knight. Hope to become Jedi Master soon.", "last_mention_date" : "Fri Jan 07 2011 12:42:34 GMT-0600 (CST)", "profile_image_url" : "http://a0.twimg.com/profile_images/427492922/6292_11294079_normal.jpg" }

Dan

[Scott Hernandez]

We can't reproduce this.

If all you are doing is querying the _id field and replacing/inserting
the new doc you can use save.

If your first example you were using "deviceId" and that would make
sense as a point of duplicate key failure if your query isn't just the
_id.

Can you post your full code where you are setting the values and
calling update/save? So far you have only posted pieces without
connecting sections. Also, I would suggest catching the duplicate
exception and logging the update operation, and the params; that will
give us a much better idea of what is going on.

[Dan Frankowski]

I am willing to believe this is programmer error, I just don't see it yet. You are right, maybe logging the info will show me some data is getting mangled. Unfortunately, I have other priorities, so this does not get my full attention.

Dan

The full code of the class:

public class MongoUserWriter implements UserWriter {
    public static final String LAST_MENTION_DATE = "last_mention_date";
    public static final String DESCRIPTION = "description";
    public static final String USER_NAME = "user_name";
    public static final String SCREEN_NAME = "screen_name";

    public static final String USERS = "users";
    public static final String PROFILE_IMAGE_URL = "profile_image_url";

    private DBCollection users;

    public MongoUserWriter(Config config) throws UnknownHostException,
            ConfigException {
        DB db = MongoDatabase.theDb(config);

        users = db.getCollection(USERS);

        // Throw errors instead of ignoring them
        users.setWriteConcern(WriteConcern.SAFE);
    }

    @Override
    public void writeUser(MentionUser user) {

        String theId = user.getDatabaseId();
        BasicDBObject userMatch = new BasicDBObject("_id", theId);

        // Upsert the basic fields
        BasicDBObject userMod = new BasicDBObject();
        userMod.put("_id", theId);

        userMod.put(SCREEN_NAME, user.getScreenName());
        if (user.getUserName() != null)
            userMod.put(USER_NAME, user.getUserName());
        String desc = user.getDescription();
        if (desc != null && !desc.isEmpty()) {
            userMod.put(DESCRIPTION, user.getDescription());
        }
        userMod.put(LAST_MENTION_DATE, user.getLastMentionDate());
        if (user.getProfileImageUrl() != null)
            userMod.put(PROFILE_IMAGE_URL, user.getProfileImageUrl());

        users.update(userMatch, userMod, true /* upsert */, false /* multi */);
    }
}

[Scott Hernandez]

I don't see anything (obviously) wrong either.

This can be re-written as users.save(userMod); It shouldn't make a
difference, since save just checks for the _id value and does an
update w/upsert like you are, but it does read better and is less code
for you to write.

Getting that logging/debug output would help a lot :)

[Mikhail Baturov]

Hi, we get the same error when we remove some rows in mongo shell and then trying to insert rows w/ same _id - it shows this error, though there is such data in it (find operation doesn't show them)

понедельник, 10 января 2011 г., 20:39:58 UTC+3 пользователь dan написал:

[Luke Kende]

I ran into this error and I know why.   

My filter for finding the doc to update and my insert fields were mismatched.  For example,  I always filter on user_id field, but then I forgot to add the user_id field into the new document that got inserted.

Now every subsequent call to update that document fails to find it with the user_id filter but then fails to insert it because the _id does exist already.

Hope that helps someone.