PyMongo BulkWriteError Error with $inc operator

[Rhys Campbell]

Hello All,

I am having problems with the following bulk upsert...

            bulk.find(
                { "_id": new_document["_id"] }
            ).upsert().update( {
                        "$setOnInsert": {
                                "xxxx_ts": new_document["xxxx_ts"],
                                "xxxx_state": new_document["xxxx_state"],
                                "xxxx_xxxx_hash": new_document["xxxx_xxxx_hash"],
                                "version": 0
                            },
                        "$set": {
                                "xxxx_ts": new_document["xxxx_ts"],
                                "xxxx_state": new_document["xxxx_state"],
                                "xxxx_xxxx_hash": new_document["xxxx_xxxx_hash"],
                                "$inc": { "version": 1 } # Doesn't work WHY???
                            }
                        }
            )

Then I execute it with;

result = bulk.execute()

Then I receive the following error;

pymongo.errors.BulkWriteError: batch op errors occurred

The problem in the $inc operator. When I remove this the update works fine. The documentation has limited examples but I'm not sure what I'm doing wrong;

http://api.mongodb.com/python/current/examples/bulk.html

Can anyone assist?

Cheers,

Rhys

[Bernie Hackett]

You can't nest one atomic operator inside of another. The $inc has to happen outside of the $set. In your example MongoDB thinks you are trying to add a key called $inc, and keys can't be prefixed with $.

[Rhys Campbell]

Mmm, ok. I think I tried placing it outside and had problems too. I'll have another crack at it tomorrow.

Cheers,

Rhys

[Rhys Campbell]

I've found some better examples here;

https://docs.mongodb.com/manual/reference/method/Bulk.find.upsert/index.html

This is what I'm working with...

            bulk.find({ "_id": new_document["_id"] }).upsert().update( {
                        "$setOnInsert": {
                                "xxxx_ts": new_document["xxxx_ts"],
                                "xxxx_state": new_document["xxxx_state"],
                                "xxxx_xxxx_hash": new_document["xxxx_xxxx_hash"],
                                "version": 0
                            },

                       "$inc": { "version": 1 },

                       "$set": {
                                "xxxx_ts": new_document["xxxx_ts"],
                                "xxxx_state": new_document["xxxx_state"],
                                "xxxx_xxxx_hash": new_document["xxxx_xxxx_hash"]
                            }
                        }

            )

I played around with the location of the $inc operator in the query all with the same error.

Traceback (most recent call last):
  File "ozfs_expected_state.py", line 85, in <module>
    raise(excep)
  File "ozfs_expected_state.py", line 73, in <module>
    result = bulk.execute()
  File "/usr/local/lib/python3.5/dist-packages/pymongo/bulk.py", line 675, in execute
    return self.__bulk.execute(write_concern)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/bulk.py", line 493, in execute
    return self.execute_command(sock_info, generator, write_concern)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/bulk.py", line 331, in execute_command
    raise BulkWriteError(full_result)

pymongo.errors.BulkWriteError: batch op errors occurred

Then I suddently noticed that it appear to work or at least didn't spit out an error... (Note the successful script execution in bold)

demo$ python3 ozfs_expected_state.py
Traceback (most recent call last):
  File "ozfs_expected_state.py", line 85, in <module>
    raise(excep)
  File "ozfs_expected_state.py", line 73, in <module>
    result = bulk.execute()
  File "/usr/local/lib/python3.5/dist-packages/pymongo/bulk.py", line 675, in execute
    return self.__bulk.execute(write_concern)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/bulk.py", line 493, in execute
    return self.execute_command(sock_info, generator, write_concern)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/bulk.py", line 331, in execute_command
    raise BulkWriteError(full_result)

pymongo.errors.BulkWriteError: batch op errors occurred

demo$ python3 ozfs_expected_state.py
demo$ python3 ozfs_expected_state.py
Traceback (most recent call last):
  File "ozfs_expected_state.py", line 85, in <module>
    raise(excep)
  File "ozfs_expected_state.py", line 73, in <module>
    result = bulk.execute()
  File "/usr/local/lib/python3.5/dist-packages/pymongo/bulk.py", line 675, in execute
    return self.__bulk.execute(write_concern)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/bulk.py", line 493, in execute
    return self.execute_command(sock_info, generator, write_concern)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/bulk.py", line 331, in execute_command
    raise BulkWriteError(full_result)

Note this is repeatable but seems random. Sometimes it work successfully after an attempt or two, other times it fails twenty times in a row. I have confirmed the increment is working when the script doesn't raise an error;

rhys:PRIMARY> db.ozfs_expected_states.find({}, { "_id": 0, "version": 1 }).pretty()
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }
{ "version" : 6 }

[Rhys Campbell]

Extra info... The error;

"pymongo.errors.BulkWriteError: batch op errors occurred"

Always occurs when the collection is empty. The variable success only occur when the collection has some data in it already.

[Rhys Campbell]

* Bump *

Any ideas on this one? The problem seems to be repeatable. Looks like a bug.

[Wan Bachtiar]

Any ideas on this one?

Hi Rhys,

It’s been a while since you posted this question, have you found the answer yet ?

This is because $setOnInsert is combined with non-insert modifiers.
If you catch the BulkWriteError exception and print out the details you should be able to see an error message as below:

'writeErrors': [{'code': 40,
                  'errmsg': "Updating the path 'xxxx_ts' would create a "
                            "conflict at 'xxxx_ts'",
...

There is currently an open ticket for this SERVER-10711, feel free to upvote/watch for notifications.

Regards,
Wan.

​