pymongo - Bulk operations can only be executed once

[Stephan]

Hi pymongo team,

is there a particular reason why a bulk cannot be executed twice ?
I have the situation that 'preparing' the bulk is a very complicated/expensive operation.
However in case of a change of master bulk.execute() fails with an error (AutoReconnect/NotMaster) and I have no way to execute the operations again.
What is your recommendation here ?

Regards,
 Stephan

[Bernie Hackett]

Not being able to execute the same bulk twice was a design decision for the fluent bulk API for all drivers, not just PyMongo. If you have a primary failover during the bulk write operation you probably don't want to just call execute() again, since some of the operations in the bulk could have already been successful. That being said, the 3.0 version of the Python driver introduces a bulk_write method that just executes a list of write operations provided by your application. If you want to retry the bulk write you can just pass the list to the bulk_write method again. The docs are here:

http://api.mongodb.org/python/current/api/pymongo/collection.html#pymongo.collection.Collection.bulk_write

[Stephan]

Hi Bernie,

thanks for the fast replay. I looked through the code (mine and pymongo) and realized its harder than a I thought.
I identified these cases

1) Assuming a unordered bulk insert like this
* bulk.insert {_id: 1, ....}
* bulk.insert {_id: 2, ....}
.......
* bulk.insert {_id: 20000, ....}
* bulk.execute()

Looking inside pymongo/message.py I could see that, if the entire bulk is to big, the driver splits up the bulk into smaller batches. Lets assume my 20k inserts are divided into two batches.
The first batch successfully ran on the primary, but during preparation of the second batch this node became secondary.
In that case I got and AutoReconnect exception, but having no idea that 10k of the documents are already inserted.
I could retry or do a find to identify if the document is already present. However then I'm unable to detect if another part/thread/process of my application already inserted this document, making it impossible to notify the caller about that.

2) Using $inc as update modifier

* bulk.find({id: 1}).update_one({$inc: {x: 1}, ....})
* bulk.find({id: 2}).update_one({$inc: {x: 1}, ....})
.......
* bulk.find({id: 20000}).update_one({$inc: {x: 1}, ....})


Lets assume again after the first 10k the server became secondary. There is no easy way to know which documents are already incremented or not.

I was thinking to avoid this 'split of the entire bulk into batches' on my application. For example just send 1000 operations per bulk, so it will never reach the max size per bulk. However I'm not sure if it will solve the problem entirely. It could still happen that during the execution of this small bulk, the mongodb-server can become secondary, right? So I will also get an AutoReconnect where a few operations already happened.


Anyway, do you know existing application code (can be any project Java/c++) which deals with bulks and AutoReconnect ?
A link to github is enough. I just wanted to see how other people are tackling this problem.

Regards,
Stephan

[Bernie Hackett]

This is indeed a difficult problem to solve. The problem is made more difficult if you are doing a non-idempotent update() operation since you have no idea how many of the documents were updated on the server. One thing PyMongo might be able to do to help a little is report everything it knows has already happened when AutoReconnect is raised during a bulk.execute(). For example, if there is a write error (e.g. DuplicateKeyError) during execute() we raise a BulkWriteError with a 'details' attribute through which you can access the "results so far". Though, to be honest, that also won't be that useful. You still won't know how many operations in a single batch were executed on the server since the driver never gets that result. You will only see the aggregate results for batches the server was able to respond to.  

[Stephan]

Hi Bernie,

 

The problem is made more difficult if you are doing a non-idempotent update() operation

I guess this is also true for ordinary multi updates like "update_many({}, {"$inc": {'x': 1}})". What does the server do if during such an operation the node becomes secondary? I would expect also an AutoReconnect error.


Regarding the bulk interface. I agree, a good option would be that the driver/server reports (via the detail attribute in the exception object) what changes were done, before the switch of roles happened.
However I understand this is nothing coming in short term.

Another question. What happens if I enable sharding. For example during an bulk update, affecting many shards, on one shard a switch of roles happens.
How does mongos handle this and what is reported to the user of pymongo ?

Regards,
 Stephan
 
 

[Bernie Hackett]

> I guess this is also true for ordinary multi updates like "update_many({}, {"$inc": {'x': 1}})".

That's a great example of a "non-idempotent" update. That is, every time you $inc a field the value increases by one. Compare that to something like $set, where no matter how many times you set the field to a particular value the result is the same.

> What does the server do if during such an operation the node becomes secondary?

When the primary becomes a secondary it will compare its oplog to that of the new primary and roll back any operations that were not replicated before the stepdown / failover. See the docs for more information:

http://docs.mongodb.org/manual/core/replica-set-rollbacks/

> I would expect also an AutoReconnect error.

Yes. When the primary steps down (because you told it to, or because it can no longer communicate with a majority of the replica set), it immediately closes all network connections.

> Regarding the bulk interface. I agree, a good option would be that the driver/server reports (via the detail attribute in the exception object) what changes were done, before the switch of roles happened.

I'll have to think about how this should work. The information might be useful for things like insert, update_one, replace_one, or remove_one (though in the case of a network error the operation could occur on the server without the driver getting a response). It's not as clear for update and remove.

> How does mongos handle this and what is reported to the user of pymongo ?

AutoReconnect is raised by PyMongo when a network error occurs between itself and the server it is connected to. If the problem is between mongos and one or more shards, and mongos reports the error back to PyMongo, the driver will raise OperationFailure (or a subclass of OperationFailure like BulkWriteError).