Implementing Optimistic Locking

Aurélien Geron

unread,

Apr 11, 2013, 6:44:00 AM4/11/13

to mongoeng...@googlegroups.com

Hi everyone,

I'm new to MongoDB, MongoEngine and I just joined this mailing list. :-)

I've read the docs and searched the web and this mailing list, and unless I'm mistaken there's no discussion about optimistic locking. Here's an example, where some data is lost because of a race condition:

from mongoengine import *

connect("test")

class Task(EmbeddedDocument):

text = StringField()

class TodoList(Document):

name = StringField()

tasks = ListField(EmbeddedDocumentField(Task))

def move_task(self, index, new_index):

self.tasks.insert(new_index, self.tasks[index])

if new_index <= index:

index += 1

del(self.tasks[index])

def __str__(self):

return self.name + ": " + ", ".join([task.text for task in self.tasks])

todo = TodoList(name="Organize birthday")

todo.tasks.append(Task(text = "Buy presents"))

todo.tasks.append(Task(text = "Send invitations"))

todo.tasks.append(Task(text = "Bake cake"))

todo.save() # <TodoList: Organize birthday: Buy presents, Send invitations, Bake cake>

todo_copy = TodoList.objects.get(id = todo.id) # another instance for the same mongodb document

todo_copy.tasks.insert(0, Task(text = "Prepare a poem"))

todo_copy.save() # <TodoList: Organize birthday: Prepare a poem, Buy presents, Send invitations, Bake cake>

todo.move_task(1, 0) # reordering arrays seems like a dangerous mongodb operation, without locking.

todo.save() # <TodoList: Organize birthday: Send invitations, Buy presents, Bake cake>

todo.reload() # <TodoList: Organize birthday: Send invitations, Buy presents, Bake cake>

When manipulating multiple Document instances that represent the same MongoDB document, we run the risk of losing data, in this case the "Prepare a poem" task is lost.

I know that pushing an element at the end of a list is atomic, and mongodb 2.4 now handles push+order, but there are lots of cases where that's just not enough, for example when you need to reorder a list.

Wouldn't it be nice if we could write something like this ?

...

class TodoList(Document):

version = IntField()

name = StringField()

...

todo.save(lock__version = true)

This would raise an exception in case the document's "version" was changed between (re)loading and saving the TodoList instance. If the save succeeds, the version field is incremented. Note that this syntax allows for multiple locks, even within one save/update operation.

We could also have something simpler, like this:

...

class TodoList(Document):

version = IntField(optimistic_lock = true)

name = StringField()

...

todo.save()

All save and updates would automatically check+increment the version field.

I guess we could also limit the lock to some specific fields:

...

class TodoList(Document):

version = IntField(optimistic_lock_for = ["tasks"])

name = StringField()

...

todo.save()

This would use the lock (meaning checking+incrementing) only if the tasks field is modified.

One last option:

...

@optimistic_lock

class TodoList(Document):

name = StringField()

...

todo.save() # all save and updates would automatically check+increment a "_ver" field.

We may not need all (or any) of the previous options. Does this sound cool? Do you have any other idea?

Cheers,

Aurélien

Ross Lawley

unread,

Apr 11, 2013, 8:59:28 AM4/11/13

to mongoeng...@googlegroups.com

Ho Aurelien,

That is definitely one solution and I'd love for an implementation to be added to mongoengine! So anyone want to send a pull request :D I'd happily help get it into the codebase :)

However, a there is a solution now with MongoDB - do not to use save ;) Its problematic because it is an overwrite and not an atomic update. Operators like $set and $push help ensure you dont overwrite data across inserts across threads.

Ross

--
You received this message because you are subscribed to the Google Groups "MongoEngine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongoengine-us...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Aurélien Geron

unread,

Apr 15, 2013, 9:05:27 AM4/15/13

to mongoeng...@googlegroups.com

Hi Rozza,

I just implemented a first draft. It's available for testing and commenting on my github clone of mongoengine:

https://github.com/ageron/mongoengine

It's not exactly what I promised, but it does the job. Basically, the idea is that every field can have a list of version_locks, which are the names of IntFields that should be checked and incremented whenever that field is modified and the document is saved.

Here's an example:

class Task(EmbeddedDocument):

description = StringField()

class TodoList(Document):

name = StringField()

tasks_version = IntField(db_field = "tver")

tasks = ListField(EmbeddedDocumentField("Task", db_field="t"), version_locks = ["tasks_version"])

Now whenever a task list is saved, if the tasks field is modified (the list itself or any task in the list), then the tasks_version IntField is checked against the value in mongodb, and incremented if it has the expected value. If it's not the expected value, then a VersionLockError is raised. For example (taken from tests/test_version_locks.py):

todo_list1 = self.TodoList(name='Test').save()

todo_list2 = self.TodoList.objects.get(id = todo_list1.id)

self.assertIsNone(todo_list1.tasks_version)