Preventing race conditions when submitting forms

[Paul Johnston]

Hi,

Consider an e-commerce site, where Alice and Bob are both editing the product listings. Alice is improving descriptions, while Bob is updating prices. They start editing the Acme Wonder Widget at the same time. Bob finishes first and saves the product with the new price. Alice takes a bit longer to update the description, and when she finishes, she saves the product with her new description. Unfortunately, she also overwrites the price with the old price, which was not intended.

It's worth noting that the controller methods are thread-safe in themselves. They use database transactions, which make them safe in the sense that if Alice and Bob try to save at the precise same moment, it won't cause corruption. The race condition arises from Alice or Bob having stale data in their browser.

Does Django have any way to prevent these race conditions? Just rejecting the second edit with an "edit conflict" message would be a start - although some intelligent merging would be even better.

Thanks,

Paul

[Simon Charette]

Hi Paul,

You might want to take a look at the third party packages listed under the concurrency category.

I've used django-concurrency in the past and it worked well for me.

Simon

[Tim Chase]

On 2014-11-25 07:57, Paul Johnston wrote:
> Consider an e-commerce site, where Alice and Bob are both editing
> the product listings. Alice is improving descriptions, while Bob is
> updating prices. They start editing the Acme Wonder Widget at the
> same time. Bob finishes first and saves the product with the new
> price. Alice takes a bit longer to update the description, and when
> she finishes, she saves the product with her new description.
> Unfortunately, she also overwrites the price with the old price,
> which was not intended.

The common solution in this case your model would have something like
a "last_modified_timestamp" or "last_modified_counter". This would
be sent with the form and then resubmitted back. If the
timestamp/counter is the same, you know it's safe to update the
timestamp or increment the counter (done within the transaction to
ensure atomicity). If it's *not* the same timestamp/counter as it was
when the form was created, you know that it's been modified in the
interim and you can present a conflict-resolution form.

I don't think Django has anything like this out of the box, but it's
fairly straightforward to implement. I've done it enough times that
I should just create some model/form mixin to handle it for me.

-tkc

[Paul Johnston]

Tim, Simon,

Thanks for the responses. Looks like django-concurrency is a good fit, and it works the way Tim suggested (so no need to write your own mixins!)

One follow-up question: do you have any ideas for conflict resolution? The django-concurrency resolution is pretty basic, but it would be a lot of work to do something better.

Paul

[Simon Charette]

I can't think of a generic way of solving conflict resolution. I'd say it's highly application specific.

[Cal Leeming]

+1 - conflict resolution is not an easy task, and really depends on your business logic/use case.

It's worth mentioning that the approach django-concurrency uses may not be suitable for your use case, and in my opinion implementing this restriction on a per model basis is not the best approach, especially when handling multiple model scenarios. Also the implementation of django-concurrency is "dumb", if you look at the example it blocks two save() calls despite no fields being changed on them, and I suspect it doesn't attempt to do any sort of checks to see if the updates would have conflicted. This in itself would be enough reason to stay away from the library (if I'm mistaken, let me know).

django-locking is even worse, it uses a shotgun approach of locking the entire model.. I've used systems (as a user) in the past which implement a similar locking system, and they are a UX anti pattern (imho).

Also remember that representing conflicts in the UI is quite important, having an error at the top of the page telling you that a field conflicted really is quite ghetto and doesn't give a good user experience.. Trello handles this nicely, but they put a lot of time/effort into making the whole process seamless.

The above doesn't really answer your original question of "what can I do", but I hope it gives you some insight into the downsides of those other options.

Cal

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/1c0c9322-65cc-4c1d-a3bb-e8c80428d0f2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[donarb]

Another option to consider for the problem you describe is to separate the product description and pricing into separate tables linked by foreign key, then restrict editing of each based on the user's group permission. I'm currently working on a sort of B2B e-commerce project where pricing can be different for each customer based on contract agreements (10 or so prices per product right now). We don't yet have a need for separate staff to edit products and prices, but the capability is there if it's ever needed.

[Paul Johnston]

Hi,

Thanks for the warning on django-concurrency. While it seems to have the right features for me, I hadn't looked at its internals.

I was thinking about the UI for conflicts and I wonder if I should treat these as form validation errors. If we have the original data (D), the other change set (O), and the conflicting change set (C). On form submit, if there has been any change, the submit fails with an error like "Someone else edited this". We can attempt an automatic merge: any fields that are changed in one of O or C, the change is kept. If both O and C change the field (to different values) then we have a conflict. Conflicting fields get validation errors. I reckon the value should be taken from C, and the error message says "someone else changed this to O". Even if there are no conflicts, we still redisplay the form, because there might be some inter-field dependencies that need manual checking. Probably a good idea to highlight fields that changed in O. With this arrangement, if the user just clicks "Save" again, then the change sets will be merged, with conflicts resolved using C, which seems a reasonable default behaviour.

I agree that the optimal workflow is application specific. For example, if this was a request/approval system, and the administrator changes status to "approved" looking at a total cost of $5, if the user simultaneously changes the total cost to $1000000 then we don't want to automatically merge those changes!

Having a timestamp or counter in the model is actually unnecessary. An alternative is to submit the original data with the form save. In fact, to do the merge, we need that data anyway. And it seems neater to have this as a change that's isolated to the form processing layer.

Well, anyway, that's me just thinking aloud. Sounds like the beginnings of a plan for django-concurrency2. If I get a chance I may look at this in more detail.

Thanks for the input guys!

Paul

[Collin Anderson]

Hi,

On websites where there's a lot of editing happening, I've been trying to minimize the number of fields available on a page for editing. Ideally only the data that changed in the UI even get sent to the server. That way there's less chance of a conflict happening in the first place.

Collin

[Cal Leeming]

Sadly I have to say "good luck with that" if you're using Django, as the forms library does not support partial form data out of the box, and it's architectural design makes this difficult to achieve. This is a particularly annoying restriction of Django, especially when building REST APIs. I'm pretty sure I did a thread about this a while back, but can't seem to find it.

I have written a set of mixins that add support for this, specifically for our use case of building REST APIs, so it's compatibility with other modules is minimal (e.g. there's no django admin support), but it would give you a good base to work from. I haven't done a proper lib release for this yet, but here's a dump for reference;

https://gist.github.com/foxx/1968fe19f759e608452a

(this is super alpha, with no regression tests and zero guarantees of future or even current stability)

However - the above won't help you if you're using the traditional approach of binding together your frontend and backend (e.g. handling forms with traditional <form> post, rather than using JS), mostly because it doesn't make any attempt to modify form rendering, only form processing/validation. But if someone is serious about building an application with a proper UI and conflict resolution, they (probably) would be separating their API anyway, so this would become a non-issue.

And then of course we go even deeper into the rabbit hole, if you want to listen for event changes within the UI (e.g. warn about conflicts before submit is pushed) then using the "browser pull" method is not good enough (e.g. pull every X seconds to update your local storage). Instead you'd probably want to look into websockets with a pub/sub style approach, and this is something which is absolutely not supported by Django (and probably never will be).

But if we put the "big picture" to one side and look at this practically, unless you're planning on building a complete client side JS package, then you can pretty much ignore all of the above. Many people get along just fine with not separating frontend/backend, and it is arguably more work to do it properly - which is disappointing.

This is yet another topic I'm planning on doing a longer talk about in the next few months, along with a skeleton project to get people started quicker and hopefully change attitudes in the way forms are handled. It's all working internally, just need to spend some time making it good enough for public consumption :)

Cal

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/4acc27fc-917e-4c2a-8bf2-7bf58637671a%40googlegroups.com.