how to handle race conditions?
146 views
Skip to first unread message
Abraham Varricatt
Feb 18, 2015, 5:01:08 AM2/18/15
to django...@googlegroups.com
Hello,
I'm trying to make an app where folks can order X quantity of an item. The condition is that the order should only be made if inventory exists. Assume that we have stock of Y items. This means that only if Y >= X should we allow the sale to go through. And once we accept an order, the inventory should be updated so that Y = Y - X.
The way I've currently implemented this is with a pair of getter/setter methods. I have a model 'ShopItem' which has two fields called 'quantity' and 'name' (this is unique). I've made a utility class (outside model source file) where I've made the following 2 functions;
def get_quantity(name):
stuff_left = 0
try:
item = ShopItem.objects.get(name=name)
stuff_left = item.quantity
except ShopItem.DoesNotExist:
pass # return zero
return stuff_left
def set_quantity(name, stuff_left):
item = ShopItem.objects.get(name=name)
item.quantity = stuff_left
item.save()
Elsewhere in my project, if I need to display the remaining quantity of an item in a view, I'll pass the result from get_quantity(). At sale time here is how I update things;
def customer_buy(name, number):
temp = get_quantity(name)
if (temp >= number):
temp = temp - number
set_quantity(name, temp)
// do other things
else:
// sale failed, do something else
I tested this on my system and things appear to work. But I'm concerned about race conditions. What if two different customers came in to buy the same item? As in; if I only have 7 items in stock, but one customer came to buy 6nos and another 5nos. There is a chance (as per my understanding) that both orders will be accepted - even worse, my inventory will not accurately reflect the updated situation.
Any ideas on how this issue can be resolved? I've come across this - https://docs.djangoproject.com/en/1.6/ref/models/instances/#updating-attributes-based-on-existing-fields , but not sure how to apply it to the current scenario.
Learning django,
Abraham V.
aRkadeFR
Feb 18, 2015, 8:57:35 AM2/18/15
to django...@googlegroups.com
Hello,
My opinion on the problem:
A database is ACID, so there will be no problem if every information
are stored in the database.
There should be a "available_quantity" property on a item.
This will allow you to have more control of the stock too.
If a customer put an item in his basket this checks the
available quantity of the item and update it. So you
succeed to sell only available item.
Then the release item problem:
If the user didnt buy the item before X minutes, you need
to forbid his basket and you put again his items inside
the available_quantity.
Don't know if it helped you?
Thanks,
PS:
May I suppose to you that the get_quantity function and set_quantity
should be in the manager of the model?
something like:
```
class ShopItemManager(Manager):
def get_quantity(self, name):
return self.get_queryset().get(name=name).quantity
class ShopItem(Model):
...
objects = ShopItemManager()
```
This allow you to tied up your function to the model.
My opinion on the problem:
A database is ACID, so there will be no problem if every information
are stored in the database.
There should be a "available_quantity" property on a item.
This will allow you to have more control of the stock too.
If a customer put an item in his basket this checks the
available quantity of the item and update it. So you
succeed to sell only available item.
Then the release item problem:
If the user didnt buy the item before X minutes, you need
to forbid his basket and you put again his items inside
the available_quantity.
Don't know if it helped you?
Thanks,
PS:
May I suppose to you that the get_quantity function and set_quantity
should be in the manager of the model?
something like:
```
class ShopItemManager(Manager):
def get_quantity(self, name):
return self.get_queryset().get(name=name).quantity
class ShopItem(Model):
...
objects = ShopItemManager()
```
This allow you to tied up your function to the model.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/7552759e-81d6-4cfb-a0ce-4fed95eef5bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Carl Meyer
Feb 18, 2015, 6:33:57 PM2/18/15
to django...@googlegroups.com
Hi Abraham,
On 02/17/2015 10:01 PM, Abraham Varricatt wrote:
> I'm trying to make an app where folks can order X quantity of an item.
> The condition is that the order should only be made if inventory exists.
> Assume that we have stock of Y items. This means that only if Y >= X
> should we allow the sale to go through. And once we accept an order, the
> inventory should be updated so that Y = Y - X.
In general, the construct you need is called a "transaction", which
ensures that a series of database operations will either all be
committed together, or rolled back if they can't be successfully
completed. The Django API for that is "django.db.transaction.atomic".
If we're only considering the changes to your ShopItem model, you don't
even need an explicit transaction to avoid race conditions, because (as
the docs you linked show) the operation can be completed in a single
database query, which is inherently atomic. This is how it would look to
do it in a single query:
from django.db.models import F
def customer_buy(name, number):
ShopItem.objects.filter(
name=name).update(quantity=F('quantity')-number)
You want this to fail if someone tries to purchase a larger quantity
than are available. The best way to do this is via a database-level
"check constraint" on the column, such that the database itself will
never permit a negative quantity. If you make 'quantity' a
PositiveIntegerField (the name is wrong, it actually allows zero too) on
your model, and your database is PostgreSQL (or Oracle), Django will add
this constraint for you automatically. Then if someone tries to purchase
more than are available, you'll get an IntegrityError, which you'd want
to catch and handle in some way:
from django.db import IntegrityError
from django.db.models import F
class InsufficientInventory(Exception):
pass
def customer_buy(name, number):
try:
ShopItem.objects.filter(
name=name).update(quantity=F('quantity')-number)
except IntegrityError:
# signal to the calling code that the purchase failed - the
# calling code should catch this exception and notify the
# user that the purchase failed due to lack of inventory,
# and tell them the updated available quantity
raise InsufficientInventory()
You also want this function to handle the case where the given product
name doesn't exist. To help with this case, the `update` method returns
the number of rows updated:
from django.db import IntegrityError
from django.db.models import F
class InsufficientInventory(Exception):
pass
def customer_buy(name, number):
try:
updated = ShopItem.objects.filter(
name=name).update(quantity=F('quantity')-number)
except IntegrityError:
raise InsufficientInventory()
if not updated:
# Here we reuse Django's built-in DoesNotExist exception;
# you could define your own exception class instead.
raise ShopItem.DoesNotExist()
With this code, you've solved the bad race conditions -- quantity will
never go negative, and a sale will never appear to succeed when it
should have failed, because of two users submitting an order simultaneously.
There is still a sort of higher-level race condition that can happen
when two people both load the order page at the same time, and then one
of them orders first. This type of race condition is basically
impossible to avoid in a web app -- the best you can usually do is
simply let a user know that their order failed because someone else
ordered that item first while they were filling out the order form.
(In general, the term for this approach is "optimistic locking", because
you are "optimistically" allowing concurrent uses and hoping they don't
conflict, but if they do, you catch it and alert the user so they can
try again. The alternative is to lock everyone else out of the order
page while one user is ordering; this is called "pessimistic locking".
Usually optimistic locking is preferable, presuming the most common case
is "no conflict" -- e.g., enough inventory that both users can
successfully complete their order -- and you can handle conflicts in
such a way that the user whose transaction fails doesn't lose all their
work.)
In a real shopping-cart scenario, it's likely that you need to do more
than just update the quantity field on your ShopItem model -- you may
need to create an Order object too, and you want the Order object and
the ShopItem quantity decrement to both succeed or fail together; you
never want one of them to succeed and the other one to fail. This is the
scenario where you need to wrap both operations in `transaction.atomic`;
you can look it up in the docs to see how it's used.
Hope this helps,
Carl
On 02/17/2015 10:01 PM, Abraham Varricatt wrote:
> I'm trying to make an app where folks can order X quantity of an item.
> The condition is that the order should only be made if inventory exists.
> Assume that we have stock of Y items. This means that only if Y >= X
> should we allow the sale to go through. And once we accept an order, the
> inventory should be updated so that Y = Y - X.
ensures that a series of database operations will either all be
committed together, or rolled back if they can't be successfully
completed. The Django API for that is "django.db.transaction.atomic".
even need an explicit transaction to avoid race conditions, because (as
the docs you linked show) the operation can be completed in a single
database query, which is inherently atomic. This is how it would look to
do it in a single query:
from django.db.models import F
def customer_buy(name, number):
ShopItem.objects.filter(
name=name).update(quantity=F('quantity')-number)
You want this to fail if someone tries to purchase a larger quantity
than are available. The best way to do this is via a database-level
"check constraint" on the column, such that the database itself will
never permit a negative quantity. If you make 'quantity' a
PositiveIntegerField (the name is wrong, it actually allows zero too) on
your model, and your database is PostgreSQL (or Oracle), Django will add
this constraint for you automatically. Then if someone tries to purchase
more than are available, you'll get an IntegrityError, which you'd want
to catch and handle in some way:
from django.db import IntegrityError
from django.db.models import F
class InsufficientInventory(Exception):
pass
def customer_buy(name, number):
try:
ShopItem.objects.filter(
name=name).update(quantity=F('quantity')-number)
except IntegrityError:
# signal to the calling code that the purchase failed - the
# calling code should catch this exception and notify the
# user that the purchase failed due to lack of inventory,
# and tell them the updated available quantity
raise InsufficientInventory()
You also want this function to handle the case where the given product
name doesn't exist. To help with this case, the `update` method returns
the number of rows updated:
from django.db import IntegrityError
from django.db.models import F
class InsufficientInventory(Exception):
pass
def customer_buy(name, number):
try:
updated = ShopItem.objects.filter(
name=name).update(quantity=F('quantity')-number)
except IntegrityError:
raise InsufficientInventory()
if not updated:
# Here we reuse Django's built-in DoesNotExist exception;
# you could define your own exception class instead.
raise ShopItem.DoesNotExist()
With this code, you've solved the bad race conditions -- quantity will
never go negative, and a sale will never appear to succeed when it
should have failed, because of two users submitting an order simultaneously.
There is still a sort of higher-level race condition that can happen
when two people both load the order page at the same time, and then one
of them orders first. This type of race condition is basically
impossible to avoid in a web app -- the best you can usually do is
simply let a user know that their order failed because someone else
ordered that item first while they were filling out the order form.
(In general, the term for this approach is "optimistic locking", because
you are "optimistically" allowing concurrent uses and hoping they don't
conflict, but if they do, you catch it and alert the user so they can
try again. The alternative is to lock everyone else out of the order
page while one user is ordering; this is called "pessimistic locking".
Usually optimistic locking is preferable, presuming the most common case
is "no conflict" -- e.g., enough inventory that both users can
successfully complete their order -- and you can handle conflicts in
such a way that the user whose transaction fails doesn't lose all their
work.)
In a real shopping-cart scenario, it's likely that you need to do more
than just update the quantity field on your ShopItem model -- you may
need to create an Order object too, and you want the Order object and
the ShopItem quantity decrement to both succeed or fail together; you
never want one of them to succeed and the other one to fail. This is the
scenario where you need to wrap both operations in `transaction.atomic`;
you can look it up in the docs to see how it's used.
Hope this helps,
Carl
Mario Gudelj
Feb 19, 2015, 12:04:39 AM2/19/15
to django...@googlegroups.com
Great reply. You're a champion, Carl!
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/54E4DA7F.5090902%40oddbird.net.
Waitman Gobble
Feb 19, 2015, 3:46:07 AM2/19/15
to django...@googlegroups.com
Hmm.. but I think you don't want to tell the customer with $1,000
order that the order 'failed' over a ten dollar item. They will go
somewhere else. IMHO It's probably better to take both orders and deal
with it on a personal level. Especially if the 'winning' customer has
a $12.50 order. Also there's the 'backorder' possibility.
For example, I ordered a raspberry pi 2 when it hit the market. When I
ordered the site specifically stated the item was in stock. Two days
after my order I sent an email asking about the status and they told
me it was backordered and I would receive in about 10 days. I was
kinda ticked off and at that time considered that I'd not order from
them again in the future, however now that I have the item I'm really
over it and probably actually would make another order from that
vendor. However, if the site indicated that my order 'failed' as I
completed checkout, I would have gone to another site without
hesitation.
--
Waitman Gobble
Los Altos California USA
510-830-7975
aRkadeFR
Feb 19, 2015, 8:18:25 AM2/19/15
to django...@googlegroups.com
On 02/19/2015 01:04 AM, Mario Gudelj
wrote:
Great reply. You're a champion, Carl!
Couldn't say more. Very thorough answer and interesting :)
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAHqTbjmfzZLSFc0Bmo3irt4X2594BkobVZvn_8brLXTV1YkkYQ%40mail.gmail.com.
Abraham Varricatt
Feb 19, 2015, 10:41:55 AM2/19/15
to django...@googlegroups.com
@Carl,
It's an interesting idea - to mark the field in the database as a positive integer, but since it doesn't apply to all DBs, I think I'll keep it aside for now.
@Waitman,
Don't think I've ever come across the term 'backorder', but its an interesting concept. And very appreciated, from the business angle. I need to think more on this.
Thanks for the help,
Abraham V.
Carl Meyer
Feb 19, 2015, 5:29:52 PM2/19/15
to django...@googlegroups.com
Hi Abraham,
On 02/19/2015 03:41 AM, Abraham Varricatt wrote:
> I've heard of the term 'atomic operations' , but didn't realize it had
> support in django. Thank you for the detailed explanation!
You're welcome.
> It's an interesting idea - to mark the field in the database as a
> positive integer, but since it doesn't apply to all DBs, I think I'll
> keep it aside for now.
This is a bit of a tangent, but when you get to the point of worrying
about things like race conditions and transactional behavior, your life
will be easier if you simply choose a database for your project and
don't worry about making everything cross-database portable.
And, frankly, if you're concerned about data integrity, your life will
be easier if the database server you choose is PostgreSQL.
Your ability to use Django effectively is much greater if you know your
specific database well, and aren't afraid to use database-specific
features and write some raw SQL now and then. Limiting yourself to
cross-database-portable ORM features is a significant handicap, one
which I think only separately-distributed reusable apps should take on.
In your particular case, without a database-level check constraint on
the quantity field, it becomes much trickier to ensure that quantity
can't go negative.
> @Waitman,
>
> Don't think I've ever come across the term 'backorder', but its an
> interesting concept. And very appreciated, from the business angle. I
> need to think more on this.
Yes - I should have said more clearly that my answer was intended as a
primer on basic atomicity and locking concepts, not a treatise on
shopping cart user experience. It's quite likely that for a production
shopping cart you'd want significantly more sophisticated behavior than
I covered.
Carl
On 02/19/2015 03:41 AM, Abraham Varricatt wrote:
> I've heard of the term 'atomic operations' , but didn't realize it had
> support in django. Thank you for the detailed explanation!
> It's an interesting idea - to mark the field in the database as a
> positive integer, but since it doesn't apply to all DBs, I think I'll
> keep it aside for now.
about things like race conditions and transactional behavior, your life
will be easier if you simply choose a database for your project and
don't worry about making everything cross-database portable.
And, frankly, if you're concerned about data integrity, your life will
be easier if the database server you choose is PostgreSQL.
Your ability to use Django effectively is much greater if you know your
specific database well, and aren't afraid to use database-specific
features and write some raw SQL now and then. Limiting yourself to
cross-database-portable ORM features is a significant handicap, one
which I think only separately-distributed reusable apps should take on.
In your particular case, without a database-level check constraint on
the quantity field, it becomes much trickier to ensure that quantity
can't go negative.
> @Waitman,
>
> Don't think I've ever come across the term 'backorder', but its an
> interesting concept. And very appreciated, from the business angle. I
> need to think more on this.
primer on basic atomicity and locking concepts, not a treatise on
shopping cart user experience. It's quite likely that for a production
shopping cart you'd want significantly more sophisticated behavior than
I covered.
Carl
Reply all
Reply to author
Forward
0 new messages