Deleting a record exactly once

[Ian Hoffman]

Hi everyone,

I work on a medium-sized app where we might have tens of concurrent requests in flight at once. The application runs on Django 1.11, Python 3.5.

The problem I have is this: the web client requests to delete a record. On a successful delete, the app creates an "audit" record memorializing this delete, containing information about the record which was deleted. This is used in ETL jobs later. (I know there might be better ways of doing this sort of tracking, but I'm not looking to change that right now.)

Now, it's possible that, by toggling a button, the web client can send a stream of delete and create requests for this record. These may be received out-of-order by various server instances running in the cloud. So the following sequence (corresponding to CREATE -> DELETE -> CREATE -> DELETE) can happen:

1. Request: Create record 1 for Server A
   
2. Request: Create record 1 for Server B
3. Server A: Record 1 created, audit record cut
   
4. Server B: No-op (Record 1 already exists)
   
5. Request: Delete record 1 for Server A
6. Request: Delete record 1 for Server B
7. Server A: Record 1 deleted, audit record cut
   
8. Server B: Record 1 deleted, audit record cut

This demonstrates that we may have 2 delete audit records for a single create audit record, which is just wrong.

I tried a fix along the following lines:

records = Record.objects.filter(pk=...)
record = objects.first()
num_deleted, _ = records.delete()
if num_deleted == 1:
    Audit.objects.create_from_record(record)

I had hoped this would work because, according to the Django docs, "The delete() is applied instantly.".

However, I still seem to be able to trigger the race condition.

I'm considering handling this on the frontend by queuing requests and waiting for the server to return a response before firing the next one, but it'd be nice to have a backend that can actually defend against this sort of thing. I'm also considering using a mutex, but it seems like Django should provide this functionality.

Wondering if anyone has suggestions around how to handle race conditions like this one. This can't be an uncommon problem, can it?

Any feedback is very much appreciated!

Thanks,

Ian

[Adam Parsons]

Hi Ian,

There's a few things you could try out. 

You could employ locking in your database, so that two workers cannot interact with the same row at the same time. 

Step 4, sever B no-op'ing implies you're already using a technique to require unique values in Records, perhaps apply the same technique to the table you're recording changes to.

Otherwise, this is almost definitely a frontend concern (it's a UI/UX issue, after all) and you may benefit from setting your event handler to use a settimeout and no-op future clicks until the server returns. If using a frontend framework with state management, it might be worth setting a lock in state.

Hope that helps,

Adam