Idempotency vs optimistic locking

[Peter Monks]

G'day everyone,

We're in the process of preparing some general guidelines for RESTful APIs, and are trying to make operations idempotent where possible (including updates i.e. PUT operations).

One apparent wrinkle we've run into is with respect to optimistic locking. If we use the classical optimistic locking pattern of:
1. storing an "update token" with each entity, and returning it along with the entity when that entity is requested
2. validating that the update token included in an update request matches the entity's current update token, before allowing the update to occur
3. mutating the update token if the update goes ahead
we find that PUTs are no longer idempotent.

Has anyone else run into this? Is it (as we believe to be the case) simply that optimistic locking is fundamentally at odds with idempotent updates?

Thanks in advance for any thoughts you have to share!

Peter

[Mike Kelly]

Hi Peter,

Is there a reason you are not using ETags and If-Match (i.e. Conditional PUT) ?

Regardless, the requests are idempotent - the important factor is sender intent. As far as the client is concerned, the request can be repeated an arbitrary amount of times, and the fact that the outcome of the requests can change over time is not of consequence to that. The important point for the client is that they can safely repeat the PUT.

This is same issue as 'non-safety' of GET requests that increment a hit-counter.

Cheers,
Mike

[Peter Monks]

Thanks for the quick response Mike.

We are considering ETags as well, but hadn't considered them as a way to detect concurrent modifications - will have to think about that a bit.  One thought we had with eTags was that in practice they seem to be mostly a browser-centric construct, and our clients won't be browsers (well, except for developers poking their way through the API to familiarise themselves with it).  What sort of adoption of eTag support on the client side should we expect, given that those clients will (mostly) not be browsers?

Cheers,
Peter

 

 

[Mike Kelly]

ETags and If-Match are standard HTTP mechanisms, so most http clients (+ caches, etc.) should sit well with their usage. Fwiw, all of the HTTP client libraries I have used recently allow me access to response headers (where the ETag can be retrieved) and to set headers in subsequent requests (to set the If-Match header), so it has been trivial to implement this conditional PUT behaviour with them.

Cheers,

Mike 

[Peter Rexer]

Have any of you considered using a Nonce plus the version as your etag?  Then you can possibly kill 2 birds with one stone.  Each client that has read an object gets a version number and a nonce, and pass them together as an ETag.   

Not sure if it's been thought about before. 

-Peter

[Nate McCall]

If your operations are truly idempotent, why do you need to lock in the first place? 

Behind the scenes, you need only some storage mechanism which can deal with this in a deterministic manner (i.e. highest-timestamp-wins).

I would not say idempotence and locking are at odds, but that they are not applicable to the same use cases.

[Daniel Roop]

Peter,

Mike is write ETags are your answer for this, there really shouldn't be a need to do something custom.  ETags are used for two things

1) Cache Validation

2) Optimistic Locking

http://en.wikipedia.org/wiki/HTTP_ETag

As far as it being a "browser" thing, I think you are going to find a lot of things right now are "browser" things.  But that doesn't necessarily mean that it is not appropriate for a REST API.  In fact I tend to use what the browser does as a litmus test for what I should be doing.  If a similar concept does not yet exist in a non-browser concept I tend to push for it's creation.  A good example is local http caches.  Browsers have them, but many people still to mange client caches on their own, whereas I believe you should have client caches follow the HTTP rules provided by the origin server.  While a year or so ago there were many libraries helping with that, you can start to see them showing up (http://hc.apache.org/httpcomponents-client-ga/index.html).

To your original question, I also am with Mike on this idempotent means the client can repeat the request as often as it wants expecting the result.  Optimistic locking is a server control to inform the client when they are working on an old version of a resource.  In your example the call is idempotent even if optimistic locking is on, because the client can repeat over and over the call if they choose with the same result.   The appropriate action would be for them to get latest, and re-apply their changes, but they can keep trying over and over without any unintended consequences.