Entity version checks / optimistic locking

[Jens]

Hi,

I currently have some problems with entity version checks / optimistic locking using JPA on server side and I wonder how RequestFactory would handle my problem.

Currently the application I am working on has an auto save mechanism. That means each time the user changed some data the application will wait a couple of milliseconds and if no additional changes are made during this time the application sends a save command (command / dispatch pattern) to the server. The server then loads the JPA entity from the database and compares its entity version with the one provided by the client entity that was send with the save command. If they match, data will be copied from client entity to the JPA entity and then saved to our database. When the entity is saved we return the new entity version to the client application which updates it in the client entity object. If they do not match, an exception will be thrown and the client application reloads the object and updates the UI because the database contains newer data. Our server is pretty stateless and we have one transaction per request.

So its similar to RequestFactory. We have a client entity class and a JPA entity and copy data between them on server side (along with some computations if needed).

Now our problem: As we do our auto save quite instant (about 100-300 ms after the last user input) it may happen that we fire two or more save commands without getting a response from our server because the network connection is bad (basically the round trip time would be longer as our auto save timer / 100-300ms). Thus the second save command will fail because the first one results in an updated entity version in our database but the client does not now about it yet and thus the second save command contains the same entity version as the first one.

How would RequestFactory handle such a situation? What happens if you call xxxRequest.persist().using(object).fire() twice without having received a result during both calls (what could happen if we integrate RF instead of a command pattern)? I know that RequestFactory currently does no real optimistic lock check (http://code.google.com/p/google-web-toolkit/issues/detail?id=6046) because it somehow disappeared in the source code. But how would it handle entity versions to make sure optimistic locking works? Are entity versions stored in the generated proxy classes or is there a server side map that remembers all entity proxys with their entity versions that have been delivered to the client? 

How do you do optimistic checking in your application and how do you have solved this issue? Do you just make sure the client application can only fire a second save request when the first one is done successfully? Currently we have increased our auto save timer by temporarily adding a factor of 10. So we auto save about 3 seconds after the last user input (although this causes some other problems like doing a forced auto save when the user navigates to a different application place within this 3 seconds) to somehow compensate a poor network quality.

We would really like to use RequestFactory in the future to get rid of the command pattern and all that server side copy code but in the meantime we can maybe adopt some RequestFactory ideas and integrate them in our command pattern to solve our issue.

I really appreciate any helpful thoughts!

[Thomas Broyer]

On Friday, March 11, 2011 11:43:48 AM UTC+1, Jens wrote:

Hi,

I currently have some problems with entity version checks / optimistic locking using JPA on server side and I wonder how RequestFactory would handle my problem.

Currently the application I am working on has an auto save mechanism. That means each time the user changed some data the application will wait a couple of milliseconds and if no additional changes are made during this time the application sends a save command (command / dispatch pattern) to the server. The server then loads the JPA entity from the database and compares its entity version with the one provided by the client entity that was send with the save command. If they match, data will be copied from client entity to the JPA entity and then saved to our database. When the entity is saved we return the new entity version to the client application which updates it in the client entity object. If they do not match, an exception will be thrown and the client application reloads the object and updates the UI because the database contains newer data. Our server is pretty stateless and we have one transaction per request.

So its similar to RequestFactory. We have a client entity class and a JPA entity and copy data between them on server side (along with some computations if needed).

Now our problem: As we do our auto save quite instant (about 100-300 ms after the last user input) it may happen that we fire two or more save commands without getting a response from our server because the network connection is bad (basically the round trip time would be longer as our auto save timer / 100-300ms). Thus the second save command will fail because the first one results in an updated entity version in our database but the client does not now about it yet and thus the second save command contains the same entity version as the first one.

Maybe you should delay autosave requests to until the previous one has returned?

 

How would RequestFactory handle such a situation? What happens if you call xxxRequest.persist().using(object).fire() twice without having received a result during both calls (what could happen if we integrate RF instead of a command pattern)?

Well, first, you can only edit() a proxy once at a time (i.e. in a single RequestContext). Second, when you fire() a RequestContext, the edit()ed proxies are locked until the response comes back.

This basically means that if you have to lock the form while you autosave it.

I'd suggest you file an enhancement request so you could, e.g. flush the editor to a different object/proxy than was passed to initialize it. That way, you could display() the proxy (so it's not edit()ed), and then flush into an edit()ed version. You'd still have to block until the response come before doing another autosave; and worse, before doing a save(), which means the user would have to wait for the autosave to finish before it can explicitly save the form and go away!

In GWT 2.3, editors will be "visitable" (using a visitor pattern), so it might be possible to implement the above this way, but I'm not even sure…

I know that RequestFactory currently does no real optimistic lock check (http://code.google.com/p/google-web-toolkit/issues/detail?id=6046) because it somehow disappeared in the source code. But how would it handle entity versions to make sure optimistic locking works? Are entity versions stored in the generated proxy classes or is there a server side map that remembers all entity proxys with their entity versions that have been delivered to the client?

Entity versions are sent to/from the client along with the ID. It's a serialized and encoded version though, because a version can be any kind of object (RequestFactory does a toString() on it, and then base64-encode it).

RequestFactory would have to either: find() with the ID and version (in String form) and let you "parse" the version back into you object type (but that would lock RF into always transforming the version to a String), or find() the entity as today and then ask for its version, serialize it and compare with the version sent by the client (same comparison as is done before returning objects to the client, to "generate" the appropriate EntityProxyChange events on the client).

 

How do you do optimistic checking in your application and how do you have solved this issue?

We don't actually; but we're in a special case that users first have to explicitly lock an object (actually, create a private working copy) before they can edit it; so a given user can only be in conflict with "himself"; and using the application in multiple tabs/windows is an officially unsupported use case (yes, we're lucky ;-) ).

[Jens]

On Friday, March 11, 2011 3:50:35 PM UTC+1, Thomas Broyer wrote:

Maybe you should delay autosave requests to until the previous one has returned?

Well.. maybe. Just keep in mind that we do not have a dedicated save button (thats why we do auto save that often) so we are forced to execute every autosave request as soon as possible in order to avoid data loss. But maybe we try it to see how many auto save requests will be queued when simulating a bad network connection. The more items we have in that queue the more data will be lost if the browser gets closed or someone else changes the same data and an optimistic lock exception forces the application to reload the data and clear the auto save queue.

 

How would RequestFactory handle such a situation? What happens if you call xxxRequest.persist().using(object).fire() twice without having received a result during both calls (what could happen if we integrate RF instead of a command pattern)?

Well, first, you can only edit() a proxy once at a time (i.e. in a single RequestContext). Second, when you fire() a RequestContext, the edit()ed proxies are locked until the response comes back.

This basically means that if you have to lock the form while you autosave it.

I'd suggest you file an enhancement request so you could, e.g. flush the editor to a different object/proxy than was passed to initialize it. That way, you could display() the proxy (so it's not edit()ed), and then flush into an edit()ed version. You'd still have to block until the response come before doing another autosave; and worse, before doing a save(), which means the user would have to wait for the autosave to finish before it can explicitly save the form and go away!

 

Hmm why do I have to lock the form while auto save the data? Is it because I have to call edit() again after receiving the RF response in order to be able to call flush().fire() again? And calling edit() would overwrite any changes in the form the user could have made if I do not lock the form in the mean time? If so, wouldn't it be a better enhancement to somehow allow multiple flush().fire() calls (of course not parallel calls) after a single edit() call has been made to initially populate the editors? I've read that the editor framework uses flow synchronization. So on each flush every editor value is just copied into the proxy instance and then the RequestContext is fired. Shouldn't be that difficult for RF/RFED to recognize that the response has been received and then implicitly set the locked flag of the proxy back to false. If we then call flush().fire() again the new editor values will be copied to the proxy and saved again...and so on.

Maybe its possible to expose this enhancement as a method. Something like RFED.startAutoSave(int intervallInMillis, RequestContext ctx, RFEDCallback callback) where RFEDCallback contains onSuccess/onFailure/onValidationError (along with RFED.stopAutoSave()). intervallInMillis would be an approximate value because of an autosave request that takes longer to complete than intervallInMillis. Internally RFED would call edit() once and then start a timer that does flush().fire() and reschedules itself once fire() completes. Would be a nice RFED feature.

 

Entity versions are sent to/from the client along with the ID. It's a serialized and encoded version though, because a version can be any kind of object (RequestFactory does a toString() on it, and then base64-encode it).

RequestFactory would have to either: find() with the ID and version (in String form) and let you "parse" the version back into you object type (but that would lock RF into always transforming the version to a String), or find() the entity as today and then ask for its version, serialize it and compare with the version sent by the client (same comparison as is done before returning objects to the client, to "generate" the appropriate EntityProxyChange events on the client).

Ah ok, so each proxy instance holds its serialized version in a field somewhere in the generated autobean and just don't expose it so the developer do not have to bother with it? If so then its basically the same what we do and I guess thats the reason why RF locks the proxy until the response is received. That way RF avoids the issue we currently have by design.

 

How do you do optimistic checking in your application and how do you have solved this issue?

We don't actually; but we're in a special case that users first have to explicitly lock an object (actually, create a private working copy) before they can edit it; so a given user can only be in conflict with "himself"; and using the application in multiple tabs/windows is an officially unsupported use case (yes, we're lucky ;-) ).

Yeah really lucky. Having a separate edit/cancel/save button where edit also means "lock that object until save or cancel is called" is always a nice luxury :) But we also thought about locking the object. We could implicitly do it on the first auto save and then unlock it a couple of minutes later if no further auto save request occurs in that time. When its locked everyone else would see a disabled form and the app would check the lock with a timer so it can enable the form once the object is unlocked again.

[Thomas Broyer]

On Friday, March 11, 2011 10:37:10 PM UTC+1, Jens wrote:

On Friday, March 11, 2011 3:50:35 PM UTC+1, Thomas Broyer wrote:

Well, first, you can only edit() a proxy once at a time (i.e. in a single RequestContext). Second, when you fire() a RequestContext, the edit()ed proxies are locked until the response comes back.

This basically means that if you have to lock the form while you autosave it.

I'd suggest you file an enhancement request so you could, e.g. flush the editor to a different object/proxy than was passed to initialize it. That way, you could display() the proxy (so it's not edit()ed), and then flush into an edit()ed version. You'd still have to block until the response come before doing another autosave; and worse, before doing a save(), which means the user would have to wait for the autosave to finish before it can explicitly save the form and go away!

 

Hmm why do I have to lock the form while auto save the data? Is it because I have to call edit() again after receiving the RF response in order to be able to call flush().fire() again? And calling edit() would overwrite any changes in the form the user could have made if I do not lock the form in the mean time?

The question is "what would you do of other changes made by the user in the mean time?"

Because the proxies are locked, you cannot flush() the driver. And because you cannot edit() a given proxy twice at the same time, you cannot do a "flush().fire() + edit()".

But thinking a bit more about it, there might be a solution: instead of edit()ing the proxy, you'd create a copy of it that you give to the editor driver. Then you periodically flush() the driver, but never fire() its context. After the flush(), you copy  the proxy values into the proxy you'll send to the server.

The problem is: how to clone the proxy? See http://code.google.com/p/google-web-toolkit/issues/detail?id=5794

Ah ok, so each proxy instance holds its serialized version in a field somewhere in the generated autobean and just don't expose it so the developer do not have to bother with it? If so then its basically the same what we do and I guess thats the reason why RF locks the proxy until the response is received. That way RF avoids the issue we currently have by design.

The reason RF locks the proxy until the response is received is that it will unlock them in case of constraint violations (so you could correct your proxies and fire the same RequestContext again).

We don't actually; but we're in a special case that users first have to explicitly lock an object (actually, create a private working copy) before they can edit it; so a given user can only be in conflict with "himself"; and using the application in multiple tabs/windows is an officially unsupported use case (yes, we're lucky ;-) ).

Yeah really lucky. Having a separate edit/cancel/save button where edit also means "lock that object until save or cancel is called" is always a nice luxury :)

It's even better than that ;-)

The user has to click a "lock" button to be able to edit the objects (creates a working copy). The form then switches to read/write mode and the "save" button appears. But the save button only saves changes to the working copy; there's a distinct "unlock" button that overwrites the "public copy" with the "working copy" (and another "free" button to "unlock" the "public copy" and simply discards the "working copy").

A user can keep his working copy for days, or even weeks or months.

[Jens]

On Saturday, March 12, 2011 9:58:11 AM UTC+1, Thomas Broyer wrote:

The question is "what would you do of other changes made by the user in the mean time?"

Because the proxies are locked, you cannot flush() the driver. And because you cannot edit() a given proxy twice at the same time, you cannot do a "flush().fire() + edit()".

But thinking a bit more about it, there might be a solution: instead of edit()ing the proxy, you'd create a copy of it that you give to the editor driver. Then you periodically flush() the driver, but never fire() its context. After the flush(), you copy  the proxy values into the proxy you'll send to the server.

The problem is: how to clone the proxy? See http://code.google.com/p/google-web-toolkit/issues/detail?id=5794

Ah ok, so each proxy instance holds its serialized version in a field somewhere in the generated autobean and just don't expose it so the developer do not have to bother with it? If so then its basically the same what we do and I guess thats the reason why RF locks the proxy until the response is received. That way RF avoids the issue we currently have by design.

The reason RF locks the proxy until the response is received is that it will unlock them in case of constraint violations (so you could correct your proxies and fire the same RequestContext again).

Ok a cloned proxy would be a solution. But I still need a queue or something in case the real non-cloned proxy isn't saved yet but the next auto save occures. So in general RF forces me to wait for the response. But its ok.. as I have to wait anyways because of the entity version (either by waiting for the response or by increasing auto save intervals).

And you are right, in case of constraint violations one should lock the form and the entity during save. But in our aggressive auto save strategy we would like to not lock the form and display any constraint violations in a non intrusive way. In most cases the user will correct these violations anyways because most violations will occur because of "unlucky" auto saves (e.g. data not completely and thus maybe incorrect edited be the user).

 

It's even better than that ;-)

The user has to click a "lock" button to be able to edit the objects (creates a working copy). The form then switches to read/write mode and the "save" button appears. But the save button only saves changes to the working copy; there's a distinct "unlock" button that overwrites the "public copy" with the "working copy" (and another "free" button to "unlock" the "public copy" and simply discards the "working copy").

A user can keep his working copy for days, or even weeks or months.

Uhh..so if I work on my private copy for 2 weeks and then unlock it to update the public copy, a second user can destroy my work by unlocking his copy a day later and thus overwriting my 2 weeks of work? :) Having working copies is a nice idea but merging them back into one public copy seems tricky. But of course it depends on the application.

[Thomas Broyer]

There's no need to merge, because when you do a working copy, you're locking the public version. The idea of a working copy is that you work in private, and the public version stays the same until you "publish" your work.

[Aldo Neto]

I apologize for such stupid question, but how can I achieve that? I mean, how should I treat the entity on the server side to get that lock? I can have the "Edit/Lock" button and that button should work for only one user at a time, until that user has committed their work. In this case, how can I lock the object? And how do I unlock it?

Again, sorry for the newbie question

Thanks

--
You received this message because you are subscribed to the Google Groups "Google Web Toolkit" group.
To post to this group, send email to google-we...@googlegroups.com.
To unsubscribe from this group, send email to google-web-tool...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-web-toolkit?hl=en.

[J.Ganesan]

1. In your server, maintain a map of objectId/semaphore. A semaphore
enables locking to be done by a thread and unlocking by another
thread, both atomically.
2. "Edit" means acquiring lock. Send a message from client to server
for acquiring semaphore. If successful, allow editing.
3. Release semaphore on saving.

J.Ganesan
www.DataStoreGwt.com

[Jens]

You have to store the lock somehow in the database. For example you can create a database table called "locked objects" that has columns like "id, referenced table name, referenced object id, user id that locked the object, date of lock". Then you put a unique constraint on "referenced table name, referenced object id" so that a object can not be locked twice. 

On server side you can then implement methods like Lock.create(CommonEntityInterface object), Lock.delete(CommonEntityInterface object) and maybe Lock.isLocked(CommonEntityInterface object) where CommonEntityInterface is just an interface (or an abstract entity base class) implemented by all your entities so you can call getId() on them. The table name can be accessed via the @Table annotation (reflection) or has to be generated if you use JPA default table names.

-- J.

[Jeffrey Chimene]

> --
Why not use a set transaction statement that locks the desired row?
Isn't that why we use a DBMS?

[J.Ganesan]

No. Aldo Neto wants to preclude "Edit" function being made available
to more than one at any point of time. Whereas, a transaction is a
postmortem action, still allowing the "Edit" function to more than one
( though only one will eventually succeed ).

J.Ganesan
www.DataStoreGwt.com

[Jeff Chimene]

On 07/06/2011 09:09 AM, J.Ganesan wrote:
> No. Aldo Neto wants to preclude "Edit" function being made available
> to more than one at any point of time. Whereas, a transaction is a
> postmortem action, still allowing the "Edit" function to more than one
> ( though only one will eventually succeed ).
>
> J.Ganesan
> www.DataStoreGwt.com

Sorry, but I must disagree with the assertion that a database
transaction unconditionally provides "... the "Edit" function to more
than one..." A transaction is not a "postmortem action"; although I have
to admit that I'm unclear on the definition of that term.

An exclusive write lock on a row helps implement the behavior desired by
the OP. Given two cooperating processes, A and B, if A holds an
exclusive write lock on one or more rows, then the DBMS will refuse any
request by B for a proper subset of those rows in an incompatible lock
mode. Such refusals provide information to the the client so that it can
then "... preclude "Edit" function being made available to more than one

[Jens]

That would imply that a transaction has to live longer than one request (if you use JPA you can only lock an entity inside a transaction and the entity will be unlocked once you commit or rollback). But I think in most common ajax web applications you do one transaction per request and that way you can not hold a lock for a longer period of time if you use DBMS read/write locks.

[Jeff Chimene]

On 07/06/2011 09:39 AM, Jens wrote:
> That would imply that a transaction has to live longer than one request
> (if you use JPA you can only lock an entity inside a transaction and the
> entity will be unlocked once you commit or rollback).

OK

> But I think in
> most common ajax web applications you do one transaction per request and
> that way you can not hold a lock for a longer period of time if you use
> DBMS read/write locks.

I disagree with that assertion of the linkage between transaction count
on the client and transaction count on the server. However, this is a
GWT forum, not a DBMS forum, and I'll end the discussion here.