GAE local webapp - datastore persistence too slow

[Davy De Waele]

Hi,

I noticed that simple persist operations on a local web app (running from Eclipse) is VERY slow.

For example, if I'm persisting an object in a servlet, it takes a very long time for it to actually persist in the DB.

I'm persisting objects with the following code.

  public static <T> T makePersistent(T object) {

    PersistenceManager manager = getPersistenceManager();

    try {

      return manager.makePersistent(object);

    } finally {

      manager.close();

    }

  } 

When this code is called from a servlet. most of the time by the time the servlet ends and redirects to my page, the object is still not in the database.

Sometimes the object is never persisted in the database.

If I debug my application, and give it more time on the thread by suspending a breakpoint, the object eventually does get persisted. But it takes several seconds.

I've experienced this on multiple machines, and multiple operating systems.

Is this a known issue and is there some kind of workaround for this ?

Regards,

Davy

[Jose Montes de Oca]

Hi Davy,

What you are experiencing is an expected behavior, this are some of the changes we made to the SDK in order to simulate High Replication Datastore behavior on the local development server.

Since SDK 1.5.1: http://code.google.com/p/googleappengine/wiki/SdkForJavaReleaseNotes#Version_1.5.1_-_June_20,_2011 The development server's datastore and testing service implementations now contains logic that closely replicates the consistency guarantees of the High Replication datastore

This will let you test how your application will feel when running on production, get a good understanding of Eventual Consistency and use Datastore characteristics to make Strongly consistent queries (using entity groups, ancestor queries and key queries), A good article to read along would be:

http://code.google.com/appengine/docs/java/datastore/hr/overview.html

Development server documentation:

http://code.google.com/appengine/docs/java/tools/devserver.html#Using_the_Datastore

Any question, don't hesitate to ask! 

Hope this helps! Happy coding on App Engine!

Best,

Jose Montes de Oca

Developer Support Specialist

[Davy De Waele]

I'll definitely have a read of the articles you mentioned, but it doesn't correspond to what I'm seeing on the google appengine itself. 

On the appengine, my page flows work perfectly, however, when running them locally, it's just not workable. Each time I persist an object it fails unless I suspend a breakpoint and give the appserver some time..... And I persist really simply, isolated objects. 

I also pent almost 2 days trying to figure out how to do a simple 1-N relationship using JDO on AppEngine and just gave up on it, although I am used to those technologies on JEE, and do much more complex things there. Just couldn't get it to work on GAE).

If you have any pointers on that, with some worked out samples that would be great. Hate spending so much time on those trivial things, as the issues seem to be very GAE specific (caused by its limitations, or the appengine JDO implementation).

[Simon Knott]

Is your app on GAE on the Master/Slave datastore or the HR datastore?

The problem with the development server, in its default configuration, is that it emulates eventual consistency but it's quite extreme.  Up to this point, I've never hit the delays that are pre-configured into the dev server when in Production.  However, just because it hasn't happened yet doesn't mean that you shouldn't code your application so that it can cope with those delays.

[steuke]

Hi Davy

it can be frustrating, but you really should consider to deal with the
write delay that you see in dev mode. Like Simon says, while your code
might work in production under ideal circumstances 9X% of the time,
you don't want it to fail under less ideal circumstances. That's just
how GAE works.

On the other hand, I think you have a couple of options that can solve
your problem pretty nicely. For example, try a different ORM framework
(I started out using JDO, too, but after very many frustrating
experiences and limitions, I switched like many others before me). I
switched to objectify because I couldn't get JDO's caching to work
with my data model. I believe that objectify would also help with your
problem, because it will use memcache transparently. I.e. when you
write to the datastore your entities should be automatically updated
in memcache and subsequent read operations will simply fetch from
memcache.

Moving away from JDO will save you many a headache...

Best,
Stefan

[John V Denley]

I just ran into this same problem, on my local system "makePersistent" is taking about 10 seconds before i can read the data I just wrote to the server. I have deal with this in the live environment with the following setting in the jdoconfig file:

<property name="datanucleus.appengine.datastoreReadConsistency" value="STRONG" />

However, this seems to have no effect on my local instance, hence up to now i have only been able to test my application in the live environment, which is irritating!

I have found from the links above that in my run configurations, I can set the unapplied job percentage to ZERO to force consistency on the dev server, but as it pointed out her by Simon, you still need to code for the live "eventual consistency" scenario.

HOWEVER, what i really don't understand is why when the user writes something to the database, the system is not clever enough to work out that if the user just wrote the data, then if they query that same data right away, then obviously they would expect to see it there. My application is a little like Google Calendar, in that once you save an appointment, if you refresh the screen you expect it still to be there, but currently with my application (without the "STRONG" setting - mentioned above) the user can refresh the screen several times, before the appointment they just created actually appears on their screen!!!!

The only way I can solve this seems to be with memcache, which up to now I have just not had the time to figure out how to use, but it seems like its now a requirement, as my customers are now noticing this "gap" between saving data and then being able to see it (or edit it!!) again!

[John V Denley]

forgot the link to how to change the local HRD consistency rate:
https://developers.google.com/appengine/docs/java/tools/devserver?csw=1#Using_the_Datastore