DeadlineExceededException and Datastore throttling

[Francois Masurel]

Was just wondering : is there some kind of Datastore throttling in place ?

I can easily reproduce these DeadlineExceededException errors just by loading a few pages of my website after a memcache reset.

All these pages normally load in less than 500ms (200-300ms on average) but under stress some Datastore API calls seem to be just blocked and generate DeadlineExceededException errors (which cause annoying instance restarts BTW).

Thanx for your help.

Francois

[mooose]

Same problem here.

[mooose]

Btw: I am using Python and not Java as Francois. Looks like the datastore brings back all these DeallineExceededExceptions.

Switching to HR seems not easily done at the moment or are there any bulk helpers for the transfer?

Siegfried

[Francois MASUREL]

I have the problem on both MS and HR applications.  It doesn't seem to make a difference.

Francois

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Robert Kluin]

Hi Francois,
You need to use Appstats. It sounds like you're probably doing a
lot of datastore operations, perhaps fetching a very large number of
entities or lots of operations in serial.

http://code.google.com/appengine/docs/java/tools/appstats.html

Robert

[Robert Kluin]

Hi Siegfried,
Have you tried to use the datastore admin? You can use it to copy
entities between apps.

http://code.google.com/appengine/docs/adminconsole/datastoreadmin.html

Robert

[Francois MASUREL]

Hi Robert,

Thanx for your answer.

Do you confirm that there is some kind of throttling for datastore usage ?  I am really interested in knowing if such limitations exist.

As all my HTML page elements are dynamically loaded from datastore when not cached (images, CSS, etc.), there are for sure quite a few datastore requests being done when memcache is cleared as all these elements need to be reloaded on the client side almost simultaneously.

What is the best solution to circumvent such problems ?

Thanx again for your help.

Francois

[Robert Kluin]

Hey Francois,
I'm not sure about datastore throttling. I know a few months ago I
was hitting a short-term throttle, but it was at high write rates,
something on the order of many thousands of API calls per minute as I
recall. I've heard that may have been lifted now, but I can't
confirm.

It really sounds like you need to look into batching your datastore
operations to reduce the number of RPCs. Using Appstats to examine
the calls being made should help you.

Robert

[Francois MASUREL]

I have used AppStats to optimize my datastore API calls : I'm using batch gets as much as possible and async puts to update my datastore cache.

When memcache is empty, I think I do about 10 datastore API calls to build a standard page and 2 async puts to update the cache.

What is strange is that is will run perfectly fine most of the time with decent times (~300ms) and suddenly for unknow reasons some requests will hang for 30s and generate DeadlineExceededExceptions.

I'll try to give you some more precise numbers as soon as possible.

Thanx again for your help.

Francois

[Francois Masurel]

Hi Robert,

Here are two appstats screenshots showing numbers for a rather standard request and a more complex one (cf. attachments).

Francois