Datastore down

[Daniel Florey]

Our app is not working since 2 days due to datastore outage. This is very bad as this is causing manual support/interaction for hundreds of clients.

I'm getting tons of errors like this:

Operation failed due to internal error, aborting
com.google.appengine.api.datastore.DatastoreTimeoutException: 
	at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:43)
	at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:70)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:94)
	at com.google.appengine.api.datastore.FutureHelper$CumulativeAggregateFuture.get(FutureHelper.java:142)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:86)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:86)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:86)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:86)
	at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:71)
	at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:32)
	at com.google.appengine.api.datastore.DatastoreServiceImpl$2.runInternal(DatastoreServiceImpl.java:94)
	at com.google.appengine.api.datastore.DatastoreServiceImpl$2.runInternal(DatastoreServiceImpl.java:91)
	at com.google.appengine.api.datastore.TransactionRunner.runInTransaction(TransactionRunner.java:29)
	at com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:91)
	at com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:80)

[Bert]

Is this happening all the time or just for some requests?

Perhaps include the code that results in the exception being thrown.

[Vinny P]

On Sat, Jul 6, 2013 at 2:54 PM, Bert <robert...@gmail.com> wrote:

Is this happening all the time or just for some requests?

Perhaps include the code that results in the exception being thrown.

 

+1. DatastoreTimeoutExceptions generally occur when applications try to put/delete large amounts of entities simultaneously. Did you change your code, or get a large amount of new users recently?

 

 Perhaps your app can reduce its usage of the datastore. For example, you could cache more into memcache, or consolidate entities.

 

Can you post your code?

 

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com

 

[Daniel Florey]

Hi,

thanks for your reply. 

I cannot believe that datastore is not able to handle many write requests simultaneously. Where did you get this information?  Of course I'm not writing to the same entity groups, just plain entities. 

Any insight would be highly appreciated!

My code is quite simpe, just writing to the datastore from concurrent tasks using low-level api.

2013/7/7 Vinny P <vinn...@gmail.com>

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/Pgs-uSUMnv4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Vinny P]

On Sun, Jul 7, 2013 at 2:57 AM, Daniel Florey <daniel...@gmail.com> wrote:

I cannot believe that datastore is not able to handle many write requests simultaneously. Where did you get this information?  Of course I'm not writing to the same entity groups, just plain entities. 

 

The same exception sometimes happens when people use the bulkloader - there's so many writes happening that the datastore starts throwing errors. You can search this list for examples.

 

Sometimes this error pops up when accessing large numbers of entities (in the millions), see here for an example: https://groups.google.com/forum/#!topic/objectify-appengine/zIFIylfqRAw 

[Jeff Schnitzer]

How many entities are you trying to write at once? There are rate limits to what you can do from a single instance. A batch write will be broken up into chunks internally, plus you can only have a certain number (10 I think) of async requests in flight at once.

In practice, writing large numbers of entities (1000+) takes substantial amounts of time, and may be best shunted off to the task queue where it will be spread out over a larger number of instances. If you have a borderline case, there are a few tricks - you might try running an Executor with multiple threads to get more parallelism.

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

[Daniel Florey]

Hi Jeff,

thanks for your valuable reply. As a workaround I've "slowed down" my app. This seems to fix the issue.

It would be great to get more detailed info on this topic in the docs.

Thanks,

Daniel

[Barry Hunter]

At a total guess, its the same underlying reason why this was implemented:

https://groups.google.com/forum/#!topic/google-appengine/Yt9W204jips