Several ApiProxy$CancelledExceptions

[aramanuj]

1. Hi,

   We are seeing several "com.google.apphosting.api.ApiProxy$CancelledException: The API call datastore_v3.Put() was explicitly cancelled." exceptions on any page trying to access the datastore (at around 10:00am PDT). This is the first time we've observed this on our application (samaanguru). It is using the master-slave datastore. Clearly, the datastore writes are not happening. Can anyone throw light on what's happening?

   The stack-trace is as below.

   Thanks,
   Arun

   
   

   com.google.apphosting.api.ApiProxy$CancelledException: The API call datastore_v3.Put() was explicitly cancelled.
   	at com.google.apphosting.runtime.ApiProxyImpl$5.get(ApiProxyImpl.java:361)
   	at com.google.apphosting.runtime.ApiProxyImpl$5.get(ApiProxyImpl.java:359)
   	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
   	at com.google.appengine.api.datastore.FutureHelper$CumulativeAggregateFuture.get(FutureHelper.java:145)
   	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
   	at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:72)
   	at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:33)
   	at com.google.appengine.api.datastore.DatastoreServiceImpl$2.runInternal(DatastoreServiceImpl.java:113)
   	at com.google.appengine.api.datastore.DatastoreServiceImpl$2.runInternal(DatastoreServiceImpl.java:110)
   	at com.google.appengine.api.datastore.TransactionRunner.runInTransaction(TransactionRunner.java:31)
   	at com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:110)
   	at com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:94)
   	at com.google.apphosting.runtime.jetty.DatastoreSessionStore.saveSession(DatastoreSessionStore.java:93)
   	at com.google.apphosting.runtime.jetty.SessionManager$AppEngineSession.save(SessionManager.java:164)
   	at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:41)
   	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
   	at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
   	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
   	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
   	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
   	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
   	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
   	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
   	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:249)
   	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
   	at org.mortbay.jetty.Server.handle(Server.java:326)
   	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
   	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
   	at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
   	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
   	at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:135)
   	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:477)
   	at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:449)
   	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455)
   	at com.google.tracing.TraceContext.runInContext(TraceContext.java:695)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325)
   	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453)
   	at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
   	at java.lang.Thread.run(Thread.java:679)
   

2. W

[c h]

i was seeing a crap-ton of deadline exceeded exceptions on starmakerapp from about 9:30-10:00am PDT today.  the exceptions were on datastore put calls, and caused instance thrashing (my average instance life was under 1 minute) and made my app unresponsive returning 500 errors to my users.

yup, i'm on master slave datastore.  i'm still waiting for a response to http://code.google.com/p/googleappengine/issues/detail?id=7434 before i migrate.

[aramanuj]

Thanks for the tip on the Blobstore issue with the migration. Nothing
much has changed on our side, but we've started seeing more of these
data store put exceptions - its frequency has slowly but surely
increased (may of the requests that face this problem also have low
latencies - <1 sec.). So, we are considering HR datastore migration.
Thanks for alerting us to the Blobstore situation you have.

Regards,
Arun

On Jun 5, 10:57 pm, c h <how...@umich.edu> wrote:
> i was seeing a crap-ton of deadline exceeded exceptions on starmakerapp
> from about 9:30-10:00am PDT today.  the exceptions were on datastore put
> calls, and caused instance thrashing (my average instance life was under 1
> minute) and made my app unresponsive returning 500 errors to my users.
>
> yup, i'm on master slave datastore.  i'm still waiting for a response

> tohttp://code.google.com/p/googleappengine/issues/detail?id=7434before i

> migrate.
>
>
>
>
>
>
>
> On Tuesday, June 5, 2012 10:03:15 AM UTC-7, aramanuj wrote:
>

> >    1.

> >    2. W