HardDeadlineExceededError (problems today)

[Ian]

Hello to all!

We have been running some tests on the Java GAE platform prior to
making a commitment to deploying some upcoming projects with have on
GAE. Everything seemed to be going ok until today!

To cut straight to the point we stared to encounter (some traces
below):

com.google.apphosting.runtime.HardDeadlineExceededError exceptions

Yesterday the code (that is unchanged) worked fine using the same load
pattern (a few mobile devices that do periodic uploads).

Now I appreciate that these errors are indicating a 30 second timeout
exception has occurred but I'm not sure why. I've seen these errors
occurring while our app tries to delete an object from an ArrayList
(that contains less than 100 items) as well as when retrieving data
from JDO.

This has only happened today so I'm guessing (and hoping) this is a
GAE issue. Has anyone else encountered similar problems over the last
24 hours?

Thanks,
Ian

Example trace:

com.google.apphosting.runtime.HardDeadlineExceededError: This request
(6c7858aa8c29829d) started at 2010/07/20 20:11:04.342 UTC and was
still executing at 2010/07/20 20:11:33.425 UTC.
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:443)
at java.util.concurrent.TimeUnit.timedWait(Unknown Source)
at com.google.apphosting.runtime.AsyncFuture.get(AsyncFuture.java:60)
at com.google.apphosting.runtime.ApiProxyImpl
$AsyncApiFuture.get(ApiProxyImpl.java:332)
at com.google.apphosting.runtime.ApiProxyImpl
$AsyncApiFuture.get(ApiProxyImpl.java:223)
at
com.google.apphosting.runtime.ApiProxyImpl.doSyncCall(ApiProxyImpl.java:
135)
at com.google.apphosting.runtime.ApiProxyImpl.access
$000(ApiProxyImpl.java:43)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:
108)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:
106)
at java.security.AccessController.doPrivileged(Native Method)
at
com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:
106)
at
com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:
43)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:98)
at
com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java:
58)
at com.google.appengine.api.datastore.DatastoreServiceImpl
$2.run(DatastoreServiceImpl.java:188)
at
com.google.appengine.api.datastore.TransactionRunner.runInTransaction(TransactionRunner.java:
30)
at
com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:
176)
at
com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:
156)
at
com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:
148)
at
org.datanucleus.store.appengine.RuntimeExceptionWrappingDatastoreService.put(RuntimeExceptionWrappingDatastoreService.java:
93)
at
org.datanucleus.store.appengine.DatastorePersistenceHandler.put(DatastorePersistenceHandler.java:
180)
at
org.datanucleus.store.appengine.DatastorePersistenceHandler.put(DatastorePersistenceHandler.java:
127)
at
org.datanucleus.store.appengine.DatastorePersistenceHandler.insertObjects(DatastorePersistenceHandler.java:
254)
at
org.datanucleus.store.appengine.DatastorePersistenceHandler.insertObject(DatastorePersistenceHandler.java:
240)
at
org.datanucleus.state.JDOStateManagerImpl.internalMakePersistent(JDOStateManagerImpl.java:
3185)
at
org.datanucleus.state.JDOStateManagerImpl.makePersistent(JDOStateManagerImpl.java:
3161)
at
org.datanucleus.ObjectManagerImpl.persistObjectInternal(ObjectManagerImpl.java:
1298)
at
org.datanucleus.sco.SCOUtils.validateObjectForWriting(SCOUtils.java:
1476)
at
org.datanucleus.store.mapped.scostore.ElementContainerStore.validateElementForWriting(ElementContainerStore.java:
380)
at
org.datanucleus.store.mapped.scostore.FKListStore.validateElementForWriting(FKListStore.java:
609)
at
org.datanucleus.store.mapped.scostore.FKListStore.internalAdd(FKListStore.java:
344)
at
org.datanucleus.store.appengine.DatastoreFKListStore.internalAdd(DatastoreFKListStore.java:
146)
at
org.datanucleus.store.mapped.scostore.AbstractListStore.add(AbstractListStore.java:
105)
at org.datanucleus.sco.backed.ArrayList.add(ArrayList.java:698)

Alternate trace:

com.google.apphosting.runtime.HardDeadlineExceededError: This request
(a8a462671465bfdf) started at 2010/07/20 15:00:31.927 UTC and was
still executing at 2010/07/20 15:01:01.031 UTC.
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:443)
at java.util.concurrent.TimeUnit.timedWait(Unknown Source)
at com.google.apphosting.runtime.AsyncFuture.get(AsyncFuture.java:60)
at com.google.apphosting.runtime.ApiProxyImpl
$AsyncApiFuture.get(ApiProxyImpl.java:332)
at com.google.apphosting.runtime.ApiProxyImpl
$AsyncApiFuture.get(ApiProxyImpl.java:223)
at
com.google.apphosting.runtime.ApiProxyImpl.doSyncCall(ApiProxyImpl.java:
135)
at com.google.apphosting.runtime.ApiProxyImpl.access
$000(ApiProxyImpl.java:43)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:
108)
at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:
106)
at java.security.AccessController.doPrivileged(Native Method)
at
com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:
106)
at
com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:
43)
at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:98)
at
com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java:
58)
at
com.google.appengine.api.datastore.PreparedQueryImpl.runQuery(PreparedQueryImpl.java:
127)
at
com.google.appengine.api.datastore.PreparedQueryImpl.asIterator(PreparedQueryImpl.java:
87)
at com.google.appengine.api.datastore.BasePreparedQuery
$1.iterator(BasePreparedQuery.java:26)
at
org.datanucleus.store.appengine.DatastoreFKListStoreSpecialization.removeAt(DatastoreFKListStoreSpecialization.java:
166)
at
org.datanucleus.store.mapped.scostore.FKListStore.removeAt(FKListStore.java:
525)
at
org.datanucleus.store.mapped.scostore.FKListStore.internalRemove(FKListStore.java:
423)
at
org.datanucleus.store.mapped.scostore.AbstractListStore.remove(AbstractListStore.java:
283)
at org.datanucleus.sco.backed.ArrayList.remove(ArrayList.java:871)
at org.datanucleus.sco.backed.ArrayList.remove(ArrayList.java:835)
at
org.datanucleus.store.mapped.mapping.PersistenceCapableMapping.preDelete(PersistenceCapableMapping.java:
1308)
at
org.datanucleus.store.appengine.DependentDeleteRequest.execute(DependentDeleteRequest.java:
71)
at
org.datanucleus.store.appengine.DatastorePersistenceHandler.deleteObject(DatastorePersistenceHandler.java:
599)
at
org.datanucleus.state.JDOStateManagerImpl.internalDeletePersistent(JDOStateManagerImpl.java:
4198)
at
org.datanucleus.state.JDOStateManagerImpl.deletePersistent(JDOStateManagerImpl.java:
4166)
at
org.datanucleus.ObjectManagerImpl.deleteObjectInternal(ObjectManagerImpl.java:
1470)
at
org.datanucleus.ObjectManagerImpl.deleteObject(ObjectManagerImpl.java:
1395)
at
org.datanucleus.jdo.JDOPersistenceManager.jdoDeletePersistent(JDOPersistenceManager.java:
754)
at
org.datanucleus.jdo.JDOPersistenceManager.deletePersistent(JDOPersistenceManager.java:
772)