Lots of HTTP 500 errors and all instances restarted by GAE!

342 views
Skip to first unread message

Hamish

unread,
Mar 8, 2012, 10:44:45 AM3/8/12
to google-a...@googlegroups.com
Hi,

We use GAE to host one of our Java apps. Recently we have been experiencing quite a lot of HTTP 500 errors and timeouts from App Engine. It appears that App Engine sometimes restarts all of our instances at the same time and does not wait for instances to finish serving requests.  See the attached images:




 In the logs I see a lot of entries like this:

"Request was aborted after waiting too long to attempt to service your request."

The request time shown in the logs was typically around 10,000ms before the request was terminated.

The screenshots were taken at 23.45 GMT on 1 March. We have since experienced similar issues. The latest being yesterday afternoon at 15.30 GMT. The one yesterday was very odd. It appears that the graphs in the dashboard all got reset when the instances restarted as if the entire app had been redeployed with no history at all.

So my questions:

1) Can someone in Google please look into this for me?
2) Why would all instances be restarted at the same time?
3) Why wouldn't App Engine let the instances finish serving requests before restarting?
4) Is there any way I can elavate this issue?
5) Is there a better place to report this issue so someone in Google will investigate?

This is very serious and we need to get to the bottom of it. 

Thanks,
Hamish



Nikolai

unread,
Mar 9, 2012, 6:24:24 AM3/9/12
to google-a...@googlegroups.com
+1 
We had to move to our backup systems.
We are using Python / Django Framework and not Java but we have got the same problems since about 20 hours.

Please Google comment on this on the status dashboard.

Regards,
Nikolai

Takashi Matsuo ♟

unread,
Mar 9, 2012, 2:00:57 PM3/9/12
to google-a...@googlegroups.com

Hi Hamish,

Could you give me your app-id?




--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/E2c_GofputsJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.



--
Takashi Matsuo | Developer Advocate | tma...@google.com | 03-6384-9224

Amit Sangani

unread,
Mar 10, 2012, 2:24:57 PM3/10/12
to google-a...@googlegroups.com
appid: textyserver

Getting lots of:

1) com.google.apphosting.runtime.HardDeadlineExceededError exceptions, 
2) Failed startup of context com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext
3) javax.jdo.JDOException: Transaction failed to commit at org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:419) at org.datanucleus.jdo.JDOPersistenceManager.close(JDOPersistenceManager.java:281)

Status page is saying everything is normal - http://code.google.com/status/appengine which does not seem true.

Please let us know if you need more information. 

-Amit



Hamish

unread,
Mar 13, 2012, 11:55:26 AM3/13/12
to google-a...@googlegroups.com
Hi,

I will email the app id rather than post it on this public forum.

We are still seeing HTTP 500 errors today. Instances seem to take a long time to come up.

We are using HR datastore.

Tanks,
Hamish


On Friday, March 9, 2012 7:00:57 PM UTC, Takashi Matsuo (Google) wrote:

Hi Hamish,

Could you give me your app-id?

On Fri, Mar 9, 2012 at 12:44 AM, Hamish <hgr...@afilias.info> wrote:
Hi,

We use GAE to host one of our Java apps. Recently we have been experiencing quite a lot of HTTP 500 errors and timeouts from App Engine. It appears that App Engine sometimes restarts all of our instances at the same time and does not wait for instances to finish serving requests.  See the attached images:




 In the logs I see a lot of entries like this:

"Request was aborted after waiting too long to attempt to service your request."

The request time shown in the logs was typically around 10,000ms before the request was terminated.

The screenshots were taken at 23.45 GMT on 1 March. We have since experienced similar issues. The latest being yesterday afternoon at 15.30 GMT. The one yesterday was very odd. It appears that the graphs in the dashboard all got reset when the instances restarted as if the entire app had been redeployed with no history at all.

So my questions:

1) Can someone in Google please look into this for me?
2) Why would all instances be restarted at the same time?
3) Why wouldn't App Engine let the instances finish serving requests before restarting?
4) Is there any way I can elavate this issue?
5) Is there a better place to report this issue so someone in Google will investigate?

This is very serious and we need to get to the bottom of it. 

Thanks,
Hamish



--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/E2c_GofputsJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

Upscale

unread,
Mar 13, 2012, 12:05:28 PM3/13/12
to Google App Engine
Hi Takashi,

we are getting error 500 in all versions now.
Deployment is checking for finished deployment for more than 7 minutes
now.

I also have a very strange bug (java): If I remove

<filter>
<filter-name>appstats</filter-name>
<filter-
class>com.google.appengine.tools.appstats.AppstatsFilter</filter-
class>
<init-param>
<param-name>logMessage</param-name>
<param-value>Appstats available: /appstats/details?
time={ID}</param-value>
</init-param>
</filter>

<filter-mapping>
<filter-name>appstats</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>


from web.xml, the static content is not available any more.

All of the strange errors we get seem to be linked with memcache and
appstats:

Uncaught exception from servlet
com.google.apphosting.runtime.HardDeadlineExceededError: This request
(43be6c5c78b8bbb6) started at 2012/03/13 15:36:13.367 UTC and was
still executing at 2012/03/13 15:37:14.451 UTC.
at com.google.appengine.repackaged.com.google.protobuf.Descriptors
$Descriptor.<init>(Descriptors.java:577)
at com.google.appengine.repackaged.com.google.protobuf.Descriptors
$Descriptor.<init>(Descriptors.java:412)
at com.google.appengine.repackaged.com.google.protobuf.Descriptors
$FileDescriptor.<init>(Descriptors.java:342)
at com.google.appengine.repackaged.com.google.protobuf.Descriptors
$FileDescriptor.buildFrom(Descriptors.java:208)
at com.google.appengine.repackaged.com.google.protobuf.Descriptors
$FileDescriptor.internalBuildGeneratedFileFrom(Descriptors.java:271)
at
com.google.appengine.tools.appstats.StatsProtos.<clinit>(StatsProtos.java:
5427)
at com.google.appengine.tools.appstats.StatsProtos
$RequestStatProto.getDescriptor(StatsProtos.java:3504)
at
com.google.appengine.tools.appstats.MemcacheWriter.persist(MemcacheWriter.java:
238)
at
com.google.appengine.tools.appstats.MemcacheWriter.commit(MemcacheWriter.java:
168)
at
com.google.appengine.tools.appstats.AppstatsFilter.doFilter(AppstatsFilter.java:
151)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:
102)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:
35)

Do you have a resolution for the problem?

Greetings
upscale

Carter

unread,
Mar 13, 2012, 12:28:56 PM3/13/12
to Google App Engine
We too are getting lots of 500s.

Example below:
2012-03-13 09:20:56.381
Failed startup of context
com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext@1abcd9b{/,/
base/data/home/apps/s~camiologger/1.357444708065977584}
org.mortbay.util.MultiException[java.lang.NullPointerException,
com.google.apphosting.runtime.HardDeadlineExceededError: This request
(059f3b5658365659) started at 2012/03/13 16:19:07.260 UTC and was
still executing at 2012/03/13 16:20:08.115 UTC.]
at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:
656)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:
1250)
at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:
517)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:
467)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:
202)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:
171)
at
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:
123)
at com.google.apphosting.runtime.JavaRuntime
$RequestRunnable.run(JavaRuntime.java:422)
at com.google.tracing.TraceContext
$TraceContextRunnable.runInContext(TraceContext.java:449)
at com.google.tracing.TraceContext$TraceContextRunnable
$1.run(TraceContext.java:455)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:
695)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:
333)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
325)
at com.google.tracing.TraceContext
$TraceContextRunnable.run(TraceContext.java:453)
at com.google.apphosting.runtime.ThreadGroupPool
$PoolEntry.run(ThreadGroupPool.java:251)
at java.lang.Thread.run(Thread.java:679)
java.lang.NullPointerException
at java.lang.Class.isAssignableFrom(Native Method)
at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:
256)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:
685)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:
1250)
at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:
517)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:
467)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:
202)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:
171)
at
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:
123)
at com.google.apphosting.runtime.JavaRuntime
$RequestRunnable.run(JavaRuntime.java:422)
at com.google.tracing.TraceContext
$TraceContextRunnable.runInContext(TraceContext.java:449)
at com.google.tracing.TraceContext$TraceContextRunnable
$1.run(TraceContext.java:455)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:
695)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:
333)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
325)
at com.google.tracing.TraceContext
$TraceContextRunnable.run(TraceContext.java:453)
at com.google.apphosting.runtime.ThreadGroupPool
$PoolEntry.run(ThreadGroupPool.java:251)
at java.lang.Thread.run(Thread.java:679)
com.google.apphosting.runtime.HardDeadlineExceededError: This request
(059f3b5658365659) started at 2012/03/13 16:19:07.260 UTC and was
still executing at 2012/03/13 16:20:08.115 UTC.
at
com.google.appengine.runtime.Request.process-059f3b5658365659(Request.java)
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:231)
at sun.misc.Resource.getBytes(Resource.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:273)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
43)
at java.lang.reflect.Method.invoke(Method.java:616)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at org.mortbay.util.Loader.loadClass(Loader.java:91)
at org.mortbay.util.Loader.loadClass(Loader.java:71)
at org.mortbay.jetty.servlet.Holder.doStart(Holder.java:73)
at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:
242)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:
685)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:
1250)
at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:
517)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:
467)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at com.google.tracing.TraceContext
$TraceContextRunnable.runInContext(TraceContext.java:449)
at com.google.tracing.TraceContext$TraceContextRunnable
$1.run(TraceContext.java:455)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:
695)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:
333)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
325)
at com.google.tracing.TraceContext
$TraceContextRunnable.run(TraceContext.java:453)
at java.lang.Thread.run(Thread.java:679)
C 2012-03-13 09:20:56.410
Uncaught exception from servlet
javax.servlet.UnavailableException: Initialization failed.
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:
211)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:
171)
at
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:
123)
at com.google.apphosting.runtime.JavaRuntime
$RequestRunnable.run(JavaRuntime.java:422)
at com.google.tracing.TraceContext
$TraceContextRunnable.runInContext(TraceContext.java:449)
at com.google.tracing.TraceContext$TraceContextRunnable
$1.run(TraceContext.java:455)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:
695)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:
333)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
325)
at com.google.tracing.TraceContext
$TraceContextRunnable.run(TraceContext.java:453)
at com.google.apphosting.runtime.ThreadGroupPool
$PoolEntry.run(ThreadGroupPool.java:251)
at java.lang.Thread.run(Thread.java:679)

Mark

unread,
Mar 13, 2012, 12:38:56 PM3/13/12
to google-a...@googlegroups.com
This is happening to me too.  Instances getting restarted, and requests taking way longer than usual.  Getting emails from unhappy users.

My app id is bedbuzzserver, on HR ( & Java) 

Mos

unread,
Mar 13, 2012, 12:45:08 PM3/13/12
to google-a...@googlegroups.com
This seems to be the issue:

http://code.google.com/p/googleappengine/issues/detail?id=7133

Please vote for it ....

On Tue, Mar 13, 2012 at 5:38 PM, Mark <markrobe...@gmail.com> wrote:
This is happening to me too.  Instances getting restarted, and requests taking way longer than usual.  Getting emails from unhappy users.

My app id is bedbuzzserver, on HR ( & Java) 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/gX9u1do57t0J.

Naresh Talluri

unread,
Mar 13, 2012, 2:17:45 PM3/13/12
to google-a...@googlegroups.com
Me too facing the same issue, my app id is beta-sb

Thanks,
Naresh T

Nicanor Babula

unread,
Mar 13, 2012, 12:41:23 PM3/13/12
to google-a...@googlegroups.com
Same thing here too!

appid: domodentweb2
datastore: HRD

Getting calls and emails from angry customers.
Please, help!
Thanks.

On Tue, Mar 13, 2012 at 5:38 PM, Mark <markrobe...@gmail.com> wrote:
This is happening to me too.  Instances getting restarted, and requests taking way longer than usual.  Getting emails from unhappy users.

My app id is bedbuzzserver, on HR ( & Java) 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/gX9u1do57t0J.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.



--
Nicanor Cristian Babula

Mos

unread,
Mar 13, 2012, 6:35:56 PM3/13/12
to google-a...@googlegroups.com
Vote for the issue:  http://code.google.com/p/googleappengine/issues/detail?id=7133

I hope Google will give some feedback soon ?!

toonetown

unread,
Mar 15, 2012, 1:24:23 PM3/15/12
to Google App Engine
I can also verify that this is happening to us. In looking through
the issue list, it appears that *SOMETHING* happened a couple weeks
ago, causing all sorts of people to have Deadline and HardDeadline
exceptions. In addition, since so many instances are starting and
stopping, it appears that performance has been hampered as well.

Also - in looking through the issue list, it seems that this is mostly
affecting Java AppEngine instances - but similar problems appear to be
happening in the Python instances as well.

From what I can tell in reading forums and postings, there was a
system update at some point on March 7th, 2012, and problems have been
happening ever since.

Here is a list (from a quick scan - may not be comprehensive) of all
the bugs I have found that are having this issue:
http://code.google.com/p/googleappengine/issues/detail?id=7133
http://code.google.com/p/googleappengine/issues/detail?id=7130
http://code.google.com/p/googleappengine/issues/detail?id=7135
http://code.google.com/p/googleappengine/issues/detail?id=7138
http://code.google.com/p/googleappengine/issues/detail?id=7143
http://code.google.com/p/googleappengine/issues/detail?id=7146
http://code.google.com/p/googleappengine/issues/detail?id=7153
http://code.google.com/p/googleappengine/issues/detail?id=7093
http://code.google.com/p/googleappengine/issues/detail?id=7097
http://code.google.com/p/googleappengine/issues/detail?id=7101
http://code.google.com/p/googleappengine/issues/detail?id=7102
http://code.google.com/p/googleappengine/issues/detail?id=7103
http://code.google.com/p/googleappengine/issues/detail?id=7108
http://code.google.com/p/googleappengine/issues/detail?id=7113
http://code.google.com/p/googleappengine/issues/detail?id=7121
http://code.google.com/p/googleappengine/issues/detail?id=7122

I would suggest that we all vote for a single one of these - so that
it get get some visibility within Google. I think we should vote for
http://code.google.com/p/googleappengine/issues/detail?id=7133 since
as of today, it has the most stars already, and it is listed as status
"Accepted" - which is better than all the other ones that are listed
as "New".

We are paying for AppEngine, we expect that there should be some level
of support. This is ridiculous to have this problem going
on for as long as it has been, with no word on what is going on or how
to resolve it.

-Nathan

On Mar 13, 4:35 pm, Mos <mosa...@googlemail.com> wrote:
> Vote for the issue:http://code.google.com/p/googleappengine/issues/detail?id=7133
>
> I hope Google will give some feedback soon ?!
>
> On Tue, Mar 13, 2012 at 5:41 PM, Nicanor Babula <nicanor.bab...@gmail.com>wrote:
>
>
>
>
>
>
>
> > Same thing here too!
>
> > appid: domodentweb2
> > datastore: HRD
>
> > Getting calls and emails from angry customers.
> > Please, help!
> > Thanks.
>
Reply all
Reply to author
Forward
0 new messages