Lots of errors recently

[Ricky Button]

As of a few hours ago, I've been getting quite a few 500 errors and deadline exceeded exceptions, but my code has not changed at all. When requests do actually go through, the response times are very high, (20-25 seconds) but the cpu ms used is very low, which leads me to believe that the problem is not on my end. Anyone else experiencing any problems?

[Ricky Button]

Also, getting 500 errors (deadline) on a missing favicon instead of a 404.

[Dan]

I've been seeing a lot of deadline exceed errors as well. Some
requests take over 60 seconds, but only a few hundred cpu ms.

[JH]

Are you using M/S or HRD? If you're using HRD I hope these type of
posts are addressed before GAE leaves preview.

[Ricky Button]

I'm using M/S but if it was a datastore outage, then they would probably tell us, plus it is happening on things that don't even come close to the datastore (favicon)

[Ricky Button]

Also this essentially means my app has been down for 12 hours. Still throwing 500s on things like static content.

[JH]

I don't think you can downgrade your app? Production servers are
running 1.5.0.

On Jun 3, 11:44 am, Ricky Button <m...@rickybutton.com> wrote:
> Well I seem to have just temporarily fixed the issue by downgrading to
> 1.4.3. I don't know if that fixed it though. It still scared the crap out of
> us, and we would love a response from Google on this issue.

[Ricky Button]

When I started to use the 1.4.3 libs it started working again for about an hour, but it has just stopped working again.

[Ricky Button]

Also, none of these errors show up on the dev server. I am completely stumped, my app has been down for about 20 hours now and I have no idea why.

[Ricky Button]

1. Sorry to have to keep bumping this but my app literally does not function. Here is a stack trace from one of the 500 errors.

   
   

   2011-06-03 14:19:36.041

   Failed startup of context com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext@1c958af{/,/base/data/home/apps/extemprep/4.350881581648851921}
   com.google.apphosting.api.DeadlineExceededException: This request (ea4addffdf95828c) started at 2011/06/03 21:17:54.818 UTC and was still executing at 2011/06/03 21:19:35.796 UTC.
   	at java.io.UnixFileSystem.getLastModifiedTime(Native Method)
   	at java.io.File.lastModified(File.java:902)
   	at java.util.zip.ZipFile.<init>(ZipFile.java:143)
   	at java.util.jar.JarFile.<init>(JarFile.java:150)
   	at java.util.jar.JarFile.<init>(JarFile.java:87)
   	at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:660)
   	at sun.misc.URLClassPath$JarLoader.access$600(URLClassPath.java:555)
   	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:622)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:614)
   	at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:598)
   	at sun.misc.URLClassPath$3.run(URLClassPath.java:348)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:337)
   	at sun.misc.URLClassPath.getLoader(URLClassPath.java:314)
   	at sun.misc.URLClassPath.findResource(URLClassPath.java:161)
   	at java.net.URLClassLoader$2.run(URLClassLoader.java:379)
   	at java.security.AccessController.doPrivileged(Native Method)
   	at java.net.URLClassLoader.findResource(URLClassLoader.java:376)
   	at com.google.apphosting.runtime.security.UserClassLoader.findResource(UserClassLoader.java:699)
   	at java.lang.ClassLoader.getResource(ClassLoader.java:977)
   	at org.mortbay.resource.Resource.newSystemResource(Resource.java:203)
   	at org.mortbay.jetty.webapp.WebXmlConfiguration.configureDefaults(WebXmlConfiguration.java:159)
   	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1230)
   	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
   	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
   	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:191)
   	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:168)
   	at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:123)
   	at com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:260)
   	at com.google.apphosting.base.RuntimePb$EvaluationRuntime$2.handleRequest(RuntimePb.java:9669)
   	at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:439)
   	at com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:573)
   	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:448)
   	at com.google.tracing.TraceContext.runInContext(TraceContext.java:688)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:326)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:318)
   	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:446)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
   	at java.lang.Thread.run(Thread.java:636)
   

2. C2011-06-03 14:19:36.048

   Uncaught exception from servlet
   javax.servlet.UnavailableException: Initialization failed.
   	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:200)
   	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:168)
   	at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:123)
   	at com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:260)
   	at com.google.apphosting.base.RuntimePb$EvaluationRuntime$2.handleRequest(RuntimePb.java:9669)
   	at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:439)
   	at com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:573)
   	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:448)
   	at com.google.tracing.TraceContext.runInContext(TraceContext.java:688)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:326)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:318)
   	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:446)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
   	at java.lang.Thread.run(Thread.java:636)

[Francois Masurel]

Check this post, it might be related :

https://groups.google.com/d/topic/google-appengine-java/6tnyfJLBIXM/discussion

[Ricky Button]

I have taken a look at that post,and I'm not sure if the two issues are related, I don't catch any DeadlineExceededExceptions anywhere

If a Googler wants to take a look at my app, the id is "extemprep"

[kevin]

I met the same problem twice.

Normal cpu time, but very high real time.
according to the logs recorded by my code, it looks like very normal.
And the response status are 200.

That means appengine performed the job well at the normal level, but
something of infrastructure slow down the response time for some
reason.

After hours, the problem suddenly leave as it suddenly come.

I think this could a very critical issue if i running my App in
business. Why no Googler come out to say something after 12 days?
btw, I'm using M/S.

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/NG0zVmVhcGo4SFVK.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

[kevin]

Now the problem comes again. A request sometimes took even more than
60s real time with 45ms cpu time.

Anybody has idea?

[Waleed Abdulla]

I'm noticing that as well. The performance of my task queues dropped significantly, and this might be the reason. The clock time that I calculate is much less than the time I see in my app engine logs. Sometimes the difference is 10 seconds or more. In other words, an extra 10 seconds delay is being added that is outside of the actual execution time of the request. There seems to be an infrastructure issue on GAE.