I only realised after posting the other thread (about the response time increase) that we're actually encountering hundreds and hundreds of database errors.
As always, the log file doesn't tell me anything. It's the generic DATASTORE_V3.Get() all over. We didn't make any deployment, we didnd't change any code recently, the load pattern
didn't change, and there is no common trend either. The errors occur for an hour or so, strike everywhere (dozens of different clients are affected) and then the problem disappears
for a few hours.
Here's the stacktrace again (from about 90 minutes ago):
https://www.small-improvements.com/dashboard javax.servlet.ServletException: com.google.apphosting.api.ApiProxy$ArgumentException: An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:268) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447) at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:452) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459) at com.google.tracing.TraceContext.runInContext(TraceContext.java:701) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456) at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) at java.lang.Thread.run(Thread.java:679) Caused by: com.google.apphosting.api.ApiProxy$ArgumentException: An error occurred parsing (locally or remotely) the arguments to DATASTORE_V3.Get(). at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.success(ApiProxyImpl.java:464) at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.success(ApiProxyImpl.java:392) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher$1.runInContext(RpcStub.java:781) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459) at com.google.tracing.TraceContext.runInContext(TraceContext.java:701) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.rpcFinished(RpcStub.java:823) at com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher.success(RpcStub.java:808) at com.google.net.rpc3.impl.client.RpcClientInternalContext.runCallbacks(RpcClientInternalContext.java:893) at com.google.net.rpc3.impl.client.RpcClientInternalContext.finishRpcAndNotifyApp(RpcClientInternalContext.java:798) at com.google.net.rpc3.impl.client.RpcNetChannel.afterFinishingActiveRpc(RpcNetChannel.java:1063) at com.google.net.rpc3.impl.client.RpcNetChannel.finishRpc(RpcNetChannel.java:911) at com.google.net.rpc3.impl.client.RpcNetChannel.handleResponse(RpcNetChannel.java:2265) at com.google.net.rpc3.impl.client.RpcNetChannel.messageReceived(RpcNetChannel.java:2068) at com.google.net.rpc3.impl.client.RpcNetChannel.access$2000(RpcNetChannel.java:143) at com.google.net.rpc3.impl.client.RpcNetChannel$TransportCallback.receivedMessage(RpcNetChannel.java:3127) at com.google.net.rpc3.impl.client.RpcChannelTransportData$TransportCallback.receivedMessage(RpcChannelTransportData.java:599) at com.google.net.rpc3.impl.wire.RpcBaseTransport.receivedMessage(RpcBaseTransport.java:417) at com.google.apphosting.runtime.udrpc.UdrpcTransport$ClientAdapter.receivedMessage(UdrpcTransport.java:424) at com.google.apphosting.runtime.udrpc.UdrpcTransport.dispatchPacket(UdrpcTransport.java:265) at com.google.apphosting.runtime.udrpc.UdrpcTransport.readPackets(UdrpcTransport.java:217) at com.google.apphosting.runtime.udrpc.UdrpcTransport$1.run(UdrpcTransport.java:81) at com.google.net.eventmanager.AbstractFutureTask$Sync.innerRun(AbstractFutureTask.java:260) at com.google.net.eventmanager.AbstractFutureTask.run(AbstractFutureTask.java:121) at com.google.net.eventmanager.EventManagerImpl.runTask(EventManagerImpl.java:578) at com.google.net.eventmanager.EventManagerImpl.internalRunWorkerLoop(EventManagerImpl.java:1002) at com.google.net.eventmanager.EventManagerImpl.runWorkerLoop(EventManagerImpl.java:884) at com.google.net.eventmanager.WorkerThreadInfo.runWorkerLoop(WorkerThreadInfo.java:136) at com.google.net.eventmanager.EventManagerImpl$WorkerThread.run(EventManagerImpl.java:1855)
And this is the error spike distribution today:
I find it very frustrating that GAE is so inconsistent. We had severe issues two months ago and had to reimburse several clients for downtimes caused by very similar problems. Then it was fine for 2 months,
and just when I go on vacation, it breaks down again. Anything I can do on my end?
Cheers,
Per
Uncaught exception from servlet org.apache.xerces.util.ObjectFactory$ConfigurationError: Provider org.apache.xerces.parsers.StandardParserConfiguration could not be instantiated: com.google.apphosting.api.DeadlineExceededException: This request (ccb0be8e8ff82131) started at 2012/10/04 01:30:10.362 UTC and was still executing at 2012/10/04 01:31:10.042 UTC. at org.apache.xerces.util.ObjectFactory.newInstance(ObjectFactory.java:303) at org.apache.xerces.util.ObjectFactory.createObject(ObjectFactory.java:224) at org.apache.xerces.util.ObjectFactory.createObject(ObjectFactory.java:119) at org.apache.xerces.parsers.SAXParser.<init>(SAXParser.java:140) at org.apache.xerces.parsers.SAXParser.<init>(SAXParser.java:125) at org.apache.xerces.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:102) at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParserImpl(SAXParserFactoryImpl.java:112) at org.apache.xerces.jaxp.SAXParserFactoryImpl.setFeature(SAXParserFactoryImpl.java:140) at org.mortbay.xml.XmlParser.makeFactorySecure(XmlParser.java:162) at org.mortbay.xml.XmlParser.setValidating(XmlParser.java:102) at org.mortbay.xml.XmlParser.<init>(XmlParser.java:91) at org.mortbay.jetty.webapp.TagLibConfiguration.configureWebApp(TagLibConfiguration.java:210) at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1247) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:219) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447) at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:452) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459) at com.google.tracing.TraceContext.runInContext(TraceContext.java:701) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456) at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) at java.lang.Thread.run(Thread.java:679)
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/U9KKQ-WMZ98J.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
59.167.133.100 - - [03/Oct/2012:16:56:02 -0700] "GET / HTTP/1.1" 500 0 "http://small-improvements.com/" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_6_7) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4"
"www.small-improvements.com" ms=2463 cpu_ms=2210 exit_code=103 instance=00c61b117ca2c5dd4a6002dd13f39c6296da66
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/zwEIyb-Sf6oJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.