500 errors + high latency + please communicate with us

[Nikolai]

this is so wrong. I dont see any of these problems reflected in the appengine system status. I don't understand why there is zero communication from google concerning the outages everybody in this forum is seeing right now.

This is not a small hickup, but something we (and others) are fighting since 20 hours.

Could someone from google please comment on the current issues und reflect the status on the appengine status page?

thank you!

nikolai

[Nikolai]

oh and sorry.

the app id for the above screenshot is spiegeltvapp

regards,

nikolai

[Fabian Topfstedt]

While we are experiencing problems on spiegeltvapp, we don't have them on dctptv - which is running the same (python) codebase.

Best regards,

Fabian

[Sharp-Developer.Net]

Our instances count sky-rocketed about 3 hours ago from usual 5-10 to over 500 and we exceeded our limits. We had no deployments in last 24 hours.

App ID is: petaclasses

We are loosing money and trust of our customers and no news from Google and no way to communicate - smells very bad to me.

I love GAE but my team is pushing me to evaluate other alternatives.

Alex

[Amit Sangani]

We are facing the same issues as others are experiencing here. High
latency, instance count sky-rocketed, lot of errors in logs
(HardDeadlineExceededError , java.lang.NoClassDefFoundError, etc).
This was working perfectly 2 days back. Ever since the last
maintenance (March 7th) GAE has been very unstable for us.

Our app id is: textyserver

On Mar 9, 6:38 am, "Sharp-Developer.Net"

<alexander.trakhime...@gmail.com> wrote:
> Our instances count sky-rocketed about 3 hours ago from usual 5-10 to over
> 500 and we exceeded our limits. We had no deployments in last 24 hours.
>
> App ID is: petaclasses
>
> We are loosing money and trust of our customers and no news from Google and
> no way to communicate - smells very bad to me.
>
> I love GAE but my team is pushing me to evaluate other alternatives.
>
> Alex
>
>
>
>
>
>
> On Friday, 9 March 2012 08:40:20 UTC-4, Nikolai wrote:
>

> > <https://lh6.googleusercontent.com/-8WF96dBSCf4/T1n5Cg1ePDI/AAAAAAAABQ...>

[Ikai Lan (Google)]

Are you guys seeing any particular errors in the logs or other observed strange behavior? Thanks for providing an application ID, it'll make it easier for us to look into this.

There seem to be several parallel issues occurring. It's still unclear to me whether or not they are related. We need to un-cross the streams.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Christopher Wong]

Same for us, on a normal day we need around 20 instances to keep the app running, the now is running 73 - 100 instances. (appid:tetris4fun)

Here're some screen shots, notice that the memory usage just went sky rocketed. Would it be a memory issue on your side? We didn't make any new deployment for the past week.

Also notice the instances with 0 request.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Nikolai]

Hey Ikai,
thank you for looking into this. I can confirm that I saw 500 server error today when viisiting out app that weren't logged. But the number of logged errors is also high.
Fumny part is, that i see a lot of errors on the dashboard main page, but when i click on these errors to get some insight the logs seem to be empty.
Appid is spiegeltvapp. We are currently using the appengine production servers because things seemed to have gotten better but we might have to swap to our backup soon because error rates are rising again.
Regards,
nikolai

[Nikolai]

[Luis]

We are suffering something pretty similar. Without having made any
kind of change either in the application configuration (in the
console) or the actual code (deployments), during the day we have been
suffering an abnormal low performance and continuous errors (500
server error, ExceptionInInitializerError, DeadlineExceededException.
Additionally we noticed that for a normal work load the total number
of instances created has increased significantly during the last 12
hours (please see attached screenshot).

Please find log files and screenshots in the ticket we opened:
http://code.google.com/p/googleappengine/issues/detail?id=7113

Our appid is reportingsuiteengine and we are using master/slave

Thanks,
Luis

> >http://groups.google.com/group/google-appengine?hl=en.- Ocultar texto de la cita -
>
> - Mostrar texto de la cita -

[Christopher Wong]

tetris4fun is the app id.

I noticed there are empty instances running. Usually I need 20 instances to keep the app alive, now it's 150+ instances. Can you please look into that?

[Ikai Lan (Google)]

Luis, do these happen on loading requests?

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com

[Ronoaldo José de Lana Pereira]

Hello Ikai,

Here is my production issue with logs and more details: http://code.google.com/p/googleappengine/issues/detail?id=7108.

If any additional information is needed, please let me know. What I observed is that when a request is aborted, the app throws a deadline in the exact line when a filesystem access is being performed (like, Classes being loaded, or an app resource loaded into code).

With threadsafe true we one instance raising this error at startup was causing lots of requests to fail, so we disabled concurrent requests to try reducing the impact of the problem to our users. We also changed the app preferences in the admin console all to automatic (after trying all sorts of combinations without any success).

Hope the team can fix this quickly, our costumers are getting impatient :)

Another info is that this is increasing after the last maintenance period (03/07).

Hope this helps,

-Ronoaldo

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Thomas]

hi ikai! same problem here: lots of 500s, bad performance and way too many instances running since last week. please help us with this!

appid: typescout

many thanks!

thomas

> > To post to this group, send email to google-appengine@googlegroups.com.

> > To unsubscribe from this group, send email to

> > google-appengine+unsubscribe@googlegroups.com.

> > For more options, visit this group at

> >http://groups.google.com/group/google-appengine?hl=en.- Ocultar texto de la cita -
>
> - Mostrar texto de la cita -

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Amit Sangani]

Hi Ikai & Google team,

Getting lots of 500s like others have experienced. Now mainly getting DatastoreTimeoutException...

com.google.appengine.api.datastore.DatastoreTimeoutException: 
	at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:46)
	at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:76)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:106)
	at com.google.appengine.api.datastore.FutureHelper$CumulativeAggregateFuture.get(FutureHelper.java:145)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
	at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:72)
	at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:33)
	at com.google.appengine.api.datastore.DatastoreServiceImpl$2.runInternal(DatastoreServiceImpl.java:113)
	at com.google.appengine.api.datastore.DatastoreServiceImpl$2.runInternal(DatastoreServiceImpl.java:110)
	at com.google.appengine.api.datastore.TransactionRunner.runInTransaction(TransactionRunner.java:31)
	at com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:110)
	at com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:94)

Please fix it as this has been going on for last 48 hours. thanks!

-Amit

appid: textyserver

[Nischal Shetty]

We too are facing a lot of 500 errors for our app with app id : 'justunfollow'  

[cz]

My app (adb2) is getting over quota errors and is no longer serving
requests. This has never happened before. According to the dashboard
my app has used 34 instance hours to process under 100 requests which
I can't imagine is possible under normal circumstances (especially
since the app doesn't use much of the service APIs or datastore).
Despite the app's low traffic there are a number of people that depend
on using it so I hope Google can look into this.

Just a little side note: There seems to be more and more recognition
that Google's lack of support for paid services is getting to be kind
of a drag. Even the el cheapo $4/mo hosting services have trouble
ticket systems and respond pretty quickly to issues. It's nice that
some of the developer support team (ie Ikai) follow up on some things
but not having a real support system is pretty janky. And before
someone says "oh but you can sign up for premier support!", it's just
not affordable for small projects.

In general GAE has been a great platform to build apps for and I
really appreciate it, but now that I'm paying for it I feel like I can
complain a little...

thanks,
- Claude

[Miroslav Genov]

Hello Ikai, 

We are experiencing similar issue. Our app id is: cmsevobg. Normally our app (java based that is using Guice) is instantiating GuiceServletContextListener in ~10 seconds on GAE. On my mac book air with i5 cpu local app is booting in "INFO: Time:4892 ms".  With the same code base on GAE we are getting initialization time in: 

Time:30569 ms

Time:25105 ms

So our app is booting extremely slow now. 

We updated front-end instance class from F1 to F4 (2 days and more) and it seems that the behavior is same. Do you need any further information  ?

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Rod Chen]

I got the same problem. My appid: badgm.

DB and memory cache API call became very slow, so that performance is very poor.

[Rafael Gutierrez]

We are having the same issue (DeadlineExceptions, InitializationExceptions, etc) since the last maintenance.

More than that, we have some remote units that make HTTP requests to post data to our application using an GPRS module.

Some of those units (more than 40) are no longer sending these requests (we can't even see them in the logs), meanwhile NOTHING changed in the remote units code (stable for over 1 year), and we verified that they still connecting to the GPRS carrier.

Please Google, give me some feedback... our clients are pushing us to have some news, and we can't even justify them the technical issues. It could kill our business reliability and reputation. 

App Id: innervision-ui :: Master/Slave

[Rui Oliveira]

Hi Ikai,

My appId is phone-menu. 

I'm with the same problem.

My conclusions:

After deploy in the next 10 minutes I get a lot of errors. After that the default version will run ok( not completely ok).

Concurrent request on the first 10 seconds will improve the probability to get errors

Default version works much better than the other versions. The probability to get an error from no default versions is very high. ( 

 an unexpected exception: com.google.apphosting.api.ApiProxy$CancelledException: The API call datastore_v3.RunQuery() was explicitly cancelled.

	at com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java:385)
	at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:588)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
	at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
	at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:102)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:249)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
	at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:135)
	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:422)
	at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:449)
	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455)
	at com.google.tracing.TraceContext.runInContext(TraceContext.java:695)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325)
	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453)
	at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
	at java.lang.Thread.run(Thread.java:679)
Caused by: com.google.apphosting.api.ApiProxy$CancelledException: The API call datastore_v3.RunQuery() was explicitly cancelled.

[leeflea]

Nikolai, I just saw your posting of the very same problem I'm having but since I just switched browsers yesterday, I just didn't know what to do about this. You see, I'm not very computer smart and many of the remarks are way over my head as I was merely a Registered Nurse. I, indeed, did make a general posting of the problems many of of us are sharing. Thank you for posting what you did as I hope someone will help us all.

With regards, Leeflea

On Friday, March 9, 2012 6:40:20 AM UTC-6, Nikolai wrote:

this is so wrong. I dont see any of these problems reflected in the appengine system status. I don't understand why there is zero communication from google concerning the outages everybody in this forum is seeing right now.

This is not a small hickup, but something we (and others) are fighting since 20 hours.

Could someone from google please comment on the current issues und reflect the status on the appengine status page?

thank you!

nikolai

[toonetown]

Same here - please see http://code.google.com/p/googleappengine/issues/detail?id=7133 for some more details (there are at least 15 other bugs logged with the same issue...we need to make this issue more visible to Google...)

-Nathan

[toonetown]

You can see the full list of bugs in this thread: https://groups.google.com/d/topic/google-appengine/nNoWv1uVqwo/discussion

-Nathan

[kusmi]

My app is experiencing the same kind of problems - suddenly the latency increased and HTTP 500 errors. I'm using HR-datastore and my app is very simple and does not a lot of things. I'm using prodeagle monitoring though (every minute there is a monitoring request) - no changes made in my app, I just re-uploaded the binary to make sure the app is restarted again.

My app: mapoutapp

[kusmi]

BTW, this is the Latency-Graph I'm currently experience: 8 days ago (approx. March 13) everything was still fine then it started climbing and the last 1-2 days it's extremely sporadic... (not sure if this is a sampling issue and the data earlier are just smoothed).

I have not made changes to my code, just upgraded to 1.6.3 on March 1.

[Mos]

Same for me.  It seems to be very obviously that there is a major problem in the GAE runtime starting around 8 days ago.
My instance scheduling went wild (see attached picture).
I updated to version 1.6.3. on March 9 -> I think the issue is not related to the SDK Version number.

How long does it take to get an answer (or better fix) from Google?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/lqP2zwJ1dhYJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[James Mortensen]

I submitted a ticket here, yesterday, for all of the 500 errors that I'm seeing in my app.  I'm hoping we can get this sorted out soon as we're hoping to launch in the next 2 weeks!

http://code.google.com/p/googleappengine/issues/detail?id=7444

I also asked this question on Stackoverflow last month and haven't received any answers. I'm seeing these exceptions as well:  Unable to load several versions of AppEngine Project and Seeing HardDeadLineExceeded Exceptions

James