Every day around 9.10 AM Brussels time, huge drop in GAE performances

[gafal]

I've been experiencing this for almost a week now.

Requests take 10x longer than usual!!

It seems to start around 9 and stop around 9:30...


my app id is myagendapro
Can anyone have a look?

[Vadim Statishin]

The same problem!

2013/2/5 gafal <gael....@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Vadim

[Ice13ill]

I also have trouble with GAE. I get lots of deadline exceeded exception and I cannot fetch external URLs most of the time. I use Java SDK.

I did not have this problem until about 30-60 minutes ago.

[gafal]

Happening again today 6th February.
Can anyone at Google just care? It would really be appreciated...

[Per]

While I haven't seen that problem, I'd love to learn more about it. Do you have more information other than "it being slow"? Did you try running appstats? Is it db-related, memcache, CPU slowness, or maybe even session-related? Google may find out by looking at your app, but us others cannot, and I'd love to hear what that actual issue is, in case we run into something similar.

[gafal]

This might help: At 9:00 AM, the GAE Daily quotas are reset. It might put GAE under a heavy load.

here is a log of a cron  (requestwhich does nothing (see code here under) execution duration.

1. 1. 2013-02-06 09:39:00.657 /cron/StayUp 200 6ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

2. 1. 2013-02-06 09:38:00.625 /cron/StayUp 200 76ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

3. 1. 2013-02-06 09:37:00.475 /cron/StayUp 200 8ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

4. 1. 2013-02-06 09:36:00.444 /cron/StayUp 200 7ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

5. 1. 2013-02-06 09:35:00.418 /cron/StayUp 200 9ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

6. 1. 2013-02-06 09:34:02.374 /cron/StayUp 200 1411ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

7. 1. 2013-02-06 09:33:00.869 /cron/StayUp 200 7ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

8. 1. 2013-02-06 09:32:00.789 /cron/StayUp 200 31ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

9. 1. 2013-02-06 09:31:00.701 /cron/StayUp 200 8ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

10. 1. 2013-02-06 09:30:00.629 /cron/StayUp 200 7ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

11. 1. 2013-02-06 09:29:00.561 /cron/StayUp 200 13ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

12. 1. 2013-02-06 09:28:02.473 /cron/StayUp 200 1432ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

13. 1. 2013-02-06 09:27:01.572 /cron/StayUp 200 525ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

14. 1. 2013-02-06 09:26:00.876 /cron/StayUp 200 11ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

15. 1. 2013-02-06 09:25:00.747 /cron/StayUp 200 534ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

16. 1. 2013-02-06 09:24:01.129 /cron/StayUp 200 82ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

17. 1. 2013-02-06 09:23:03.786 /cron/StayUp 200 2988ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

18. 1. 2013-02-06 09:22:00.555 /cron/StayUp 200 503ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

19. 1. 2013-02-06 09:21:01.022 /cron/StayUp 200 224ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

20. 1. 2013-02-06 09:20:01.810 /cron/StayUp 200 1274ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

21. 1. 2013-02-06 09:19:02.328 /cron/StayUp 200 1778ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

22. 1. 2013-02-06 09:18:00.498 /cron/StayUp 200 54ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

23. 1. 2013-02-06 09:17:04.781 /cron/StayUp 500 3733ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

    2. W 2013-02-06 09:17:04.781 A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the nex

24. 1. 2013-02-06 09:16:01.497 /cron/StayUp 200 1246ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

25. 1. 2013-02-06 09:15:02.913 /cron/StayUp 200 2100ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

26. 1. 2013-02-06 09:14:00.588 /cron/StayUp 200 44ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

27. 1. 2013-02-06 09:13:01.350 /cron/StayUp 200 375ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

28. 1. 2013-02-06 09:12:29.899 /cron/StayUp 200 89203ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

    2. I 2013-02-06 09:11:23.884 javax.servlet.ServletContext log: Initializing Spring root WebApplicationContext

    3. I 2013-02-06 09:12:29.895 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This requ

29. 1. 2013-02-06 09:10:00.559 /cron/StayUp 200 10ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

30. 1. 2013-02-06 09:09:00.335 /cron/StayUp 200 55ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

31. 1. 2013-02-06 09:08:01.068 /cron/StayUp 200 370ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

32. 1. 2013-02-06 09:07:00.547 /cron/StayUp 200 100ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

33. 1. 2013-02-06 09:06:00.059 /cron/StayUp 200 11ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

34. 1. 2013-02-06 09:05:01.017 /cron/StayUp 200 161ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

35. 1. 2013-02-06 09:04:00.693 /cron/StayUp 200 15ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

36. 1. 2013-02-06 09:03:00.600 /cron/StayUp 200 25ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

37. 1. 2013-02-06 09:02:00.558 /cron/StayUp 200 9ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

38. 1. 2013-02-06 09:01:00.338 /cron/StayUp 200 143ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

39. 1. 2013-02-06 09:00:01.092 /cron/StayUp 200 230ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

40. 1. 2013-02-06 08:59:00.819 /cron/StayUp 200 182ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

41. 1. 2013-02-06 08:58:00.553 /cron/StayUp 200 34ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

42. 1. 2013-02-06 08:57:01.346 /cron/StayUp 200 401ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

43. 1. 2013-02-06 08:56:00.868 /cron/StayUp 200 210ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

44. 1. 2013-02-06 08:55:00.561 /cron/StayUp 200 12ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

45. 1. 2013-02-06 08:54:00.454 /cron/StayUp 200 9ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

46. 1. 2013-02-06 08:53:00.331 /cron/StayUp 200 9ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

47. 1. 2013-02-06 08:52:00.242 /cron/StayUp 200 12ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

48. 1. 2013-02-06 08:51:01.118 /cron/StayUp 200 70ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

49. 1. 2013-02-06 08:50:00.903 /cron/StayUp 200 223ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

50. 1. 2013-02-06 08:49:00.592 /cron/StayUp 200 42ms 0kb AppEngine-Google; (+http://code.google.com/appengine)

public class StayUp extends HttpServlet {

    /**
     *
     */
    private static final long serialVersionUID = -3371245850830199827L;

    public void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {

[Jacob G]

Since Monday evening, I've seen terrible performance throughout the day on GAE. Almost every request spawns a new instance, and the prior instance(s) disappear without any information in the logs. Latency is terrible!

Issue created here:  http://code.google.com/p/googleappengine/issues/detail?id=8781

Make sure you star it so it gets attention!

[gafal]

Same problem today same time.
I've created an issue:
Issue 8788: Every day around 9.00 AM Brussels time, huge drop in GAE performances

Please star it.

[gafal]

It's happening AGAIN!!!!!!!!!!!!!!
the instances that are spun up are unable to serve any request!

H E L P

This is not acceptable!!

[Philip Kilner]

Hi,

On 11/02/13 08:20, gafal wrote:
> It's happening AGAIN!!!!!!!!!!!!!!
> the instances that are spun up are unable to serve any request!
>

> *H E L P*
>
> This is not acceptable!!
>

I've just experienced something similar: -

- The app was apparently temporarily frozen.
- Browsers just showed a "loading" spinner.
- Console showed no instances.
- The bulk import that I was running aborted with error, but it's not
100% clear to me right now whether that was the same issue.

There was certainly something odd going on, around 8am UK time today -
from my PoV, it started at or a whisker before 8am GMT, and lasted c.20
mins.

Will star issue.


--

Regards,

PhilK


'a bell is a cup...until it is struck'

[Philip Kilner]

Hi Again,

On 11/02/13 08:33, Philip Kilner wrote:
> There was certainly something odd going on, around 8am UK time today -
> from my PoV, it started at or a whisker before 8am GMT, and lasted c.20
> mins.
>

Spoke too soon - am still getting anomalous behaviour, specifically
spinning (un-served) requests, and console reports no instances - looks
like it's having trouble spinning up instances to serve requests. It's
improving over time, but is still not what I'd expect.

Based on what I'm seeing, I'm astonished that this does not show as a
spike in the latency on the status page.


--

Regards,

PhilK


e: ph...@xfr.co.uk - m: 07775 796 747

'work as if you lived in the early days of a better nation'
- alasdair gray

[Sameer Lodha]

One of the reason for this could be that the billing cycle for the day ends/starts at that time.

Everyday, we see that all running Instances are shut down & new ones go up at this time.

Sameer

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

[gafal]

I'm afraid to see the same thing happening tomorrow.
Today my app has almost been unavailable for 30 minutes!!!!!

Could anyone from Google react on this?

Le lundi 11 février 2013 11:32:31 UTC+1, Sam a écrit :

One of the reason for this could be that the billing cycle for the day ends/starts at that time.

Everyday, we see that all running Instances are shut down & new ones go up at this time.

Sameer

On Mon, Feb 11, 2013 at 2:25 PM, Philip Kilner <ph...@xfr.co.uk> wrote:

Hi Again,

On 11/02/13 08:33, Philip Kilner wrote:

There was certainly something odd going on, around 8am UK time today -
from my PoV, it started at or a whisker before 8am GMT, and lasted c.20
mins.

Spoke too soon - am still getting anomalous behaviour, specifically spinning (un-served) requests, and console reports no instances - looks like it's having trouble spinning up instances to serve requests. It's improving over time, but is still not what I'd expect.

Based on what I'm seeing, I'm astonished that this does not show as a spike in the latency on the status page.


--

Regards,

PhilK


e: ph...@xfr.co.uk - m: 07775 796 747

'work as if you lived in the early days of a better nation'
- alasdair gray

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

[Per]

It is indeed a bit embarrassing that nobody at Google picks this up. I only visit the forum every two or three days and I noticed this thread. How hard can it be for someone at Google to take a look.

But still. I'd also urge you to provide a lot more information. Show us data from external sources (Pingdom, NewRelic etc). Run Appstats and post the results. Provide more information that highlights better how severe the problem is. Show everyone that you have done your homework and that it's basically impossible to be your own fault.  It's good practice, and it will make it more probable that someone at Google picks this up.  (I'd also add "increase to F4 unless you already have, since it's better to pay a bit extra than to be down, and see if that hels")

Cheers,
Per

[gafal]

How could it be my fault if it starts every day at the same time and lasts for about same duration and I do not have any cron process running at that time?
How could I screw the instances start up so that they don't even fully load? :-)

I cross my fingers that it doesn't happen today...

[gafal]

Apparently it did not happen this morning. Just in case, I had put my instances as F2 before 9.00
To whoever did anything to improve thid:

THANKS!

[gafal]

I talkd too soon!!!

This morning 9.00 am brussels time it happens again!!

Instances are started but are not healthy. They cannot process any request and they cannot be shut down!!!
Beside them are 2 healthy instances which are fro any obscure reason not receiving any request.

I don't know what you are trying to achieve but we would be pleased to know it.

Until you get them: Could you implement temporary work around: to serve request only with healthy (instances that do respond to request) and warmed up instances)?

I'm in the process of quitting GAE and every day, you prove I'm right but until I do I hope that you'll solve this issue!

[gafal]

The best part on this is still that in order to attempt to make the system more responsive, we use F2 or F4.
Because of that, we pay more money to Google for a bad service!

Le mardi 5 février 2013 09:18:55 UTC+1, gafal a écrit :

[Mos]

> Because of that, we pay more money to Google for a bad service!

That's one of the biggest faults of GAE.
Whenever there is a problem on the platform, the customer has to pay even more.
The last days the instance scheduler had the problem and was restarting instances like crazy.
Usually we don't have to pay extra  "Frontend Instance hours" for our application because of low traffic.
But in these days we do not only have downtimes but had to pay a lot of "Frontend Instance hours"

That sucks!

--

[gafal]

SAME PROBLEM OCCURS AGAIN !!!!!!!!!!!!!

[jon]

Seeing you get ignored by Google makes me worried. What if a recurring problem strikes my app and I get ignored like you? Surely I'm not the only fellow developer who feels nervous about Google's inaction.

A bit of history: I have in the past experienced a recurring problem, but very infrequently. It was caused by a bug in the memcache implementation. I filed a bug report but Google was very slow to react. It took about 1 year for Google to release a fix. I still remember the pain and the feeling of helplessness when my app went down. I really do not want to go through what you're going through :-(

[gafal]

I'm worried too!
Could the lack of support be a sign that Google is planning to deprecate GAE??

[Joakim]

Or perhaps the coming Google I/O is causing crunch time all around, leaving little time to attend to forums.
I'm not here to make excuses, but it is unwise to spread negative speculation, Fear, Uncertainty and Doubt. I agree that Google should have responded here a long time ago.
Have you created a production issue in the tracker? That is the only officially supported way to report problems, as far as I know.

[gafal]

Hi Joakim,

Sorry I had to write my previous message to get an answer from Google. At least it has reached its goal.

Yes, a critical Production issue has been created 12 days ago without any reaction from Google ( Issue 8788: Every day around 9.00 AM Brussels time, huge drop in GAE performances). Other people have created related issues.
Why didn't we get any feedback on this?
Do we have to wait until Google I/O to get help?

[Joakim]

Hi,
I'm not from Google, just another customer. Sorry if I caused confusion.
I just wanted to say that I think it's poor of Google to not even say hello in a thread such as this, but none of us will benefit from such speculation.
I've starred your issue.

[stevep]

Feb 3rd 2013, The Wall Street Journal did an in-depth article about MSFT, Amazon and Google's competing cloud offerings - http://on.wsj.com/11jeOgD. Despite Google App Engine being the real competitor to the described cloud services, it was never mentioned. Perhaps it was the reporting, but more likely G. only talked about Compute Engine. (If you are not familiar with WSJ articles -- this was a in-depth effort). The article directly quotes Shailesh Rao "...who runs Google's 'cloud platform' group...". Man do I ever hope this was just a "public relations positioning" moment rather than a "canary-in-the-coal-mine" moment.

[gafal]

AND TODAY AGAIN!!!!!!!!!
GAE is just giving 500 errors to all requests!!! Application completely down in peak hour!!!


THANKS A LOT FOR YOUR SUPPORT.

What do we need to get some help? Contact the press?

HELP!!!!!!!!!!!

IIf you read this post and know some people at Google. Could you ask them what's happening?

[gafal]

Google might need a few hundred people to star an issue to take action.

I suggest we ask our impacted customers to star the issue http://code.google.com/p/googleappengine/issues/detail?id=8788

[Vinny P]

Out of curiosity, what is the average instance startup time at offpeak hours?

-Vinny

[gafal]

This morning again:
Instances cannot even complete their loading phase.
Application down for 40 min in peak hour...

Normal startup time is 20s.

I'm probably running on same VM as another application that does heavy processing every day at that time.
Do you know a way to trigger application VM migration?

[doright]

I have had an unusable system for the last 5 days. every morning.  (Paris time, until around 9:30 am)

here is a typical log.  During this time period it simply won't startup instances.

appid mybikeshopmanager.com  GAE/J - just java + JDO; no heavy plugins. 

Google, please just tell me you even care that your platform is completely unusable EVERY DAY, at predictable times.

1. 2013-03-11 08:37:19.528

   Failed startup of context com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext@fd4662{/,/base/data/home/apps/mybikeshopmanager/252b.365767866105007180}
   org.mortbay.util.MultiException[java.lang.ExceptionInInitializerError, com.google.apphosting.runtime.HardDeadlineExceededError: This request (1c53170a9d992a42) started at 2013/03/11 07:36:18.921 UTC and was still executing at 2013/03/11 07:37:19.510 UTC.]
   	at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:656)
   	at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
   	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
   	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
   	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
   	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:219)
   	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194)
   	at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
   	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447)
   	at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)
   	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)
   	at com.google.tracing.TraceContext.runInContext(TraceContext.java:703)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330)
   	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)
   	at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
   	at java.lang.Thread.run(Thread.java:679)
   java.lang.ExceptionInInitializerError
   	at com.tasteprovence.VVbooking.server.BookingServiceImpl.init(BookingServiceImpl.java:226)
   	at javax.servlet.GenericServlet.init(GenericServlet.java:212)
   	at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
   	at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
   	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   	at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
   	at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
   	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
   	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
   	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
   	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:219)
   	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194)
   	at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
   	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447)
   	at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)
   	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)
   	at com.google.tracing.TraceContext.runInContext(TraceContext.java:703)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330)
   	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)
   	at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
   	at java.lang.Thread.run(Thread.java:679)
   Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
   NestedThrowables:
   java.lang.reflect.InvocationTargetException
   	at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186)
   	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
   	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:1086)
   	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:914)
   	at com.tasteprovence.VVbooking.server.util.PMF.<clinit>(PMF.java:8)
   	... 23 more
   Caused by: java.lang.reflect.InvocationTargetException
   	at com.google.appengine.runtime.Request.process-1c53170a9d992a42(Request.java)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:616)
   	at java.security.AccessController.doPrivileged(AccessController.java:63)
   	at javax.jdo.JDOHelper.invoke(JDOHelper.java:1951)
   	at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
   	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
   	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:1086)
   	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:914)
   	at com.tasteprovence.VVbooking.server.util.PMF.<clinit>(PMF.java:8)
   	at com.tasteprovence.VVbooking.server.BookingServiceImpl.init(BookingServiceImpl.java:226)
   	at javax.servlet.GenericServlet.init(GenericServlet.java:212)
   	at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
   	at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
   	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   	at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
   	at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
   	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
   	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
   	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
   	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   	at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)
   	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)
   	at com.google.tracing.TraceContext.runInContext(TraceContext.java:703)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330)
   	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)
   	... 1 more
   Caused by: com.google.apphosting.api.DeadlineExceededException: This request (1c53170a9d992a42) started at 2013/03/11 07:36:18.921 UTC and was still executing at 2013/03/11 07:37:18.715 UTC.
   	at java.util.zip.ZipFile.read(Native Method)
   	at java.util.zip.ZipFile.access$1200(ZipFile.java:57)
   	at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:476)
   	at java.util.zip.ZipFile$1.fill(ZipFile.java:259)
   	at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
   	at java.io.DataInputStream.readFully(DataInputStream.java:195)
   	at java.util.jar.JarFile.getBytes(JarFile.java:377)
   	at java.util.jar.JarFile.getManifestFromReference(JarFile.java:176)
   	at java.util.jar.JarFile.getManifest(JarFile.java:163)
   	at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:335)
   	at org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:222)
   	at org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:153)
   	at org.datanucleus.plugin.PluginManager.registerExtensionPoints(PluginManager.java:82)
   	at org.datanucleus.OMFContext.<init>(OMFContext.java:160)
   	at org.datanucleus.OMFContext.<init>(OMFContext.java:141)
   <continued in next message>

2. W2013-03-11 08:37:19.528

   <continued from previous message>
   	at org.datanucleus.ObjectManagerFactoryImpl.initialiseOMFContext(ObjectManagerFactoryImpl.java:144)
   	at org.datanucleus.jdo.JDOPersistenceManagerFactory.initialiseProperties(JDOPersistenceManagerFactory.java:316)
   	at org.datanucleus.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:260)
   	at org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManagerFactory.<init>(DatastoreJDOPersistenceManagerFactory.java:71)
   	at org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManagerFactory.getPersistenceManagerFactory(DatastoreJDOPersistenceManagerFactory.java:126)
   	... 29 more
   com.google.apphosting.runtime.HardDeadlineExceededError: This request (1c53170a9d992a42) started at 2013/03/11 07:36:18.921 UTC and was still executing at 2013/03/11 07:37:19.510 UTC.
   	at com.google.appengine.runtime.Request.process-1c53170a9d992a42(Request.java)
   	at java.io.FileInputStream.readBytes(Native Method)
   	at java.io.FileInputStream.read(FileInputStream.java:231)
   	at sun.misc.Resource.getBytes(Resource.java:124)
   	at java.net.URLClassLoader.defineClass(URLClassLoader.java:273)
   	at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:616)
   	at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
   	at org.mortbay.util.Loader.loadClass(Loader.java:91)
   	at org.mortbay.util.Loader.loadClass(Loader.java:71)
   	at org.mortbay.jetty.servlet.Holder.doStart(Holder.java:73)
   	at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:242)
   	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   	at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
   	at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
   	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
   	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
   	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
   	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   	at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)
   	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)
   	at com.google.tracing.TraceContext.runInContext(TraceContext.java:703)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330)
   	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)
   	at java.lang.Thread.run(Thread.java:679)
   

3. C2013-03-11 08:37:19.535

   Uncaught exception from servlet
   javax.servlet.UnavailableException: Initialization failed.
   	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:228)
   	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194)
   	at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
   	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447)
   	at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)
   	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)
   	at com.google.tracing.TraceContext.runInContext(TraceContext.java:703)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338)
   	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330)
   	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)
   	at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
   	at java.lang.Thread.run(Thread.java:679)
   

4. I2013-03-11 08:37:19.544

   This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.

5. W2013-03-11 08:37:19.544

   A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)

[gafal]

Hi Dough,

Google has (finally) opened an internal ticket:
Please report it here: https://code.google.com/p/googleappengine/issues/detail?id=8901

[doright]

its now 10am.  System un-accessbile. Instances will not start.

my production users who depend on my system have been unable to login all morning.

I have logged a production issue.

I have asked for a refund.

I have crossed my fingers.

I have spent 2.5 years building up customers and creating trust.  Only to have to sit here looking like an idiot again thanks to Google.

Anything else I'm supposed to be doing?  Other than continuing my migration planning.

pathetic Google, really pathetic.

[doright]

10:09 system back up without explanation.

until now instances were unable to start.  

if you can't figure out how to solve this CRITICAL problem, please at least tell me you're looking into it.

[doright]

Dear Google, thought you might be interested to hear a concern direct from a client of mine this morning:

"I'm getting more and more worried about the use of <my app> when high season is here.
I have the feeling that Google platform cannot handle professionnal heavy systems as <my app> is becoming.
Is there anyway to have an harddrive version for us to use as backup when we cannot connect to Google?"

Of course I'd love to implement a full local->central failover solution that seamlessly handles the non-existence of the cloud and restores when re-available.  But I have a feeling that might take a very long time!

- In choosing google I had thought I'd have the sort of availability that would make this sort of infrastructure un-necessary.  How wrong you can be.

Anyway, I thought you might like to know that you're lack of service is percolating down through your app-development community to end-users who are now starting to view Google as a non-professional choice.

regards,

Doug Stoddart

[Rafael Sanches]

The same here. It seems to be related to my app be on datastore. Is anyone having this issue with HDR?

There's no chance I will be able to migrate to HDR in the near future. It would be easier to migrate off appengine.

This issue is happening for the past 5 days already. Maybe they used to cache datastore stuff in memcache, but now memcache has shrunk and this is happening? 

My DAU's already dropped from 130k to 90k. I don't sleep for two days already and I hope the appengine team is doing the same. 

thanks!

[Rafael Sanches]

I will also add that 10 minutes ago all of my instances have restarted. 

It looks like one of those "cronjobs" to restart the server that people used to use in 1996.