More then 100 complains and no feedback from Google. Was: Outages?

[Mos]

Problem still exists: High latency and a significant number of requests fail or result in uncontrolled creation of new instances.

GAE customers complain and complain for days.  It seems like many website have to accept that the 500-error is an integral feature of the GAE run-time.
Even worse:  The broken platform leads to increase usage of instances and therefore leads to higher costs for paying customer.

No communication from Google; no feedback on the mailing list; no status updates on the issue tracker.
Instead a always green status:   http://code.google.com/status/appengine     (I think the GAE community needs to develop a crowed sources status page to get a picture what's really is going on)

That makes me crazy.  I saw many platforms and business support forums the last years, but never encountered such an ignorance regarding custom complains.

The only official response I saw, was the "Escalated your issue to the reliability team" in one of the related issues:
http://code.google.com/p/googleappengine/issues/detail?id=7133

So, 'realiability team' what's going on? How is the status?

On Tue, Mar 13, 2012 at 3:22 PM, Mos <mos...@googlemail.com> wrote:

> What is your application id?

krisen-talk    (www.krisentalk.de)

> Feel free to open a production issue

There is already an issue from someone else (following this thread a lot of people are affected):

http://code.google.com/p/googleappengine/issues/detail?id=7133

Johan, what's going on with GAE the last days?  It doesn't feel like a PaaS in production mode.
Perhaps Google should reintroduce the Beta status. ;) 

On Tue, Mar 13, 2012 at 3:10 PM, Johan Euphrosine <pro...@google.com> wrote:

What is your application id?

Feel free to open a production issue, if you want to investigate this offthread:

http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

On Tue, Mar 13, 2012 at 11:44 AM, Mos <mos...@googlemail.com> wrote:

Same thing the last minutes on our app (HRD, Java, Low-Traffic, one instance, no new deployment, simple page just hitting MemCache):

"Request was aborted after waiting too long to attempt to service your request."   -->  User sees 500er

GAE-Team, what is going on the last days?  In my opinion the Google App Engine is unreliable and looks more like a alpha- or beta- cloudenvrionment....

Please Google share you analysis with us.

Cheers
Mos

 

 

2012/3/13 Sébastien Tromp <sebasti...@gmail.com>

Hello,

Same thing here, since around an hour ago:

AppID: fiveorbsgame & fiveorbsgame-test

On Tue, Mar 13, 2012 at 9:59 AM, Miroslav Genov <mge...@gmail.com> wrote:

I'm encountering the same issue with HR app. The spike got started in about ~1 hour from now. 

AppID: cmsevobg

Datastore: HR

On Tuesday, March 13, 2012 10:36:38 AM UTC+2, Richard Watson wrote:

In case you're keeping track of issues thinking it's generally cleared up:

I'm on HR and have noticed higher latencies (couple seconds instead of e.g. 300ms) lately and sometimes higher error rates (a few instead of 0-3). Yesterday over about 6 hours I got a ton of 60-second requests that threw 500's with accompanying messages [1], usually on memcache sets hitting a deadline exceeded.  Also, over the last couple weeks I've been running 3 instances permanently despite sometimes shutting them down manually.  Usually I get by on one just fine with bursts of 2 or 3.  I've noticed that one instance serves the majority of traffic with the other two serving maybe 50 requests over many hours, so the shutdown isn't aggressive enough.

"This request caused a new process to be started for your application..."

+

"A problem was encountered with the process that handled this request, causing it to exit" in the same request.

App-id: 2dumo-hr

On Tuesday, March 13, 2012 10:36:38 AM UTC+2, Richard Watson wrote:

In case you're keeping track of issues thinking it's generally cleared up:

I'm on HR and have noticed higher latencies (couple seconds instead of e.g. 300ms) lately and sometimes higher error rates (a few instead of 0-3). Yesterday over about 6 hours I got a ton of 60-second requests that threw 500's with accompanying messages [1], usually on memcache sets hitting a deadline exceeded.  Also, over the last couple weeks I've been running 3 instances permanently despite sometimes shutting them down manually.  Usually I get by on one just fine with bursts of 2 or 3.  I've noticed that one instance serves the majority of traffic with the other two serving maybe 50 requests over many hours, so the shutdown isn't aggressive enough.

"This request caused a new process to be started for your application..."

+

"A problem was encountered with the process that handled this request, causing it to exit" in the same request.

App-id: 2dumo-hr

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/YUHiXXkGPAgJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Brandon Wirtz]

Nah it is all shiny… And that couple of hours when there are no logs, that was just cause everyone was sleeping…

This is just a pretty picture… It’s not related to this thread. J

 

[alex]

same here.

[Mos]

And here are two that are related to this thread  (application has same traffic, load and configuration for days  --> major GAE problem for more then 24 hours):

--

[Mos]

And here are two that are related to this thread  (application has same traffic, load and configuration for days  --> major GAE problem for more then 24 hours):

Attached....

[alex]

Funny... maybe I'm not getting something here but, where's the issue? It is crystal clear from your graphs that you are being billed for 1 instance only, all the times.

Can you show CPU ms used/sec and Errors/sec graphs?

On Wednesday, March 14, 2012 5:50:04 PM UTC+1, Mos wrote:

And here are two that are related to this thread  (application has same traffic, load and configuration for days  --> major GAE problem for more then 24 hours):

Attached....

On Wed, Mar 14, 2012 at 5:29 PM, Brandon Wirtz <dra...@digerat.com> wrote:

Nah it is all shiny… And that couple of hours when there are no logs, that was just cause everyone was sleeping…

This is just a pretty picture… It’s not related to this thread. J

 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Mos]

Hi Alex,

right, 1 instance is billed, cause this application runs in non payed mode, but the scheduler spans up to four instances (without real need).
In payed mode I assume the green line would not be on 1 instance all the time...  (I'm glad I didn't activate payment yet)

Graphs are attached.

According to Google Analytic and application logging there was similar usages then the days before.  (Around one to two request the minute)
I think we don't need to discuss that something bad is going on the last 24h to 48h hours for many GAE users?
Please check issue tickets and other postings here.....

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/tk8Xo_fvQ04J.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[alex]

Well, I don't personally think that every filed ticket means a real issue. It often happens that the issue is in the code or something else, unrelated.

In fact, in your graphs the API calls CPU is as low as it always was. That means the cause is neither Datastore, nor Memcache or any other service your app might be using (e.g. URLfetch). 

Most likely that means the actual code of your app is consuming the CPU, so the scheduler simply launches more instances because the average latency of your app has increased. 

Even though the traffic to your app is the same as it was before, it very much might be that your code is not covering all edge cases. I don't know, some missing entity property that happend to be there only in the past 24 hours, or maybe you rely on memcached query results that are not there anymore and you just not re-populating (re-caching) some of it. Stuff like that.

[Brandon Wirtz]

Non-payed apps have rules all of their own, they get slow sometimes for no reason. I would not say this is an issue, it just is you are in the lowest tier so if you get squished no one cares….

 

I think QoS/QoE is set this way on purpose.

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Mos
Sent: Wednesday, March 14, 2012 10:16 AM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] More then 100 complains and no feedback from Google. Was: Outages?

 

Hi Alex,

right, 1 instance is billed, cause this application runs in non payed mode, but the scheduler spans up to four instances (without real need).
In payed mode I assume the green line would not be on 1 instance all the time...  (I'm glad I didn't activate payment yet)

Graphs are attached.

According to Google Analytic and application logging there was similar usages then the days before.  (Around one to two request the minute)
I think we don't need to discuss that something bad is going on the last 24h to 48h hours for many GAE users?
Please check issue tickets and other postings here.....

On Wed, Mar 14, 2012 at 5:58 PM, alex <al...@cloudware.it> wrote:

Funny... maybe I'm not getting something here but, where's the issue? It is crystal clear from your graphs that you are being billed for 1 instance only, all the times.

 

Can you show CPU ms used/sec and Errors/sec graphs?

On Wednesday, March 14, 2012 5:50:04 PM UTC+1, Mos wrote:

And here are two that are related to this thread  (application has same traffic, load and configuration for days  --> major GAE problem for more then 24 hours):

Attached....

On Wed, Mar 14, 2012 at 5:29 PM, Brandon Wirtz <dra...@digerat.com> wrote:

Nah it is all shiny… And that couple of hours when there are no logs, that was just cause everyone was sleeping…

This is just a pretty picture… It’s not related to this thread. J

 

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/tk8Xo_fvQ04J.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

 

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[Mos]

Thanks for your thoughts....

> In fact, in your graphs the API calls CPU is as low as it always was

CPU for API calls is not related to latency and timeouts, right? To the contrary:  Waiting for a an API call could lower CPU, doesn't it?
--> It seems not to help for this analysis

>  ... using (e.g. URLfetch). ...  re-populating (re-caching)

No nothing of these.

> Well, I don't personally think that every filed ticket means a real issue ...

Please check the Outage-Thread and the other posting the last hours/days. It's really obvious that something is going on.....

Cheers
Mos

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/9EjTUlYsfnQJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[alex]

"Waiting for a an API call could lower CPU" - well, that depends. If you're running in threadsafe mode enabled then yes, maybe.

Total CPU means exactly what it says: total, i.e. API calls + your app code CPU. Say, if you have a Total CPU of 10000ms and the blue line is at 9900ms, that means only 100ms were used for a service API call. The rest is somewhere very close to your app. 

At least, that's my understanding. 

Another example of a (very unexpected) error I've seen quite often is this: "'ascii' codec can't decode byte 0xe9 in position 0: ordinal not in range(128)". This, again, error in the app code. The developer completely forgot that we have different encodings. The app was working just fine for months, and it just happend that one day a non-US user signed up with a name that contained non ascii chars.

Have you tried to run your app with AppStats and see where it's taking most of the CPU?

[alex]

What Brandon is trying to say here, I think, is that SLA covers only paid apps.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/tk8Xo_fvQ04J.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Ikai Lan (Google)]

Mos,

There have been several responses posted to various threads. I'll go back to those threads and update if there is new information. The disruption to HRD has been posted to the System Status site, and a new maintenance period has been announced on Monday to deal with Master/Slave serving issues (https://groups.google.com/forum/?fromgroups#!forum/google-appengine-downtime-notify).

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/xE-3gLJqg-cJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Enrique]

You are announcing a maintenance of Master/Slave for Monday, but we can do in the meantime? Today one of my apps start creating allot of instances filling the quota and forcing me to change the billing settings to allocate more money.

Enrique

[toonetown]

Yes - we are still seeing problems too - I think it is ridiculous that we aren't hearing anything about the issue from Google...how they are planning on addressing it, etc.

There are over 15 bugs that are similar to http://code.google.com/p/googleappengine/issues/detail?id=7133 - all have been logged in the past 2 weeks, so that tells me that *SOMETHING* is wrong...

-Nathan

 

 

2012/3/13 Sébastien Tromp <sebasti...@gmail.com>

Hello,

AppID: fiveorbsgame & fiveorbsgame-test

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Rick Mangi]

They have responded. Look at the last comment in the thread before yours. I'm not trying to defend them by any means, I think the lack of response from google in general is quite disturbing, but Ikai did reply finally saying that they are working on it.

[toonetown]

That's great!  I mis-read that comment before, and thought it had to do with a billing issue or High-replication datastore.  (Mos's question appeared to be about billing, and the response seemed to indicate something with the HRD - which we are not using HRD).

I'm glad that there is actually some work going in to this...

-Nathan

[johnP]

My app is responding 10x slower than usual, and # of instances is up
about 5-8x. My daily costs are 10x usual, and I am at risk of
exceeding my daily quota.

I am on M/S.

Question:
a. Do I need to expand my quota to cover these additional charges
(shutting down is not an option).
b. Will we get refunded for the excess charges?
c. My appid is thinkwave3; can it be moved back to a legacy serving
platform?

Thank you.

johnP

On Mar 15, 12:48 pm, toonetown <nathan.to...@gmail.com> wrote:
> That's great!  I mis-read that comment before, and thought it had to do
> with a billing issue or High-replication datastore.  (Mos's question
> appeared to be about billing, and the response seemed to indicate something
> with the HRD - which we are not using HRD).
>
> I'm glad that there is actually some work going in to this...
>
> -Nathan
>
>
>
>
>
>
>
> On Thursday, March 15, 2012 1:35:27 PM UTC-6, Rick Mangi wrote:
>
> > They have responded. Look at the last comment in the thread before yours.
> > I'm not trying to defend them by any means, I think the lack of response
> > from google in general is quite disturbing, but Ikai did reply finally
> > saying that they are working on it.
>
> > On Thursday, March 15, 2012 1:59:45 PM UTC-4, toonetown wrote:
>
> >> Yes - we are still seeing problems too - I think it is ridiculous that we
> >> aren't hearing anything about the issue from Google...how they are planning
> >> on addressing it, etc.
>
> >> There are over 15 bugs that are similar to

> >>http://code.google.com/p/googleappengine/issues/detail?id=7133- all

> >> have been logged in the past 2 weeks, so that tells me that *SOMETHING* is
> >> wrong...
>
> >> -Nathan
>
> >> On Wednesday, March 14, 2012 9:42:21 AM UTC-6, Mos wrote:
>
> >>> Problem still exists: High latency and a significant number of requests
> >>> fail or result in uncontrolled creation of new instances.
>
> >>> GAE customers complain and complain for days.  It seems like many
> >>> website have to accept that the 500-error is an integral feature of the GAE
> >>> run-time.
> >>> Even worse:  The broken platform leads to increase usage of instances
> >>> and therefore leads to higher costs for paying customer.
>
> >>> No communication from Google; no feedback on the mailing list; no status
> >>> updates on the issue tracker.
> >>> Instead a always green status:  http://code.google.com/status/appengine
> >>> (I think the GAE community needs to develop a crowed sources status page to
> >>> get a picture what's really is going on)
>
> >>> That makes me crazy.  I saw many platforms and business support forums
> >>> the last years, but never encountered such an ignorance regarding custom
> >>> complains.
>
> >>> The only official response I saw, was the "Escalated your issue to the
> >>> reliability team" in one of the related issues:
> >>>http://code.google.com/p/googleappengine/issues/detail?id=7133
>
> >>> So, 'realiability team' what's going on? How is the status?
>

> >>> On Tue, Mar 13, 2012 at 3:22 PM, Mos <mosa...@googlemail.com> wrote:
>
> >>>> > What is your application id?
>
> >>>> krisen-talk    (www.krisentalk.de)
>
> >>>> > Feel free to open a production issue
>
> >>>> There is already an issue from someone else (following this thread a
> >>>> lot of people are affected):
>
> >>>>http://code.google.com/p/googleappengine/issues/detail?id=7133
>
> >>>> Johan, what's going on with GAE the last days?  It doesn't feel like a
> >>>> PaaS in production mode.
> >>>> Perhaps Google should reintroduce the Beta status. ;)
>
> >>>> On Tue, Mar 13, 2012 at 3:10 PM, Johan Euphrosine <pro...@google.com>wrote:
>
> >>>>> What is your application id?
>
> >>>>> Feel free to open a production issue, if you want to investigate this
> >>>>> offthread:
>

> >>>>>http://code.google.com/p/googleappengine/issues/entry?template=Produc...

>
> >>>>> On Tue, Mar 13, 2012 at 11:44 AM, Mos <mosa...@googlemail.com> wrote:
>
> >>>>>> Same thing the last minutes on our app (HRD, Java, Low-Traffic, one
> >>>>>> instance, no new deployment, simple page just hitting MemCache):
>
> >>>>>> "Request was aborted after waiting too long to attempt to service
> >>>>>> your request."   -->  User sees 500er
>
> >>>>>> GAE-Team, what is going on the last days?  In my opinion the Google
> >>>>>> App Engine is unreliable and looks more like a alpha- or beta-
> >>>>>> cloudenvrionment....
>
> >>>>>> Please Google share you analysis with us.
>
> >>>>>> Cheers
> >>>>>> Mos
>

> >>>>>> 2012/3/13 Sébastien Tromp <sebastien.tr...@gmail.com>

> >>>>>>>> google-a...@googlegroups.com.

> >>>>>>>> To unsubscribe from this group, send email to

> >>>>>>>> google-appengi...@googlegroups.com.

> >>>>>>>> For more options, visit this group at
> >>>>>>>>http://groups.google.com/group/google-appengine?hl=en.
>
> >>>>>>>  --
> >>>>>>> You received this message because you are subscribed to the Google
> >>>>>>> Groups "Google App Engine" group.
> >>>>>>> To post to this group, send email to

> >>>>>>> google-a...@googlegroups.com.

> >>>>>>> To unsubscribe from this group, send email to

> >>>>>>> google-appengi...@googlegroups.com.

> >>>>>>> For more options, visit this group at
> >>>>>>>http://groups.google.com/group/google-appengine?hl=en.
>
> >>>>>>  --
> >>>>>> You received this message because you are subscribed to the Google
> >>>>>> Groups "Google App Engine" group.
> >>>>>> To post to this group, send email to

> >>>>>> google-a...@googlegroups.com.

> >>>>>> To unsubscribe from this group, send email to

> >>>>>> google-appengi...@googlegroups.com.

> >>>>>> For more options, visit this group at
> >>>>>>http://groups.google.com/group/google-appengine?hl=en.
>
> >>>>> --
> >>>>> Johan Euphrosine (proppy)
> >>>>> Developer Programs Engineer
> >>>>> Google Developer Relations
>
> >>>>> --
> >>>>> You received this message because you are subscribed to the Google
> >>>>> Groups "Google App Engine" group.

> >>>>> To post to this group, send email to google-a...@googlegroups.com
> >>>>> .

> >>>>> To unsubscribe from this group, send email to

> >>>>> google-appengi...@googlegroups.com.

> >>>>> For more options, visit this group at
> >>>>>http://groups.google.com/group/google-appengine?hl=en.
>
> > On Thursday, March 15, 2012 1:59:45 PM UTC-4, toonetown wrote:
>
> >> Yes - we are still seeing problems too - I think it is ridiculous that we
> >> aren't hearing anything about the issue from Google...how they are planning
> >> on addressing it, etc.
>
> >> There are over 15 bugs that are similar to

> >>http://code.google.com/p/googleappengine/issues/detail?id=7133- all

> >> have been logged in the past 2 weeks, so that tells me that *SOMETHING* is
> >> wrong...
>
> >> -Nathan
>
> >> On Wednesday, March 14, 2012 9:42:21 AM UTC-6, Mos wrote:
>
> >>> Problem still exists: High latency and a significant number of requests
> >>> fail or result in uncontrolled
>

> ...
>
> read more »

[Brandon Wirtz]

>> b. Will we get refunded for the excess charges?

There is no SLA on M/S
Move to HR

[Brandon Wirtz]

> Thanks alot, Brandon, for the excellent advice.

I am considering making an auto responder:

Dear Google App Engine Patron,

Thank you for your inquiry about why your GAE experience sucks. It appears
from your posting that you are running on Master Slave. We regret to inform
you that any issues you have are related to this decision which you made
early on in your work with GAE and you will continue to regret this decision
until you move to High Replication. If you are unwilling, or unable to move
off of Master / Slave we make the following suggestions:

Make a Funny Custom Error page with lots of pop-ups so that you can monetize
your downtime.
Mark scheduled downtimes on the calendar in advance and claim they are
religious holidays (Google is God, and if God is resting so should you)
Modify your code to work in a read only, mode so that maintenance has
minimal impact.
Put Migration to High Replication on your roadmap, schedule a vacation
during the month the migration is scheduled to take place so it is someone
else's problem.

In the unlikely event that you still believe the Master Slave offers the
best choice for your application we might suggest you visit a psychiatrist,
or a neurologist as clearly your brain is also experiencing some sort of
malfunction. We are aware that Master Slave sounds much sexier than High
Replication. We are also aware that replication is what threatened the SyFy
Stargate:SG1 Universe, but we remind you that was a work of fiction.

Have a great day, and we thank you for your continued patronage.

Google App Engine Volunteer Support (We are in no way affiliated with
Google, and we don't really volunteer any support)

PS
Did you like that marketing thing Google did where they pre-select High
Replication, then on the Master Slave description imply that Master Slave
will cost 1/3 as much, but with downtime and performance issues it actually
costs more? Yeah, we wish we thought of it too. We'd do similar things with
our marketing but Google probably patented it.

[toonetown]

So, explain this to me...is this only a master/slave issue, and those with high replication data store aren't seeing the harddeadlinexceeded exceptions? That wouldn't make sense in our case, as we use absolutely no data store functionality.

I'd be willing to migrate my app if it would make any difference...but I can't see how it would be any different.

[Brandon Wirtz]

> I'd be willing to migrate my app if it would make any difference...but I
can't see how it would be any different.

I tested this. Hello world is down a LOT on MS, and is almost never down on
HR.

Even Static files go down on MS and basically never go down on HR (Except
when Edge or AppsForDomains has issues)

[Jeff Schnitzer]

On Thu, Mar 15, 2012 at 7:36 PM, toonetown <nathan...@gmail.com> wrote:
> So, explain this to me...is this only a master/slave issue, and those with high replication data store aren't seeing the harddeadlinexceeded exceptions? That wouldn't make sense in our case, as we use absolutely no data store functionality.
>
> I'd be willing to migrate my app if it would make any difference...but I can't see how it would be any different.

The difference is that Google actually cares about uptime on the HRD.

Jeff

[Jeff Schnitzer]

On Thu, Mar 15, 2012 at 10:21 AM, Enrique <enm...@gmail.com> wrote:
> You are announcing a maintenance of Master/Slave for Monday, but we can do
> in the meantime? Today one of my apps start creating allot of
> instances filling the quota and forcing me to change the billing settings to
> allocate more money.

1) Put up a "sorry" page.
2) Emergency-migrate to the HRD now.

I'm pretty sure Google feels like the M/S datastore is an albatross
hanging around their neck. I don't work for Google and *I* already
feel like the M/S datastore should be shut down just to improve the
signal-to-noise ratio on this list. They may not say it's deprecated,
but it's just a question of time. Expect support for yesterday's
technology to grow more and more tepid.

I understand the dilema - I have old M/S apps that I haven't migrated
because it would take time I don't have right now. This is a genuine
indication that I don't really care about those old apps. If I cared,
I would do the work and migrate. The writing on the wall has been up
for a long time... the HRD is over a year old. Six months ago you
would have gotten sympathy. Today you get ambivalence. Six months
from now, we (the non-Googlers) are going to be openly hostile. No
point in waiting, just migrate now, even if it puts you offline for a
few hours.

Jeff

[Brandon Wirtz]

> Today you get ambivalence. Six months from now, we (the non-Googlers)
> are going to be openly hostile.

I'm not openly hostile now? I'm going to have to up my game.

[Jeff Schnitzer]

It's hard to appear threatening dressed as a fish.

Jeff

[Brandon Wirtz]

>> It's hard to appear threatening dressed as a fish.

http://www.facebook.com/photo.php?fbid=10150727382517803&set=a.1015014635178
7803.330674.692177802

if you'd move to Phoenix you could dress as a fish as well.

[Richard Watson]

On Friday, March 16, 2012 1:26:54 AM UTC+2, Brandon Wirtz wrote:

We are also aware that replication is what threatened the SyFy
Stargate:SG1 Universe, but we remind you that was a work of fiction.

If you're interested in looking at another work of fiction, I invite you to spend some time on our status page, here:

http://code.google.com/status/appengine 

(Actually the status page seems to have woken up, thanks GAE team.  My guess is before it couldn't load the [x] icons because of the static-image problem! :))

[toonetown]

So - spent the weekend and this morning migrating to HRD...took all of 12 minutes before I started getting the same HardDeadlineExceeded exceptions. :(

BOOOOOO!

-Nathan

[Dennis]

I'm planning to do the same (migrate to HRD), but now my expectation have been lowered...

Any tips on doing the migration??  The docs make it look pretty automatic, but it sounds like you spent a more realistic amount of time doing the migration....

Dennis

[Mos]

The current problems have nothing to do with M/S in my experience .... 

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/uXyyhklSK58J.

[toonetown]

It was actually pretty automatic - but I didn't have any data (I wasn't using a datastore at all)...so it was more an effort of just building and uploading to a different application.  The biggest issue we had was getting billing set up (not a google/appengine issue - a company accounting process issue).

-Nathan

[toonetown]

That's what I had thought as well - since we aren't using datastore at all - but others suggested that there might be more stability from migrating off of M/S - and it actually *has* been more stable.  The error I got may have been a fluke...it's the only one I've had in 24 hours.

-Nathan

On Tuesday, March 20, 2012 5:28:53 AM UTC-6, Mos wrote:

The current problems have nothing to do with M/S in my experience .... 

On Tue, Mar 20, 2012 at 8:02 AM, Dennis <denni...@gmail.com> wrote:

I'm planning to do the same (migrate to HRD), but now my expectation have been lowered...

Any tips on doing the migration??  The docs make it look pretty automatic, but it sounds like you spent a more realistic amount of time doing the migration....

Dennis

On Tuesday, March 20, 2012 2:28:00 AM UTC+8, toonetown wrote:

So - spent the weekend and this morning migrating to HRD...took all of 12 minutes before I started getting the same HardDeadlineExceeded exceptions. :(

BOOOOOO!

-Nathan

On Thursday, March 15, 2012 5:49:02 PM UTC-6, Jeff Schnitzer wrote:

On Thu, Mar 15, 2012 at 7:36 PM, toonetown <nathan...@gmail.com> wrote:
> So, explain this to me...is this only a master/slave issue, and those with high replication data store aren't seeing the harddeadlinexceeded exceptions? That wouldn't make sense in our case, as we use absolutely no data store functionality.
>
> I'd be willing to migrate my app if it would make any difference...but I can't see how it would be any different.

The difference is that Google actually cares about uptime on the HRD.

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/uXyyhklSK58J.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[kusmi]

I'm running on the HR datastore(gae/j), but still experience the same problem... 20% of my requests fail (http 500 errors) and latency skyrocked by a factor of 5 at least...

I have not changed my app in the last month and don't do any fancy stuff...

[toonetown]

What goes around comes around, I guess.... http://appengine.google.com hangs for a long time (about a minute) and then throws up a server error...

How much do you want to bet it's throwing "DeadlineExceededException"?  :)

-Nathan

On Wednesday, March 14, 2012 9:42:21 AM UTC-6, Mos wrote:

Problem still exists: High latency and a significant number of requests fail or result in uncontrolled creation of new instances.

GAE customers complain and complain for days.  It seems like many website have to accept that the 500-error is an integral feature of the GAE run-time.
Even worse:  The broken platform leads to increase usage of instances and therefore leads to higher costs for paying customer.

No communication from Google; no feedback on the mailing list; no status updates on the issue tracker.
Instead a always green status:   http://code.google.com/status/​appengine     (I think the GAE community needs to develop a crowed sources status page to get a picture what's really is going on)

That makes me crazy.  I saw many platforms and business support forums the last years, but never encountered such an ignorance regarding custom complains.

The only official response I saw, was the "Escalated your issue to the reliability team" in one of the related issues:
http://code.google.com/p/​googleappengine/issues/detail?​id=7133

So, 'realiability team' what's going on? How is the status?

On Tue, Mar 13, 2012 at 3:22 PM, Mos <mos...@googlemail.com> wrote:

> What is your application id?

krisen-talk    (www.krisentalk.de)

> Feel free to open a production issue

There is already an issue from someone else (following this thread a lot of people are affected):

http://code.google.com/p/​googleappengine/issues/detail?​id=7133

Johan, what's going on with GAE the last days?  It doesn't feel like a PaaS in production mode.
Perhaps Google should reintroduce the Beta status. ;) 

On Tue, Mar 13, 2012 at 3:10 PM, Johan Euphrosine <pro...@google.com> wrote:

What is your application id?

Feel free to open a production issue, if you want to investigate this offthread:

http://code.google.com/p/​googleappengine/issues/entry?​template=Production%20issue

On Tue, Mar 13, 2012 at 11:44 AM, Mos <mos...@googlemail.com> wrote:

Same thing the last minutes on our app (HRD, Java, Low-Traffic, one instance, no new deployment, simple page just hitting MemCache):

"Request was aborted after waiting too long to attempt to service your request."   -->  User sees 500er

GAE-Team, what is going on the last days?  In my opinion the Google App Engine is unreliable and looks more like a alpha- or beta- cloudenvrionment....

Please Google share you analysis with us.

Cheers
Mos

 

 

2012/3/13 Sébastien Tromp <sebasti...@gmail.com>

Hello,

Same thing here, since around an hour ago:

AppID: fiveorbsgame & fiveorbsgame-test

On Tue, Mar 13, 2012 at 9:59 AM, Miroslav Genov <mge...@gmail.com> wrote:

I'm encountering the same issue with HR app. The spike got started in about ~1 hour from now. 

AppID: cmsevobg

Datastore: HR

On Tuesday, March 13, 2012 10:36:38 AM UTC+2, Richard Watson wrote:

In case you're keeping track of issues thinking it's generally cleared up:

I'm on HR and have noticed higher latencies (couple seconds instead of e.g. 300ms) lately and sometimes higher error rates (a few instead of 0-3). Yesterday over about 6 hours I got a ton of 60-second requests that threw 500's with accompanying messages [1], usually on memcache sets hitting a deadline exceeded.  Also, over the last couple weeks I've been running 3 instances permanently despite sometimes shutting them down manually.  Usually I get by on one just fine with bursts of 2 or 3.  I've noticed that one instance serves the majority of traffic with the other two serving maybe 50 requests over many hours, so the shutdown isn't aggressive enough.

"This request caused a new process to be started for your application..."

+

"A problem was encountered with the process that handled this request, causing it to exit" in the same request.

App-id: 2dumo-hr

On Tuesday, March 13, 2012 10:36:38 AM UTC+2, Richard Watson wrote:

In case you're keeping track of issues thinking it's generally cleared up:

I'm on HR and have noticed higher latencies (couple seconds instead of e.g. 300ms) lately and sometimes higher error rates (a few instead of 0-3). Yesterday over about 6 hours I got a ton of 60-second requests that threw 500's with accompanying messages [1], usually on memcache sets hitting a deadline exceeded.  Also, over the last couple weeks I've been running 3 instances permanently despite sometimes shutting them down manually.  Usually I get by on one just fine with bursts of 2 or 3.  I've noticed that one instance serves the majority of traffic with the other two serving maybe 50 requests over many hours, so the shutdown isn't aggressive enough.

"This request caused a new process to be started for your application..."

+

"A problem was encountered with the process that handled this request, causing it to exit" in the same request.

App-id: 2dumo-hr

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/​msg/google-appengine/-/​YUHiXXkGPAgJ.

To post to this group, send email to google-appengine@googlegroups.​com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@​googlegroups.com.
For more options, visit this group at http://groups.google.com/​group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.​com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@​googlegroups.com.
For more options, visit this group at http://groups.google.com/​group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.​com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@​googlegroups.com.
For more options, visit this group at http://groups.google.com/​group/google-appengine?hl=en.

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[toonetown]

OH - but don't worry....everything is still fine at http://code.google.com/status/appengine - so there isn't any *real* problem</sarcasm>

-Nathan

[Mos]

Same for me, same time; different error-message. I updated: http://code.google.com/p/googleappengine/issues/detail?id=7133

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/4wzFm7wkaIwJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.