Evidence showing high DeadlineExceededError rates are specific to parts of googles appengine infrastructure

[Tim Hoffman]

Hi

For the last week I have been continually complaining about a high
incidence of DeadlineExceededErrors hitting only one of two instances
running the same code at exactly the same time each day. I have yet
to get any sort of response from google.

As promised I have written up in more detail what we are experiencing,
graphed response times over several dats, presented debugs logs and
generally discussed what I believe is going on.

The full document and supporting data can be viewed here,

http://docs.google.com/Doc?docid=0Ad1LTa6ONStgZGYzcXhjeHBfMTIwZGs2NWd4ZmY&hl=en

It is clear to me that app engine instance performance (specifically
startup times) can be heavily influenced by where your app has been
deployed and what is going on inside google, and this can have more
impact on how your applications perform than your own code.

I hope have done enough to attract some attention from google staff
and would really like to discuss with them their take on my findings
and look at ways of resolving this issue.

Regards

Tim Hoffman

[Brandon Thomson]

Thank you for posting the charts; very interesting to compare the
results between the two apps.

A few months ago I had similar slow instance startup problems whereby
the request timer was often already between 5s and 30s before the
first logging statement at the very top of my app's main py file had
been executed. I posted here and in the issue tracker but never found
out if Google took an action or the problem cleared up on its own.

Needless to say it seems clear that individual applications can have
very slow instance startup (in my estimation caused by queuing because
all the app server slots which the app is eligible to deploy to are
busy) even when the app engine status is itself indicated as "normal".
All the developer can do is ask for a fix.

On Nov 30, 4:12 am, Tim Hoffman <zutes...@gmail.com> wrote:
> Hi
>
> For the last week I have been continually complaining about a high
> incidence of DeadlineExceededErrors hitting only one of two instances
> running the same code at exactly the same time each day.  I have yet
> to get any sort of response from google.
>
> As promised I have written up in more detail what we are experiencing,
> graphed response times over several dats, presented debugs logs and
> generally discussed what I believe is going on.
>
> The full document and supporting data can be viewed here,
>

> http://docs.google.com/Doc?docid=0Ad1LTa6ONStgZGYzcXhjeHBfMTIwZGs2NWd...

[chadwackerman]

I can't imagine a more sensible write-up of the issue.

I'm hoping Google responds. Yes, the forums are full of idiots
who really need to RTFM, but seeing important and sensible issues
like these go unanswered week after week really gives me the willies
about running my serious app on Google's stack.

Like "keep me up at nights" kind of worrying.

Web 2.0 isn't winning any awards for customer service, but at some
point Google is going to have to step up and actually provide some.

Slap some adsense on it, I'm sure it'll pay for everything (ha ha).

On a serious note --

Perhaps a private support area for people with billing enabled?
And perhaps hosted on something other than Google Groups which
frankly kind of stinks? Email is fine, some issues need to remain
anonymous, too. The "community groupthink" model for this isn't
going to work... all of StackOverflow still can't figure out how to
upload a client's file to S3 via AppEngine.

I realize something has to pay for all this, yet Amazon, which
charges more than Google, really stinks at dev support, too.

So I understand where fees might come in. But there's obviously
both cultural and financial issues here. All of Google's products
have ugly management UI, weird missing features due to shoddy
account and accounting tech, and are effectively unsupported.

So I don't have high hopes about seeing this stuff get fixed in
the short term.

But still, I'd happily pay an annual fee for 8-hour support answers.

Also: a tip of the hat to Nick, who does a great job.
But one clever engineer with A+ social skills and a few other folks
spitting out ten posts a year on blogspot.com is not enterprise
support.

On Nov 30, 4:12 am, Tim Hoffman <zutes...@gmail.com> wrote:

> For the last week I have been continually complaining about a high

> incidence of DeadlineExceededErrors..

>
> As promised I have written up in more detail what we are experiencing,
> graphed response times over several dats, presented debugs logs and
> generally discussed what I believe is going on.
>

> http://docs.google.com/Doc?docid=0Ad1LTa6ONStgZGYzcXhjeHBfMTIwZGs2NWd...

[Stephen]

On Nov 30, 9:12 am, Tim Hoffman <zutes...@gmail.com> wrote:
>
> As promised I have written up in more detail what we are experiencing,
> graphed response times over several dats, presented debugs logs and
> generally discussed what I believe is going on.

Have you seen this?

http://groups.google.com/group/google-appengine/browse_thread/thread/b9c9d3ee587d9bde/b454708512acae3a?#b454708512acae3a

It doesn't directly address your problem, which seems to be on
Google's end, but it might help you profile imports in general.

[Chris]

+1

My background is the MS world. I love the concept of GAE. I don't mind
learning Python, messing around with "beta" issues, working with
incomplete documentation etc. All the fun stuff that early adopters
usually have to deal with in our world ;-)
What I'm dissappointed about is the lack of Google involvement in
these groups. As you point out Nick is doing a great job - but other
than that I can't name a single Google employee off the top of my head
after reading every post in here for 3+ months.

It is especially worriesome to see posts related to the core stability
& performance (such as this one) remain un-addressed. The thought of
hanging out there with a real app experiencing very poor performance
day after day without being able to do something makes me go all cold.
And since there is silence from Google on these issues I start to
wonder: Is this a common problem, just hitting a few unlucky app
owners, an intermittent issue that will go away in a week or not, does
it hit everybody once in a while etc. There's supposedly 80K+ GAE
applications out there so it is probably not as bad as it looks from
reading these groups.

/Chris

[Nick Johnson (Google)]

Hi Tim,

Apologies for my delayed response - I've been having email delivery problems, as well as being very busy with conferences.

The problem you describe is a known issue that's being actively worked on; it affects only a small proportion of apps, and as you've observed, only happens at certain times of day. The data you've collected will likely be invaluable, especially given that you've reproduced it with two instances of the same app - thanks!

I've forwarded your data to the core eng team; they, or I, will get back to you very shortly.

-Nick Johnson

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Nick Johnson, Developer Programs Engineer, App Engine
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047

[Tim Hoffman]

Hi Nick

Awesome! Thank very much for the response.

If you need some more information I have a two other instances I have
set up to try and build a reliable production environment.
One is working nicely (as per psc-dev1 instance) the other didn't get
hit with the 11:00pm (google time) DEE's but did to a lesser extent
get hit
high DEE rate 1 hour later.

Let me know if you want those appids as well.

Oh and just for the record.

I would like add my voice to chorus thanking Nick for all of his help
on these forums.

And I would like to state I personally think app engine is a great
platform. I have apps that have been running since August last year
(baon.appspot.com (a full zope3 based stack)), and my own small
business website (www.fishandlily.com.au) which is works so well.
On the basis of what I have seen I convinced various people that the
Swan TAFE (just renamed to Polytechnic West) new web site should
be deployed on google app engine.) The group I work for has
traditionally been a Zope/Plone shop. So I stuck my neck out (and so
did the group)
but I have so much confidence in the app engine platform . Its just
teething issues along the way that we have to help google resolve, and
unfortunately
at times it possibly takes a little more effort than one would like to
be heard. (That and sometimes timing of problems suck ;-).

Once again Nick thanks for the response, and if you need any more
detail, repeatable test cases etc....
Please let me know.

Thanks

Tim Hoffman

On Nov 30, 11:20 pm, "Nick Johnson (Google)" <nick.john...@google.com>
wrote:

> > google-appengi...@googlegroups.com<google-appengine%2Bunsu...@googlegroups.com>
> > .

[johnP]

Tim -

Thanks for pounding the drum. I've been wanting to jump earlier, when
your multiple high-value posts were encountering delivery issues, to
provide you with a word of encouragement. I agree with your
assessment that AppEngine is a game-changer with teething issues.
What I would like to add is that the main problem with Appengine is
not technological, but organizational. It suffers from the age-old
Achilles heel of thousands of Great Technologies: the mistaken belief
that all problems are Technical issues, not People issues.

Why do inferior technologies sometimes succeed while superior ones
fail? Often, because the winning technology has people who answer the
phone. Is Amazon a technology company or a customer service company?

I am writing this to be construed as constructive criticism, because I
need Appengine to succeed, because I placed my bet on Appengine.
Please, Appengine managers: realize that your product offering is
deficient in the customer-service aspect, and that this is a critical
deficiency which can lead to ultimate failure.

johnP

[Greg]

I too bet on Appengine, and so far that bet is paying off handsomely.
My company's latest service is built entirely on Appengine, and has
been very successful - we have hundreds of paying customers, and it is
becoming mission-critical. It's fair to say that Appengine support is
now our biggest concern.

As well as the issue in this thead, it seems that index rebuilding is
still a problem, and I'm guessing there are other issues too. So given
there are potential show-stoppers, anyone providing a commercial
application on Appengine needs support in minutes - not when Nick gets
back from his conference.

And if this isn't resolved before Azure launches, Google will lose the
commercial applications in droves. IT managers are by nature risk-
averse, and a better support system will win against technical
superiority and a track record. Want proof? Just look at how
technically superior Linux doesn't make headway in the corporate
world, unless it's from Red Hat.

Red Hat offers paid support. Google needs to do this too.

I know this would be a big departure for Google - they seem to use the
same faceless group support system for all their properties. That's
fine for the hundreds of "I didn't get my SMS code" and "Ooooops I
just deleted my source code, how can I get it back" enquiries. But if
my company ever hits the rebuilding index problem, I'm going to look
really stupid when I tell our screaming sales manager that I've posted
to a group... and that's really all I can do.

What I'd like is to be able to go to a premium support page, enter my
credit card details for $100, and get a phone number or a chat session
with a real live engineer. I get my issue resolved, Google makes
money, and IT managers get the warm fuzzies about Appengine.

Finally Nick, if you want appengine to have the future it deserves,
please raise this issue with your management. If they don't respond,
just let me know your mobile number - $100 a call OK with you? ;)

[Eli Jones]

It really can't be repeated enough.. This is a "preview release" of appengine.

You're in the wild west... And you gotta deal with it.

No doubt, Google is dedicating all of their resources to developing
the infrastructure. If you need white glove support, you shouldn't be
setting clients up on appengine.

> --
>
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to

> google-appengi...@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
>

--

Sent from my mobile device

[Tim Hoffman]

Just for the record we got hit again 11:00pm (3:00pm GMT+8) today.

I have nearly finished moving all of our data to a new instance, and
am just testing everything.
Then we will cut over to an instance that doesn't exhibit these
problems.

T

On Nov 30, 5:12 pm, Tim Hoffman <zutes...@gmail.com> wrote:
> Hi
>
> For the last week I have been continually complaining about a high
> incidence of DeadlineExceededErrors hitting only one of two instances
> running the same code at exactly the same time each day.  I have yet
> to get any sort of response from google.
>
> As promised I have written up in more detail what we are experiencing,
> graphed response times over several dats, presented debugs logs and
> generally discussed what I believe is going on.
>
> The full document and supporting data can be viewed here,
>

> http://docs.google.com/Doc?docid=0Ad1LTa6ONStgZGYzcXhjeHBfMTIwZGs2NWd...

[johnP]

"""
It really can't be repeated enough.. This is a "preview release" of

appengine...

No doubt, Google is dedicating all of their resources to developing
the infrastructure.
"""

In a preview release, users are expected to communicate what's great
and what sux. That's what we're doing.

Also, you don't know if Google is dedicating any resources to
developing a support infrastructure... because... they have never said
they are! Which is kind of another facet of the overall point.

Appengine is a great technology. Let's hope it becomes a great
product.

johnP

On Nov 30, 6:23 pm, Eli Jones <eli.jo...@gmail.com> wrote:
> It really can't be repeated enough.. This is a "preview release" of appengine.
>
> You're in the wild west... And you gotta deal with it.
>
> No doubt, Google is dedicating all of their resources to developing
> the infrastructure.  If you need white glove support, you shouldn't be
> setting clients up on appengine.
>

[Jason C]

Nick - thank you for posting on this thread.

"The problem you describe is a known issue that's being actively
worked on;
it affects only a small proportion of apps"

We have applications that are impacted by this, though at a different
time of day than Tim. Is it safe to assume that whatever work is being
done will be automatically applied to all affected applications, or
should we be posting our affected application IDs?

j

On Nov 30, 9:20 am, "Nick Johnson (Google)" <nick.john...@google.com>
wrote:

> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> > .