DeadlineExceededError Errors for 5% of all Requests Since 9-2 4AM

[Koen Bok]

Since 2 September around 4PM I get random DeadlineExceededErrors for
around 5% of all my requests. The tracebacks are pretty random, but
almost always have to do something with I/O (read django template,
stat file, import module). I can hardly imagine I'm the only one
seeing this, and while annoying at worst I hoped a Googler could
confirm I'm not crazy and maybe even provide some insight in what's
happening.

[bvelasquez]

/project/userlist/agdqYWNvYi02cg4LEgdQcm9qZWN0GIUQDA 200 22522ms
1299cpu_ms 94api_cpu_ms

Currently, my response times are horrible. This request usually takes
200ms or less, and I optimized it to use memcache as much as
possible. Now, this simple request is taking 22 seconds! This seems
to be happening on requests that access the datastore.

I've noticed this performance change since the maintenance. I don't
think you are crazy.

Barry

[Kenneth]

Something has gone really wrong with app engine.

Here's my post:
http://groups.google.com/group/google-appengine/browse_thread/thread/5373a2efe1a973bb

I would also suggest that people star this issue relating to giving us
an sla, at least for paid accounts: http://code.google.com/p/googleappengine/issues/detail?id=501

Lastly I would suggest to google that if they are not aware of this
issue then their monitoring tools need to be improved. They need to
know the overall 500 error rate for appengine applications. If they
do, then they could easily see that this has spiked recently. There
will always be a certain background level of 500 errors but a sudden
increase needs be proactively investigated.

I know that google is looking into this, but at what level I don't
know. I'm thinking it might be related to this issue, loading of an
instance takes too long: http://code.google.com/p/googleappengine/issues/detail?id=1695
that's pure speculation on my part. It might be that a single server
or cluster of servers are acting up and when a request goes to that
group we see the problem.

[Koen Bok]

I'm glad you guys can back me up on this :-)

> I know that google is looking into this, but at what level I don't
> know.

How do you know? I mean, I would assume it but they're mostly pretty
open about errors and I haven't read anything official on this.

[bvelasquez]

I received my first DeadlineExceeded after many responses taking 25
seconds or more at http://www.jacobsix.com (jacob-6) These responses
normally take 200-300ms to complete. I'm sure there is a reasonable
explanation for this behavior given the complexity of the Google
network. I just wish someone could provide me some insight on the
reasons so I understand. Maybe there is something that can be done
further on my end. All seems ok again now, responses taking 200-300ms
again across the site.

[johnP]

Just wanted to add my 2c that this is problem started affecting me on
the evening of 9/04. I have received 7 such errors - all limited to a
group of users located in Israel - occurring on various pages.

To Summarize:
+1 on 'New error for me too'.
It seems to be concentrated geographically.
It occurs on various pages - and often while loading modules.

johnP

On Sep 8, 9:31 am, bvelasquez <bvelasq...@gmail.com> wrote:
> I received my first DeadlineExceeded after many responses taking 25

> seconds or more athttp://www.jacobsix.com(jacob-6)  These responses