DeadlineExceededError oddities

[Wes]

It started last Monday (8/8): I uploaded a new version of the site
(v18) and started testing before making it the default. I was
receiving DeadlineExceededError and other weirdness like not being
able to upload new versions. Figured it may coincide with some google
downtime, so I waiting another night.

The live site (v17) was still functional at that time. No issues
accessing the same models as the newest version. As of today, the live
site is now giving some DeadlineExceededErrors whereas it *had* been
working for over 30 days without issue.

So, I started thinking maybe it was the data. Perhaps my app had
inefficient code and at I was only now seeing the errors when I
reached some threshold of data records. After doing some testing and
thinking, I decided to deploy the latest version (v18) of my site to
another App Engine Application ID I use for testing. I also uploaded
the production data set.

*** Everything works fine on the alternate Application ID instance
***

Same code. Same data. Different results.


At this point I'm stumped. I have no idea how to debug that.
Everything seems to be identical except for it being deployed to the
alternate App ID.

My next step is to redirect my DNS to the alt App temporarily, but I
would like to get some feedback from Google or others.

What gives ???


Details:

- SDK 1.5.0
- M/S configuration on both Apps


Wes

[Robert Kluin]

Hey Wes,
Under the M/S datastore you will see occasional latency spikes; some
will be large enough to cripple your app for a period of time. I
would suggest migrating to the HR datastore if possible.

You can also use Appstats to profile your application. If you've not
been using it, you'll likely be able to find some optimizations you
can make to improve your apps performance.
http://code.google.com/appengine/docs/python/tools/appstats.html

You've not really given us much information to help us offer
specific suggestions (are the timeouts on read or write, do you do
multiple reads/writes, are you querying, are you writing in batches,
is this in a transaction, are these tasks or user requests, etc...).
If you're not batching your writes, try it. If you are batching, try
smaller batches. If they're writes perhaps you can defer the writes
to a task to minimize the impact on your users.

Robert

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>

[Wes]

>   You've not really given us much information to help us offer
> specific suggestions (are the timeouts on read or write, do you do
> multiple reads/writes, are you querying, are you writing in batches,
> is this in a transaction, are these tasks or user requests, etc...)

I've not given the specifics because I don't know what to give you.
The stack traces are all random. No apparent patterns!
It seems to be failing in django startup** but outside of posting all
of the stack traces, it didn't seem helpful.

It's not evening making it to any models or my code, it's failing
before that.

Here's a couple of the most recent stack traces: http://pastebin.com/cnwEmB6z


** I'm using django-nonrel, but again this has been working fine for
over a year and works fine on another app ID.

On Aug 14, 12:24 am, Robert Kluin <robert.kl...@gmail.com> wrote:
> Hey Wes,
>   Under the M/S datastore you will see occasional latency spikes; some
> will be large enough to cripple your app for a period of time.  I
> would suggest migrating to the HR datastore if possible.
>
>   You can also use Appstats to profile your application. If you've not
> been using it, you'll likely be able to find some optimizations you
> can make to improve your apps performance.
>    http://code.google.com/appengine/docs/python/tools/appstats.html
>
>   You've not really given us much information to help us offer
> specific suggestions (are the timeouts on read or write, do you do
> multiple reads/writes, are you querying, are you writing in batches,
> is this in a transaction, are these tasks or user requests, etc...).
> If you're not batching your writes, try it.  If you are batching, try
> smaller batches.  If they're writes perhaps you can defer the writes
> to a task to minimize the impact on your users.
>
> Robert
>
>
>
>
>
>
>
> On Sun, Aug 14, 2011 at 00:39, Wes <westho...@gmail.com> wrote:
> > It started last Monday (8/8): I uploaded a new version of the site
> > (v18) and started testing before making it the default. I was

> > receivingDeadlineExceededErrorand other weirdness like not being

[Robert Kluin]

Hey Wes,
If it is dying on the initial app setup, you could try to open a
production issue, they'll need your appid to investigate.

Robert

[Tim Hoffman]

I would also do some logging in your main handler before you even get into your django code, that way you 

can see timestamps and you might see where time is getting eaten.  If you have lost 20sec before you even get into your code,

it could indicate your stuck in one of those appengine poor performance backwaters.

Rgds

T

[Wes]

Seems that disabling DEBUG mode has an effect and it works again. Not
sure why the other app ID had no issues with debug being enable, but
this resolves my issue for now. I'll return to this [hopefully] and do
some tracing to determine if there's some noticeable difference
between the two apps.