app instance is completely broken, DeadlineExceededError

[Kenneth]

Hi,

I appear to have completely broken my application instance. If I
upload the application into another instance it runs correctly, but
I'm only seeing deadline exceeded errors in the instance that has my
data. On the client side I just see a blank page after 30 seconds.

I've cleared out my cookies, pulled out all my middleware, pulled out
all my urls except my home page and changed that to:

def home(request):
return http.HttpResponse('hello world')

If anyone has any suggestions on how to debug this problem I'd love to
hear them.

The exceptions change every time but here's a sample:

#

1.
09-02 10:20AM 01.322 / 500 31421ms 1315cpu_ms 0kb Mozilla/5.0
(Windows; U; Windows NT 6.1; en-GB; rv:1.9.1.2) Gecko/20090729 Firefox/
3.5.2 (.NET CLR 3.5.30729),gzip(gfe)
See details

- - [02/Sep/2009:10:20:32 -0700] "GET / HTTP/1.1" 500 0 -
"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.1.2) Gecko/
20090729 Firefox/3.5.2 (.NET CLR 3.5.30729),gzip(gfe)"
"easyschooladmin.appspot.com"

2.
E 09-02 10:20AM 32.706

<class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
File "/base/data/home/apps/easyschooladmin/
1.336057423874075874/django_bootstrap.py", line 64, in <module>
main()
File "/base/data/home/apps/easyschooladmin/
1.336057423874075874/django_bootstrap.py", line 61, in main
util.run_wsgi_app(application)
File "/base/python_lib/versions/1/google/appengine/ext/webapp/
util.py", line 76, in run_wsgi_app
result = application(env, _start_response)
File "/base/python_lib/versions/third_party/django-1.0/django/
core/handlers/wsgi.py", line 239, in __call__
response = self.get_response(request)
File "/base/python_lib/versions/third_party/django-1.0/django/
core/handlers/base.py", line 128, in get_response
return self.handle_uncaught_exception(request, resolver,
exc_info)
File "/base/python_lib/versions/third_party/django-1.0/django/
core/handlers/base.py", line 141, in handle_uncaught_exception
from django.core.mail import mail_admins
File "/base/python_lib/versions/third_party/django-1.0/django/
core/mail.py", line 5, in <module>
import mimetypes

#

1.
09-02 10:44AM 24.212 / 500 29251ms 1455cpu_ms 0kb Mozilla/4.0
(compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC
6.0; Tablet PC 2.0),gzip(gfe)
See details

- - [02/Sep/2009:10:44:53 -0700] "GET / HTTP/1.1" 500 0 -
"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0;
SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729;
Media Center PC 6.0; Tablet PC 2.0),gzip(gfe)"
"easyschooladmin.appspot.com"

2.
E 09-02 10:44AM 53.447

<class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
File "/base/data/home/apps/easyschooladmin/
1.336057955581180609/django_bootstrap.py", line 68, in <module>
main()
File "/base/data/home/apps/easyschooladmin/
1.336057955581180609/django_bootstrap.py", line 65, in main
util.run_wsgi_app(application)
File "/base/python_lib/versions/1/google/appengine/ext/webapp/
util.py", line 76, in run_wsgi_app
result = application(env, _start_response)
File "/base/python_lib/versions/third_party/django-1.0/django/
core/handlers/wsgi.py", line 228, in __call__
self.load_middleware()
File "/base/python_lib/versions/third_party/django-1.0/django/
core/handlers/base.py", line 38, in load_middleware
mod = __import__(mw_module, {}, {}, [''])
File "/base/data/home/apps/easyschooladmin/
1.336057955581180609/security.py", line 6, in <module>
from django import shortcuts
File "/base/python_lib/versions/third_party/django-1.0/django/
shortcuts/__init__.py", line 9, in <module>
from django.db.models.manager import Manager
File "/base/python_lib/versions/third_party/django-1.0/django/
db/models/__init__.py", line 5, in <module>
from django.db.models.query import Q
File "/base/python_lib/versions/third_party/django-1.0/django/
db/models/query.py", line 7, in <module>
from django.db.models.fields import DateField
File "/base/python_lib/versions/third_party/django-1.0/django/
db/models/fields/__init__.py", line 16, in <module>
from django import forms
File "/base/python_lib/versions/third_party/django-1.0/django/
forms/__init__.py", line 14, in <module>
from widgets import *
File "/base/python_lib/versions/third_party/django-1.0/django/
forms/widgets.py", line 21, in <module>
from urlparse import urljoin



Thanks very much.

[Koen Bok]

I'm seeing the same with our app.

[Kenneth]

Yet the system status says everythng is fine. I guess all the
engineers are away fixing gmail. :-)

Google, how am I supposed to run a business on this platform?

[Kenneth]

It seems to be back. I sort feel like a network engineer pulling out
all the patch cables when the power was out. Oh well.

It would be great if someone from google could comment on this.

[Brandon N. Wirtz]

> Google, how am I supposed to run a business on this platform?

That is why there is versioning. Upload to one version, test that it works,
push the button to change the default version.

Always have two copies of your application, so you don't risk corrupting
your live data.

Make sure you only have one guy with the rights to deploy to your primary
app so that a momentary lapse in Yaml configuration doesn't bork your site.

"With Great power comes great responsibility" :-) so make sure that when
leveraging the Power of the Goog, you don't break the baby, cause it can be
a real pain if your have 100 megs of application and screw it up, cause it's
kind of slow to put back.

[Kenneth]

Actually I did have two versions of my application. Both of them went
down, including the production version that hadn't been updated in
nearly a week. It was never a problem with my code; the problem is
quite clearly with google's infrastructure, although I'm willing to be
proven wrong.

I uploaded my code (both production and latest) to a new application
instance by changing the app.yaml file and it ran perfectly. I pulled
out all datastore touches from the code on the broken instance to see
if that was the issue, it wasn't.

If you look at the stack traces they appear to be happening during
module import, not during my application running.

I do appreciate the input though, it is good advice.

[Kenneth]

This is happening again. I even made my application a paid
application this morning. Everything was working fine and then around
8 am PST everything starting blowing up again.

Maybe it has something to with using django 1.0 with use_library
('django', '1.0')?

Please help.

I'm seeing this on the client side:

Error: Server Error
The server encountered an error and could not complete your request.
If the problem persists, please report your problem and mention this
error message and the query that caused it.

And this in the logs:

#

1.
09-03 08:38AM 01.904 /class/list 500 36197ms 1185cpu_ms
38api_cpu_ms 0kb Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1;

Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR
3.0.30729; Media Center PC 6.0; Tablet PC 2.0),gzip(gfe)
See details

- - [03/Sep/2009:08:38:38 -0700] "GET /class/list HTTP/1.1" 500
0 "http://easyschooladmin.appspot.com/class/list" "Mozilla/4.0

(compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC
6.0; Tablet PC 2.0),gzip(gfe)" "easyschooladmin.appspot.com"

2.

E 09-03 08:38AM 38.069

<class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
File "/base/data/home/apps/easyschooladmin/

1.336079106509095336/django_bootstrap.py", line 64, in <module>
main()
File "/base/data/home/apps/easyschooladmin/
1.336079106509095336/django_bootstrap.py", line 61, in main

util.run_wsgi_app(application)
File "/base/python_lib/versions/1/google/appengine/ext/webapp/
util.py", line 76, in run_wsgi_app
result = application(env, _start_response)
File "/base/python_lib/versions/third_party/django-1.0/django/
core/handlers/wsgi.py", line 239, in __call__
response = self.get_response(request)
File "/base/python_lib/versions/third_party/django-1.0/django/
core/handlers/base.py", line 128, in get_response
return self.handle_uncaught_exception(request, resolver,
exc_info)
File "/base/python_lib/versions/third_party/django-1.0/django/
core/handlers/base.py", line 141, in handle_uncaught_exception
from django.core.mail import mail_admins
File "/base/python_lib/versions/third_party/django-1.0/django/
core/mail.py", line 5, in <module>
import mimetypes