patching App Engine Patch

[GAEfan]

Has anyone been able to get App Engine Patch working without importing
Django.zip every time? We are getting far too many
DeadlineExceededErrors.

Unfortunately, AEP has essentially created a custom version of Django,
so it's not really a 'patch' but a different version. Would be nice
to make it a real patch, where it adds to the current Django, instead
of rebuilding it.

We need to find a way to stop importing AEP's Django every time. Any
ideas?

We use user authentication, (standard username, password, not Google
ID)

[Stephen]

Another way to tackle this would be to speed up the process of
importing from zip files:

http://bugs.python.org/issue7216
http://pastebin.com/f6153b5bd

[Stephen]

On Nov 10, 1:16 am, GAEfan <ken...@gmail.com> wrote:

You could try splitting the single, large zipfile into multiple,
smaller zipfiles. Zip all the contrib modules individually, for
example. It may be a win to leave some parts of django unzipped (the
parts which are used on every request, say), but leave less-used parts
zipped. The __path__ variable and similar can be used to stitch the
pieces back together:

http://docs.python.org/tutorial/modules.html#packages-in-multiple-directories

It may also be worth experimenting with the compression ratio of the
zipfile. If i/o is the bottleneck, a higher compression setting may
produce quicker imports. If cpu is the bottleneck, lower compression
may be faster. It is possible to use zero compression in a zipfile, if
needed, in which case it is like tar.

You could also alter the order of the source files within the zipfile.
Experimentally, trace the import order for a popular sample URL, from
a cold start. Add those files, in import order, to the zipfile first,
then append the rest. Hopefully at runtime they will stream out of the
zip in the order in which they are used and this will be more
efficient.

[Waldemar Kornewald]

On Nov 10, 2:16 am, GAEfan <ken...@gmail.com> wrote:
> Has anyone been able to get App Engine Patch working without importing
> Django.zip every time?  We are getting far too many
> DeadlineExceededErrors.

Whenever you don't import Django what do you use instead? Why do you
use aep's main.py, then?

If you do use Django then it won't help to monkey-patch it at runtime
because you still need to load Django before monkey-patching it. I'd
say you can't make the process considerably faster except by modifying
Django to load more of its modules lazily.

I'd love to know what is really causing occasional slow imports on App
Engine and how to prevent it from happening. Does anyone from Google
work on this issue?

Bye,
Waldemar

[theillustratedlife]

use_library('django', '1.1'); will import GAE's Django install into
your app, which I think is what GAEfan is trying to do.

use_library ought to load much faster than out of a ZIP, and is less
likely to result in half-baked GAE instances if you hit a
DeadlineExceededError while your app is still importing modules. From
that standpoint, it would be helpful to take your patches out of
django.zip and separate them into their own module.

Thanks!

[Waldemar Kornewald]

On Nov 11, 5:11 am, theillustratedlife <toonscr...@nvbell.net> wrote:
> use_library('django', '1.1'); will import GAE's Django install into
> your app, which I think is what GAEfan is trying to do.
>
> use_library ought to load much faster than out of a ZIP, and is less
> likely to result in half-baked GAE instances if you hit a
> DeadlineExceededError while your app is still importing modules.  From
> that standpoint, it would be helpful to take your patches out of
> django.zip and separate them into their own module.

I already discussed this option with Guido and use_library is not
faster. You still have to import Django and that's what causes the
deadline errors. So, separating the patches won't help. In fact, it
would introduce huge maintainability problems. The real solution is to
1) fix the root cause of the deadline errors
2) make Django officially support App Engine
3) always preload the latest Django release (as far as this is
possible without requiring to load user code like the settings; or
maybe fixing (1) makes this obsolete)

Points (1) and (3) must be done by Google. Right now, we're working on
(2), but I can already say that this won't get finished unless we get
real contributions (we can help with getting you started if anyone is
interested).

Bye,
Waldemar

[theillustratedlife]

That's unfortunate.

I have been using use_library's Django with the App Engine Helper, and
a friend has been using App Engine Patch. I figured since he gets
Deadline errors all the time, and I never do, that use_library might
be involved.

Good luck!

[Waldemar Kornewald]

Did you also take overall code size and number of loaded modules (in
particular on first load) into account? If you have 100% the same code
and the only difference is use_library vs aep then this might indicate
an io bottleneck in case the integrated Django package is loaded from
a different volume/network than the user code.

[Baptiste Lepilleur]

I've put together a small script to collect the import tree and time (I hadn't had time to clean it yet). You can get it there:
http://bazaar.launchpad.net/~pylons-gae/pylons-gae/trunk/annotate/head%3A/importtimer.py

It's just a decorator around the built-in __import__. I found this pretty useful to find and cut down non desired dependency of existing web framework.

You can get an example of the csv output there (may not work since I'm messing around with this app):
http://pylons-gae.appspot.com/hellomako/1/import.csv
or
http://pylons-gae.appspot.com/hellomako/1/import.txt

2009/11/10 Stephen <sde...@gmail.com>