Making Django 1.0 work well under Google App Engine
Guido van Rossum
broke with Google App Engine. I'm posting here in the hope that in the
future the Django devs will try to keep in mind the unique environment
that App Engine is and how to be prepared to run under it.
(FYI, my application is Rietveld, a code review tool. See
http://codereview.appspot.com for the live service; there are links to
the source code at the top of the page. It's open source.)
The first big problem was when the gis package was added, and Django
went over 1000 files. I'm not sure what to do about that apart from
simply removing some files from my workspace, but I found a work-
around.
My work-around however means that Django is now run entirely from a
zip file. This is a good idea anyway, it seems to speed up imports and
makes deploying the app much quicker. However there are some places
(not too many in the code that I actually end up using) that don't
work in this case. I'll file a ticket about a specific case.
Another issue is that App Engine doesn't make all standard Python
modules fully available. An example is the imp module -- while imp
itself can be imported, most of the APIs it provides are unavailable.
As a general guideline, in order to at least make it through the
import phase, it's good to use
import imp
.
.
.
... = imp.find_module(...)
instead of
from imp import find_module
.
.
.
... = find_module(...)
since the latter cause the import to fail catastrophically. In general
I like this as a guideline -- is there any reason why Django generally
seems to prefer the early binding form "from X import Y"?
I'd write more but I'll first wait for responses, file some tickets,
and think about what else Django could do to improve its cooperation
with App Engine... because Django is App Engine's favorite framework!
Tom Tobin
> since the latter cause the import to fail catastrophically. In general
> I like this as a guideline -- is there any reason why Django generally
> seems to prefer the early binding form "from X import Y"?
I'd say that code just looks cleaner when you're doing:
from datetime import datetime, timedelta
x = datetime( ... )
y = x + timedelta( ... )
than:
import datetime
x = datetime.datetime( ... )
y = x + datetime.timedelta( ... )
... especially when compounded by hundreds or thousands of lines of code.
Jacob Kaplan-Moss
Thanks for these notes; I'll certainly keep them in mind as we go towards 1.0.
Just so we're all here on the same page, I'm mentally grouping
AppEngine/Django issues with the other "Django on $VM" stuff (even if
AppEngine isn't "really" an alternate VM). That is, we've made
alternate VM support a priority for 1.0 (and I've personally been very
interested in this area), and so we'll do our best to fix any related
issues.
There's a bit of give and take, though -- I'm not really inclined to
start removing files from Django to sneak under the 1000 file limit,
for example! But within reason I'll certainly make issues you find a
priority for 1.0.
On Mon, Aug 11, 2008 at 2:11 PM, Guido van Rossum <gvanr...@gmail.com> wrote:
> My work-around however means that Django is now run entirely from a
> zip file. This is a good idea anyway, it seems to speed up imports and
> makes deploying the app much quicker.
BTW, this isn't just true of AppEngine; making Django run properly
from alternate import mechanisms helps with Jython (i.e. JARs) and
also any sort of "frozen" app using Django. So these are pretty
important issues to work out!
> Is there any reason why Django generally
> seems to prefer the early binding form "from X import Y"?
I personally find it more readable when dealing with "deep" imports
(``from django.contrib.syndication.feeds import Feed``, for example)
but I think I'd agree that as general style stdlib imports ought to be
of the ``import foo`` variety.
Is there somewhere a list of modules that AppEngine modifies? Might be
worth quickly auditing Django's use of those modules to check.
Thanks!
Jacob
Guido van Rossum
> I'd say that code just looks cleaner when you're doing:
>
> from datetime import datetime, timedelta
> x = datetime( ... )
> y = x + timedelta( ... )
>
> than:
>
> import datetime
> x = datetime.datetime( ... )
> y = x + datetime.timedelta( ... )
>
> ... especially when compounded by hundreds or thousands of lines of code.
You would be wrong (unless you got your examples swapped around :-).
For example, it's part of the Google Python style guides that all
imports must import a module, not a class or function from that
module. There are way more classes and functions than there are
modules, so recalling where a particular thing comes from is much
easier if it is prefixed with a module name. Often multiple modules
happen to define things with the same name -- so a reader of the code
doesn't have to go back to the top of the file to see from which
module a given name is imported.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum
<jacob.ka...@gmail.com> wrote:
> Just so we're all here on the same page, I'm mentally grouping
> AppEngine/Django issues with the other "Django on $VM" stuff (even if
> AppEngine isn't "really" an alternate VM). That is, we've made
> alternate VM support a priority for 1.0 (and I've personally been very
> interested in this area), and so we'll do our best to fix any related
> issues.
Sort of. I know that Jython and PyPy are also excited to be able to
say that they support Django (and perhaps IronPython too?). But App
Engine isn't so much a different VM as a different platform. VM-wise
it is CPython; however the platform is missing certain features like
threads, writable files, sockets, and so on.
Running from a zipfile is a bit of a different constraint however --
I'd say this pretty orthogonal to the platform.
> There's a bit of give and take, though -- I'm not really inclined to
> start removing files from Django to sneak under the 1000 file limit,
> for example! But within reason I'll certainly make issues you find a
> priority for 1.0.
And I don't expect you to. By using a zipfile I've got this under control.
> On Mon, Aug 11, 2008 at 2:11 PM, Guido van Rossum <gvanr...@gmail.com> wrote:
>> My work-around however means that Django is now run entirely from a
>> zip file. This is a good idea anyway, it seems to speed up imports and
>> makes deploying the app much quicker.
>
> BTW, this isn't just true of AppEngine; making Django run properly
> from alternate import mechanisms helps with Jython (i.e. JARs) and
> also any sort of "frozen" app using Django. So these are pretty
> important issues to work out!
Great to hear this!
>> Is there any reason why Django generally
>> seems to prefer the early binding form "from X import Y"?
>
> I personally find it more readable when dealing with "deep" imports
> (``from django.contrib.syndication.feeds import Feed``, for example)
> but I think I'd agree that as general style stdlib imports ought to be
> of the ``import foo`` variety.
I'm totally in favor of dropping the packages from the path; but I
personally have grown very fond of Google's style convention of always
importing modules, never classes or functions.
> Is there somewhere a list of modules that AppEngine modifies? Might be
> worth quickly auditing Django's use of those modules to check.
Unfortunately there's no definitive list. There is this:
http://code.google.com/appengine/kb/general.html#libraries
but it is pretty incomplete. You can also scrape much of this info
from the SDK (which is open source). The file
google/appengine/tools/dev_appserver.py has a class
HardenedModulesHook which starts with long lists of modules that are
white-listed or not. Unfortunately one of the problems is that e.g.
blacklisting most of socket means that most of urllib doesn't work,
even though it isn't explicitly blacklisted. This is why it's hard to
get a definitive list.
Another source of information would be the "Google App Engine Helper
for Django": http://code.google.com/appengine/articles/appengine_helper_for_django.html.
Its authors have done some more research into running Django under App
Engine.
Tom Tobin
> Often multiple modules
> happen to define things with the same name -- so a reader of the code
> doesn't have to go back to the top of the file to see from which
> module a given name is imported.
In those cases, I'd go with importing the module, too — partly because
I'd be clobbering the same name if I didn't. ^_^
Readability is what's important; I use whichever style seems sane for
the situation at hand.
Tom Tobin
Yay for replying to myself.
Another problem with importing modules in Django is that *modules*
tend to have the same name: just look at "django.db.models",
"django.contrib.auth.models", "django.contrib.admin.models",
"django.contrib.comments.models" ... I could keep going. ;-) IMHO
this makes module organization a breeze, albeit it makes the "always
import modules" style of Google somewhat painful. (I think I'd gouge
my eyes out if I had to write "django.contrib.auth.models.User" every
time I wanted to touch the User model!) I suppose a workaround for
Google-style might be to alias each import, e.g., "import
django.contrib.auth.models as auth_models".
Tim Chase
>>
>> from datetime import datetime, timedelta
>> x = datetime( ... )
>> y = x + timedelta( ... )
>>
>> than:
>>
>> import datetime
>> x = datetime.datetime( ... )
>> y = x + datetime.timedelta( ... )
>>
>> ... especially when compounded by hundreds or thousands of
lines of code.
> Often multiple modules happen to define things with the same
> name -- so a reader of the code doesn't have to go back to the
> top of the file to see from which module a given name is
> imported.
Fortunately, that crazy python-creator :) allowed for a nice
solution that reduces verbosity while keeping distinguished
namespaces:
import datetime as dt
x = dt.datetime(...)
y = x + dt.timedelta(...)
I've found my willingness to use fully-qualified names is
directly proportional to the length of the module-name. Datetime
gets a lot of shortening from me ("BeautifulSoup" is another one
that usually gets renamed as "bs"), while the "re" module almost
always sails through uneffected in my code.
My personal threshold is around 6 characters for the module-name,
weighted by number of uses. I'm more forgiving of 6-7 character
module namespace-specifications if I only use them in a couple
places in my code. But if I use them abundantly in the code, I
tend to hack them down with "as" for sanity's sake.
-tim
Tom Tobin
<django...@tim.thechases.com> wrote:
> I've found my willingness to use fully-qualified names is
> directly proportional to the length of the module-name. Datetime
> gets a lot of shortening from me ("BeautifulSoup" is another one
> that usually gets renamed as "bs"), while the "re" module almost
> always sails through uneffected in my code.
I think you've hit the nail on the head; I always leave, e.g., "os",
"re", and other pleasantly-short friends alone, whereas the more
verbose denizens of my libraries tend to get the from...import
treatment. Of course, "from re import compile" would leave you one
quite-generic-sounding "compile"; I tend to feel more comfortable
doing from...import when the imported names are fairly descriptive.
Ian Holsman
>
> I'm totally in favor of dropping the packages from the path; but I
> personally have grown very fond of Google's style convention of always
> importing modules, never classes or functions.
>
>
of google?
Guido van Rossum
> it's a bit offtopic, but is Google's style convention documented outside
> of google?
Not yet. Most of it is simply PEP-8 with 2-space indents.
There is talk of publishing the Google style guides (if only so people
contributing to Google open source projects can use the right style)
but IIRC so far only the C++ guide has been outed:
http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml.
Augie Fackler
On Aug 11, 2008, at 6:36 PM, Guido van Rossum wrote:
>
> On Mon, Aug 11, 2008 at 4:18 PM, Ian Holsman <kry...@gmail.com> wrote:
>> it's a bit offtopic, but is Google's style convention documented
>> outside
>> of google?
>
> Not yet. Most of it is simply PEP-8 with 2-space indents.
> There is talk of publishing the Google style guides (if only so people
> contributing to Google open source projects can use the right style)
> but IIRC so far only the C++ guide has been outed:
> http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml.
Actually, if memory serves this is quite similar to the internal style
guide:
http://code.google.com/p/soc/wiki/PythonStyleGuide
Guido van Rossum
Right! I didn't even know about that one. Though the internal one
mutates regularly (usually to clarify ambiguities).