Zipping Pylons components on App Engine

[Mike Orr]

I tried zipping some eggs for Pylons under Appengine-Monkey today.,
For some reason the App Engine environment seems to honor directories
listed in a .pth file but not zips listed in a .pth file, so I had to
put the zips individually into sys.path. So that was one step.

The first time around I set zip_ok=true in
lib/python2.5/distutils/distutils.cfg, thinking that would nake
easy_install zip the egg only if it was zip safe. Instead it zipped
everything (except appengine-monkey and setuptools which were already
installed). FormEncode and Simplejson both failed. Formencode
because formencode.api calls get_localedir(); Simplejson because
because _speedups.py wanted to extract its C component (i.e., the
thing that doesn't work on App Engine anyway). Actually, these would
have worked if pkg_resources had been able to extract the components
to a temporary directory, but it couldn't because os.makedirs doesn't
exist in the App Engine world. So the actual error was "NameError:
module has no attribute 'makedirs'".

Then I tried it with zip_ok commented out (that is, set neither false
nor true). This worked except Pylons didn't install Pygments (which
is used by WebError). For some reason that only installs with the
application. I still had to put all the egg paths into sys.path.
Then it worked. But it zipped only four packages: decorator, Tempita,
WebOb, and wsgiref. These are all small packages, which kind of
defeats the purpose of zipping them in the first place. It's the
large packages you want to zip. I gave up at that point.

For reference, these are the biggest hogs in terms of number of files.
(With manual sorting afterwards. *.pyc files were removed prior to
count.)

===
% (for i in $(ls lib/python2.5/site-packages/) ;do echo
lib/python2.5/site-packages/$i; find lib/python2.5/site-packages/$i
-type f | wc -l ; done) >! /tmp/hogs.txt

lib/python2.5/site-packages/Pylons-0.9.7rc2-py2.5.egg
146

llib/python2.5/site-packages/FormEncode-1.0.1-py2.5.egg
65

lib/python2.5/site-packages/Pygments-0.9-py2.5.egg
62

lib/python2.5/site-packages/WebHelpers-0.6.3-py2.5.egg
60

lib/python2.5/site-packages/WebHelpers-0.6.1-py2.5.egg
57

lib/python2.5/site-packages/PasteScript-1.6.3-py2.5.egg
45

lib/python2.5/site-packages/WebError-0.9-py2.5.egg
44

lib/python2.5/site-packages/nose-0.10.3-py2.5.egg
43

lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg
43

[my smallish Pylons application with 2 controllers and 4 templates]
41

lib/python2.5/site-packages/Mako-0.2.2-py2.5.egg
29

lib/python2.5/site-packages/simplejson-2.0.3-py2.5-linux-i686.egg
28

lib/python2.5/site-packages/Beaker-1.0.3-py2.5.egg
23

ib/python2.5/site-packages/PasteDeploy-1.3.2-py2.5.egg
22

lib/python2.5/site-packages/appengine_monkey-0.1dev_r36-py2.5.egg
21

lib/python2.5/site-packages/Routes-1.10.1-py2.5.egg
12
===

A comparision by disk space comes in pretty much the same order.

I couldn't believe Pylons had some many files so I looked at them.
The largest chunk are images for the error pages, application
templates, and tests.

===
% (for i in lib/python2.5/site-packages/Pylons-0.9.7rc2-py2.5.egg/{pylons/media,pylons/templates,tests}
;do echo $i; find $i -type f | wc -l ; done)
lib/python2.5/site-packages/Pylons-0.9.7rc2-py2.5.egg/pylons/media
19
lib/python2.5/site-packages/Pylons-0.9.7rc2-py2.5.egg/pylons/templates
53
lib/python2.5/site-packages/Pylons-0.9.7rc2-py2.5.egg/tests
40
===

So, you can delete the tests in any case. The templates are only
needed for creating new applications, and even then there are template
types like "minimal_project" that nobody uses. I think the media
directory is only for styling error pages, so if you comment out
StatusCodeRedirect in middleware.py (like I always do because I don't
want fancy error pages with graphics), you can eliminate those.

easy_install did install Mako as a directory, even though people have
had success zipping Mako manually.

--
Mike Orr <slugg...@gmail.com>

[Ian Bicking]

Mike Orr wrote:
> For reference, these are the biggest hogs in terms of number of files.
> (With manual sorting afterwards. *.pyc files were removed prior to
> count.)

Incidentally, this opens up an issue: eggs normally contain .pyc files,
but GAE will never load .pyc files so when you upload a zip with those
in it it's a waste. But maybe not an important waste so long as you
stay under 1MB.

--
Ian Bicking : ia...@colorstudy.com : http://blog.ianbicking.org

[Mike Orr]

On Fri, Oct 24, 2008 at 9:32 AM, Ian Bicking <ia...@colorstudy.com> wrote:
>
> Mike Orr wrote:
>> For reference, these are the biggest hogs in terms of number of files.
>> (With manual sorting afterwards. *.pyc files were removed prior to
>> count.)
>
> Incidentally, this opens up an issue: eggs normally contain .pyc files,
> but GAE will never load .pyc files so when you upload a zip with those
> in it it's a waste. But maybe not an important waste so long as you
> stay under 1MB.

I believe the skip_files regex is set to ignore .pyc files when uploading.

But it does bring up a general fact that we're putting a lot of stuff
in eggs that many people will never use, and there's no easy way to
exclude parts of them like Gentoo's USE variable (which avoids
compiling parts of packages; e.g., GTk support, if you have the 'gtk'
use flag turned off). Something more for setuptools to support...

--
Mike Orr <slugg...@gmail.com>

[Ian Bicking]

Mike Orr wrote:
> On Fri, Oct 24, 2008 at 9:32 AM, Ian Bicking <ia...@colorstudy.com> wrote:
>> Mike Orr wrote:
>>> For reference, these are the biggest hogs in terms of number of files.
>>> (With manual sorting afterwards. *.pyc files were removed prior to
>>> count.)
>> Incidentally, this opens up an issue: eggs normally contain .pyc files,
>> but GAE will never load .pyc files so when you upload a zip with those
>> in it it's a waste. But maybe not an important waste so long as you
>> stay under 1MB.
>
> I believe the skip_files regex is set to ignore .pyc files when uploading.

Yes, but it can't skip the .pyc files inside a zip/egg file.

[Ben Bangert]

On Oct 24, 2008, at 2:19 AM, Mike Orr wrote:

> lib/python2.5/site-packages/Pylons-0.9.7rc2-py2.5.egg
> 146

I almost have the latest Pylons zip-safe. I was able to modify
PasteScript so that template directories can be zipped, and I got that
bit working. I don't think the Static URL parser supports zip
resources though, so the error pages bit didn't work. Once the new
PasteScript is released, I'll update Pylons to use the new tuple
syntax for the templates, then its just a matter of fixing the
StaticURLParser, and the Pylons package will be fully zip-safe.

Making your own package zip-safe is a little trickier, as more than a
few things still rely on __file__ and such.

BTW, why is there 2 versions of WebHelpers running there?

- Ben

[Ian Bicking]

Ben Bangert wrote:
> On Oct 24, 2008, at 2:19 AM, Mike Orr wrote:
>
>> lib/python2.5/site-packages/Pylons-0.9.7rc2-py2.5.egg
>> 146
>
> I almost have the latest Pylons zip-safe. I was able to modify
> PasteScript so that template directories can be zipped, and I got that
> bit working. I don't think the Static URL parser supports zip resources
> though, so the error pages bit didn't work. Once the new PasteScript is
> released, I'll update Pylons to use the new tuple syntax for the
> templates, then its just a matter of fixing the StaticURLParser, and the
> Pylons package will be fully zip-safe.

There's a paste.urlparser.PkgResourcesParser that serves via
pkg_resources instead of files.

[Mike Orr]

On Sat, Oct 25, 2008 at 9:58 AM, Ben Bangert <b...@groovie.org> wrote:
> BTW, why is there 2 versions of WebHelpers running there?

Because my application depends on the newer one. :) I installed
Pylons first, then the application, and didn't notice there were two.

--
Mike Orr <slugg...@gmail.com>

[Jorge Vargas]

On Fri, Oct 24, 2008 at 3:19 AM, Mike Orr <slugg...@gmail.com> wrote:
>
> I tried zipping some eggs for Pylons under Appengine-Monkey today.,
> For some reason the App Engine environment seems to honor directories
> listed in a .pth file but not zips listed in a .pth file, so I had to
> put the zips individually into sys.path. So that was one step.
>
>

> A comparision by disk space comes in pretty much the same order.
>
> I couldn't believe Pylons had some many files so I looked at them.
> The largest chunk are images for the error pages, application
> templates, and tests.
>

This happen in Turbogears too back when we first find out about the
1000 files limit, and started working towards getting this working on
appengine (of course that won't happen until pylons can) but some of
TG is already going this way. It was decided to split the package into
two, that is why you currently install tg.devtools and turbogears2.
The idea is that anything related to paster (templates and commands
currently) is only useful in development environments (like the GAE
sdk), but it's just cruft on the deployment (real GAE), this made the
tg package less than 20 files + tests, which can be stripped of a
deployment egg if needed.