Pylons on Google App Engine
Brian O'Connor
--
Brian O'Connor
Mike Orr
I was doing research on this for a book I was writing, but I got so
frustrated at the limitations and bugs and sudden changes in App
Engine that I gave up. I turned the book over to another author, and
I don't know whether he's going to keep the Pylons focus. What's left
is a howto I wrote re installing Pylons with
Appengine-Monkey/appengine-boot.
http://wiki.pylonshq.com/display/pylonscookbook/Creating+a+Pylons+application+for+Google+App+Engine
Since then, several things have happened:
1) Ian wrote an appengine_homedir script in Appengine-Monkey, with a
better application structure.
2) Google added some of the Appengine-Monkey code to App Engine, so
now it runs Setuptools without the monkey.
3) Google raised the file limit from 1000 to 3000. Pylons and a
default app is around 950 files, so this gives more breathing room
without zipping packages.
4) Beaker worked with Datastore, then it didn't. but I think it's working now.
5) The interactive traceback was working in the development
environment, then somebody said it wasn't. I haven't figured out what
causes it to work in some cases but not others.
The best advice now is to use the appengine_homedir script. The howto
is still useful but you'll have to make allowances for the differences
between appengine_boot and appengine_homedir. You can disable the
monkey by commenting out the "import appengine_monkey" line in
paste-deploy.py . It needs more testing before the howto can be
updated.
In the farther future I'd like to see:
- Extract appengine_homedir to a separate distribution without the monkey.
- Make a simple tarball installer containing Pylons, the dependencies, and
appengine_homedir.
- A tarball containing a default application would be ideal, but I'm
not sure it's feasable given Virtualenv's symlinks to the system
Python, which are different on each system.
- Better distribution of the uploadable and non-uploadable files. The
appengine_homedir structure helps by having an application
subdirectory which is uploaded, and the bin scripts are installed
outside that. But Nose gets installed inside even though it's not
useful on the server. Setuptools doesn't know about inside vs
outside; it treats all Pylons dependencies identically. You can get
around this with command-line options, but that would complicate the
howto or have to be built into the installer.
- Better installation for simplejson. The C speedups are not useful
on the server, but it's hard to get Setuptools to not try to compile
it anyway, and to fail on some systems.
- Use pip. This has a more enlightened installer, and it can zip and
unzip installed packages. But given that you have to easy_install
pip, it would add a complication to the howto, which is especially
cumbersome to explain to new Python users. And given that you never
install packages on the server anyway, perhaps it's easier to just use
easy_install.
But somebody else will have to do these.
--
Mike Orr <slugg...@gmail.com>
Matt Feifarek
The best advice now is to use the appengine_homedir script. The howto
is still useful but you'll have to make allowances for the differences
between appengine_boot and appengine_homedir. You can disable the
I'd like to use the most recommended way, but am having trouble figuring it out.
Mike, I've found your bitbucket repo, and am trying to get the docs here:
http://bitbucket.org/sluggo/appengine-homedir/src/tip/docs/pylons.txt
That seems like some kind of doctest file, but if I try and run it with nose, I get "ran 0 tests". If I run it in python (either from the relevant virtualenv or from stock python, I get a traceback... so I'm now guessing this is not the intent.
I can't easily read the instructions in pylons.txt because they're interwoven into code... and because with GAE labels become very tricky very quickly ("{app}" is used as a GAE launcher, a pylons app, and a VE in some docs).
So, how can I 'read' the docs for this? Or, does anyone have an updated state-of-the-art for Pylons on GAE?
Thanks!
Mike Orr
Those are Ian's notes; I didn't modify them. Actually, I've never
seen them before. I went by what Ian put on the web in the
appengine-monkey project.
The only thing I did to the appengine-homedir project was to remove
the monkey code. I was going to make it into a more complete
installer but I never got that far.
> So, how can I 'read' the docs for this? Or, does anyone have an updated
> state-of-the-art for Pylons on GAE?
None has been mentioned on the list. The thread you found is the
latest. If you have time to figure out the easiest install procedure
for Pylons on App Engine and to update the howto, it would be a
contribution to Pylons. There's a redirect set up for the latest
instructions:
http://pylonshq.com/appengine
but you've probably seen it already.
--
Mike Orr <slugg...@gmail.com>
Matt Feifarek
None has been mentioned on the list. The thread you found is the
latest. If you have time to figure out the easiest install procedure
for Pylons on App Engine and to update the howto, it would be a
contribution to Pylons. There's a redirect set up for the latest
instructions:
http://pylonshq.com/appengine
but you've probably seen it already.
Brian O'Connor
--
Brian O'Connor
Matt Feifarek
latest. If you have time to figure out the easiest install procedureNone has been mentioned on the list. The thread you found is the
for Pylons on App Engine and to update the howto, it would be a
contribution to Pylons. There's a redirect set up for the latest
instructions:
http://pylonshq.com/appengine
but you've probably seen it already.
I've added a new page to the Wiki:
http://wiki.pylonshq.com/pages/viewpage.action?pageId=17465426
(Wikis suck. It totally roasted all my formatting from rst/html. Editing yet-another-markup in a textarea is painful. Argh.)
It would be great if a few people looked it over; I expect that in particular some of the syntax is now wrong (over-roasted by the import).
Also, I probably have some terminology wrong: I'm not sure what the meta files like MANIFEST.in and ez_setup.py actually are... are they egg files? Paster files? Setuptools? I don't know what to properly call them when I talk about deleting them.
Some questions that I still have and can't answer:
1. Can one delete the *.egg-info directories in the "app/lib" portion of a GAE tree, or does that metadata get used in the standard operating of python (one of the reasons I HATE all of this packaging stuff; we have "webob" just where it's supposed to be, then the cruft "WebOb-0.9.6.1-py2.5.egg-info" (which is NOT a package) right next to it. But I digress...
2. There are still files in an "appengine-homedir" from monkey: ./app/lib/python/appengine_monkey_files. I don't know what they're for (if anything). I removed them in my experimentation, seems ok so far.
3. Much of anything about zipped / unzipped packages.
Mike Orr
> On Wed, Oct 7, 2009 at 1:27 AM, Mike Orr <slugg...@gmail.com> wrote:
>>
>> None has been mentioned on the list. The thread you found is the
>> latest. If you have time to figure out the easiest install procedure
>> for Pylons on App Engine and to update the howto, it would be a
>> contribution to Pylons. There's a redirect set up for the latest
>> instructions:
>> http://pylonshq.com/appengine
>> but you've probably seen it already.
>
> I've added a new page to the Wiki:
> http://wiki.pylonshq.com/pages/viewpage.action?pageId=17465426
>
> (Wikis suck. It totally roasted all my formatting from rst/html. Editing
> yet-another-markup in a textarea is painful. Argh.)
>
> It would be great if a few people looked it over; I expect that in
> particular some of the syntax is now wrong (over-roasted by the import).
It looks good. And better organized than I had. Feel free to link to
it on the other page, or to paste the whole thing into that page. We
need to keep the redirect pointing to the latest instructions, and
it's easier to modify the page in place than to ask Ben to change the
redirect.
The first link under "other materials" is not a link.
Change the "Thanks to Ian Bicking and Mike Orr" to "Thanks to Ian
Bicking". He wrote all the code. All my fork does is delete some
obsolete portions. My main contribution was the older howto on the
Pylons site. Besides that, I've had extensive discussions with a few
others on how we might streamline the install process for GAE, which
has not been written down. I can share that with anybody who wants to
take over maintenance of appengine-homedir and the Pylons/AppEngine
subproject.
> Also, I probably have some terminology wrong: I'm not sure what the meta
> files like MANIFEST.in and ez_setup.py actually are... are they egg files?
> Paster files? Setuptools? I don't know what to properly call them when I
> talk about deleting them.
The egg-info directory contains package metadata. These are necessary
because Paste uses various runtime services provided by Setuptools,
which depend on the metadata for package introspection and entry
points.
ez_setup.py is unnecessary, and I think MANIFEST.in is too.
ez_setup.py is a script that downloads and installs Setuptools. It's
customary to include it in packages whose setup.py depend on
Setuptoos, as Pylons applications do. MANIFEST.in is used by setup.py
to build the egg-info metadata. Neither of these are needed on the
server, which never installs anything.
> Some questions that I still have and can't answer:
>
> 1. Can one delete the *.egg-info directories in the "app/lib" portion of a
> GAE tree, or does that metadata get used in the standard operating of python
> (one of the reasons I HATE all of this packaging stuff; we have "webob" just
> where it's supposed to be, then the cruft "WebOb-0.9.6.1-py2.5.egg-info"
> (which is NOT a package) right next to it. But I digress...
The metadata has to go somewhere, and there are two standard locations
for it in Python. easy_install does it one way, and pip does it the
other way. Pip's way has just one directory on sys.path, like in the
olden days. Easy_install's way puts every friggin' package directory
on sys.path using *.pth files. Besides being ugly, this is one more
point of failure if the path doesn't get set right. As has happened
to me before on App Engine, when it ignored *.pth files sometimes.
I wish Paste did not depend on Setuptools and entry points, but it
does, and we're not going to get away from that until we write another
framework. (Which the Pype framework might be someday.)
> 2. There are still files in an "appengine-homedir" from monkey:
> ./app/lib/python/appengine_monkey_files. I don't know what they're for (if
> anything). I removed them in my experimentation, seems ok so far.
Those are probably things I missed when I was deleting the monkey code.
> 3. Much of anything about zipped / unzipped packages.
Google raised the quotas significantly after we bitched long enough
about it. I think it's 3000 files now, which is enough for a
good-sized Pylons application wiithout zipping. Before it was 1000
files, which just barely fit Pylons. The issue of course is that no
other runtime environment imposes a file quantity restriction, so the
packages were not designed to economize files. And more than half of
Pylons and Routes is backward compatibility code, which is needed now
but will be gone by Pylons 1.0.
--
Mike Orr <slugg...@gmail.com>
Mike Orr
> Also, I probably have some terminology wrong: I'm not sure what the meta
> files like MANIFEST.in and ez_setup.py actually are... are they egg files?
> Paster files? Setuptools? I don't know what to properly call them when I
> talk about deleting them.
The complete answer is at peak.telecommunity.com. A more concise
answer is the PyCon talk Ian gave in 2006.
http://ianbicking.org/docs/pycon2006/plugins.html
Click on each page to go to the next. On some pages you may have to
click several times to get it to change.
An egg is a Python package accompanied by its metadata. An egg can
contain more than one package, but always one metadata. There are two
file formats for eggs:
site-packages/
Foo-1.0-*.egg/
EGG-INFO/
foo/
__init__.py
easy-install.pth
In this case, the Foo directory is the egg. easy-install.pth contains
a line that adds the Foo directory to sys.path. The package directory
can be zipped and it will behave exactly the same, as long as the zip
file ends in *.egg instead of *.zip.
The other format is the one pip uses:
site-packages/
foo/
Foo-1.0-*.egg-info/
In this case, the egg spans two sibling directories. It's impossible
to tell just by looking at site-packages which packages belong to the
egg. You'd have to introspect the metadata to find out that. But
this format is arguably simpler. If you tell pip to zip a package, it
produces the zip format above (I think). If you tell pip to unzip a
package, it reconstructs this multi-directory format.
Zipped eggs can be uploaded to PyPI. Easy_install can install them;
pip can't. Pip can only install from source tarballs. Zipped eggs
were seen as an advance in distribution technique, but that seems less
true now because eggs often don't include the documentation and
examples necessary to learn how to use the package. The source
tarball is just as easy to distribute, so why not stick with that?
But zipped eggs have an advantage if the package contains C
extensions: eggs are precompiled, which is important on Windows where
users don't usually have compilers or know how to use them.
--
Mike Orr <slugg...@gmail.com>
Matt Feifarek
Feel free to link to it on the other page, or to paste the whole thing into that page.
Will-do. And I'll fix the error you pointed out, change the thanks attribution, etc.
Pylons site. Besides that, I've had extensive discussions with a few
others on how we might streamline the install process for GAE, which
has not been written down. I can share that with anybody who wants to
take over maintenance of appengine-homedir and the Pylons/AppEngine
subproject.
I'm not sure I'm up to that task (I use Pylons a lot, but kindof "tune-out" the parts about PasteDeply/Script, SetupTools, etc. I don't really understand them, so I'm not sure I could tweak an effective installer, but I'll look into appengine-homedir and see if I can grok it).
The egg-info directory contains package metadata. These are necessary
because Paste uses various runtime services provided by Setuptools,
which depend on the metadata for package introspection and entry
points.
Yes, I understand, but does GAE use Paste? I think my ignorance has bled over... I don't really get setuptools, but I also don't get how GAE is different. Or the same.
This leads to two questions:
1. Can we simply delete that stuff ? It would make it harder to manage the app going forward.
2. Can we keep it for local management with pip et al, but NOT send it to GAE when we update? IE, does GAE use entry points?
The metadata has to go somewhere, and there are two standard locations
for it in Python. easy_install does it one way, and pip does it the
I guess that's the question: does it actually have to go somewhere? If I do "import simplejson" I don't need the egg-info, of course. Does Pylons use entrypoints a lot internally, or simple good-ole imports?
I wish Paste did not depend on Setuptools and entry points, but it
does, and we're not going to get away from that until we write another
framework. (Which the Pype framework might be someday.)
Yes, I keep hearing about this Pype. It's starting to sound mythical and religious. I look forward to it solving all of our problems! ;-) Are you guys discussing these ideas somewhere, or just amongst yourselves at this point? I'd be interested to read the discourse so far.
Thanks, Mike!
Matt Feifarek
An egg is a Python package accompanied by its metadata. An egg can
contain more than one package, but always one metadata. There are two
file formats for eggs:
...
Thanks for all of that. It helps. A lot. I'll look at the link to Ian's talk (this year was my first PyCon, so didn't see that in 06)
I did notice something:
this format is arguably simpler. If you tell pip to zip a package, it
produces the zip format above (I think). If you tell pip to unzip a
package, it reconstructs this multi-directory format.
Assume you have a virtualenv somewhere (mine is ~/boopah):
Searching for pip
Reading http://pypi.python.org/simple/pip/
Reading http://pip.openplans.org
Best match: pip 0.5
Downloading http://pypi.python.org/packages/source/p/pip/pip-0.5.tar.gz#md5=63eed8673e40628534cc0aa9c98e8f3d
Processing pip-0.5.tar.gz
Running pip-0.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-SE9kq2/pip-0.5/egg-dist-tmp-1qBPg7
warning: no previously-included files matching '*.txt' found under directory 'docs/_build'
no previously-included directories found matching 'docs/_build/_sources'
zip_safe flag not set; analyzing archive contents...
pip: module references __file__
Adding pip 0.5 to easy-install.pth file
Installing pip script to /home/mjf/boopah/bin
Installed /home/mjf/boopah/lib/python2.6/site-packages/pip-0.5-py2.6.egg
Processing dependencies for pip
Finished processing dependencies for pip
$ ./bin/pip install webhelpers
Downloading/unpacking webhelpers
Downloading WebHelpers-0.6.4.tar.gz (199Kb): 199Kb downloaded
Running setup.py egg_info for package webhelpers
Installing collected packages: webhelpers
Running setup.py install for webhelpers
Successfully installed webhelpers
$ ./bin/pip install webob
Downloading/unpacking webob
Downloading WebOb-0.9.6.1.tar.gz (103Kb): 103Kb downloaded
Running setup.py egg_info for package webob
Installing collected packages: webob
Running setup.py install for webob
Successfully installed webob
$ ls lib/python2.6/site-packages/
easy-install.pth webhelpers
pip-0.5-py2.6.egg WebHelpers-0.6.4-py2.6.egg-info
setuptools-0.6c9-py2.6.egg webob
setuptools.pth WebOb-0.9.6.1-py2.6.egg-info
$ ./bin/pip zip -l
In ./lib/python2.6/site-packages:
Zipped packages:
setuptools-0.6c9-py2.6.egg
Unzipped packages:
pip-0.5-py2.6.egg (7 files)
webhelpers (52 files)
webob (17 files)
$ ./bin/pip zip webhelpers
Zip webhelpers (in ./lib/python2.6/site-packages/webhelpers)
$ ./bin/pip zip -l
In ./lib/python2.6/site-packages:
Zipped packages:
setuptools-0.6c9-py2.6.egg
webhelpers.zip
Unzipped packages:
pip-0.5-py2.6.egg (7 files)
webob (17 files)
$ ls lib/python2.6/site-packages/
easy-install.pth webhelpers.zip
pip-0.5-py2.6.egg webhelpers.zip.pth
setuptools-0.6c9-py2.6.egg webob
setuptools.pth WebOb-0.9.6.1-py2.6.egg-info
WebHelpers-0.6.4-py2.6.egg-info
Anyway, that's probably The Right way to do it, but thought you might be interested. FYI.
Can GAE eat eggs?
Ian Bicking
> Also, I probably have some terminology wrong: I'm not sure what the metaThe egg-info directory contains package metadata. These are necessary
> files like MANIFEST.in and ez_setup.py actually are... are they egg files?
> Paster files? Setuptools? I don't know what to properly call them when I
> talk about deleting them.
because Paste uses various runtime services provided by Setuptools,
which depend on the metadata for package introspection and entry
points.
ez_setup.py is unnecessary, and I think MANIFEST.in is too.
ez_setup.py is a script that downloads and installs Setuptools. It's
customary to include it in packages whose setup.py depend on
Setuptoos, as Pylons applications do. MANIFEST.in is used by setup.py
to build the egg-info metadata. Neither of these are needed on the
server, which never installs anything.
The metadata has to go somewhere, and there are two standard locations
> Some questions that I still have and can't answer:
>
> 1. Can one delete the *.egg-info directories in the "app/lib" portion of a
> GAE tree, or does that metadata get used in the standard operating of python
> (one of the reasons I HATE all of this packaging stuff; we have "webob" just
> where it's supposed to be, then the cruft "WebOb-0.9.6.1-py2.5.egg-info"
> (which is NOT a package) right next to it. But I digress...
for it in Python. easy_install does it one way, and pip does it the
other way. Pip's way has just one directory on sys.path, like in the
olden days. Easy_install's way puts every friggin' package directory
on sys.path using *.pth files. Besides being ugly, this is one more
point of failure if the path doesn't get set right. As has happened
to me before on App Engine, when it ignored *.pth files sometimes.
I wish Paste did not depend on Setuptools and entry points, but it
does, and we're not going to get away from that until we write another
framework. (Which the Pype framework might be someday.)
> 2. There are still files in an "appengine-homedir" from monkey:Those are probably things I missed when I was deleting the monkey code.
> ./app/lib/python/appengine_monkey_files. I don't know what they're for (if
> anything). I removed them in my experimentation, seems ok so far.
Google raised the quotas significantly after we bitched long enough
> 3. Much of anything about zipped / unzipped packages.
about it. I think it's 3000 files now, which is enough for a
good-sized Pylons application wiithout zipping. Before it was 1000
files, which just barely fit Pylons. The issue of course is that no
other runtime environment imposes a file quantity restriction, so the
packages were not designed to economize files. And more than half of
Pylons and Routes is backward compatibility code, which is needed now
but will be gone by Pylons 1.0.
--
Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker
Mike Orr
> I'm not sure I'm up to that task (I use Pylons a lot, but kindof "tune-out"
> the parts about PasteDeply/Script, SetupTools, etc.
Even if you can just keep testing the process and adding advice to the
howto, that would be a big help. What we need is somebody who
regularly uses Pylons on App Engine to maintain the howto. A
corollary is that being a howto maintainer makes it easier to get your
own questions answered, because people will spend more time explaining
things to a documentor, knowing that he'll recycle some of the info
back to the entire community.
>> The egg-info directory contains package metadata. These are necessary
>> because Paste uses various runtime services provided by Setuptools,
>> which depend on the metadata for package introspection and entry
>> points.
>
> Yes, I understand, but does GAE use Paste? I think my ignorance has bled
> over... I don't really get setuptools, but I also don't get how GAE is
> different. Or the same.
GAE is a Python runtime environment. It runs whatever the top-level
script tells it to run. Paste is a package of utilities which are
used by Pylons. GAE does not use the "paster" command. Instead you
have to instantiate the Pylons application manually, which is done by
the script referenced in app.yaml. Start with that script and you'll
see how Pylons is started up, and follow the code into
myapp/config/middleware, etc.
Setuptools provides utilities for distributing, installing, and
introspecting packages. These go beyond what Python itself provides.
There's an ongoing debate about how much of this functionality should
be pushed into the Python core. And if it is, how much code should be
taken from Setuptools, pip, or an emerging Setuptools competitor
(whose name I forget).
> This leads to two questions:
> 1. Can we simply delete that stuff ? It would make it harder to manage the
> app going forward.
Can we delete what?
You found some modules which look like things I intended to delete
from the package but missed. Google claims that GAE fully supports
Setuptools now, so none of the monkey code should be needed on the
server. If it is, file GAE bugs. Your workstation Python does not
need the monkey code because it's a regular Python environment. The
GAE SDK tries to mimic the server environment by disabling features
not available on the server.
However, appengine-monkey and appengine-homedir also contain the
bootstrap code used to create an application. That's what I made
appengine-homedir into, by deleting the monkey code. And now it just
needs somebody to maintain it.
> 2. Can we keep it for local management with pip et al, but NOT send it to
> GAE when we update?
I'm not sure what "it" is.
> IE, does GAE use entry points?
Applications use entry points by calling 'pkg_resources' functions.
So applications running on the GAE server can use entry points.
>> The metadata has to go somewhere, and there are two standard locations
>> for it in Python. easy_install does it one way, and pip does it the
>
> I guess that's the question: does it actually have to go somewhere? If I do
> "import simplejson" I don't need the egg-info, of course. Does Pylons use
> entrypoints a lot internally, or simple good-ole imports?
Ian says only the initial application loader (Paste Deploy) uses entry
points. However, I have used Pylons in two environments that were
Setuptools-challenged: App Engine and py2exe. Both raised exceptions
beyond that point due to 'pkg_resources.require()' calls. So it's
safest to make sure that Setuptools and the egg-info directories are
uploaded to the server.
> Yes, I keep hearing about this Pype. It's starting to sound mythical and
> religious. I look forward to it solving all of our problems! ;-)
I think it's a great concept and I want to start trying out the code,
but first I have other tasks to finish.
--
Mike Orr <slugg...@gmail.com>
Jonathan Vanasco
On Oct 8, 12:26 am, Matt Feifarek <matt.feifa...@gmail.com> wrote:
> (Wikis suck. It totally roasted all my formatting from rst/html. Editing
> yet-another-markup in a textarea is painful. Argh.)
in trac you just format it like this:
{{{
#!rst
RST HERE
}}}
I'm not sure what the pylons wiki uses, but maybe there is a plugin ?
Mike Orr
It does support ReStructured Text. Put {rst}...{rst} around the
content. The wiki software is Confluence.
--
Mike Orr <slugg...@gmail.com>
Mike Orr
> Correct; I have in the past basically flattened things for appengine.
> Unfortunately that doesn't work well with paster create -t pylons. It might
> however be more productive to have an appengine template, since you already
> know what the models will have to look like, you can skip setup.py etc., and
> generally make a tighter setup.
That's worth considering. However, most of the differences are in the
relationship of the Pylons application to the virtualenv, not in the
application itself. That would be too much for "paster create".
However, we could experiment with creating an application directory
with app.yaml and without setup.py, and see if it's feasable. When
the template is finished it can be distributed as a PyPI package.
--
Mike Orr <slugg...@gmail.com>
Matt Feifarek
> This leads to two questions:Can we delete what?
> 1. Can we simply delete that stuff ? It would make it harder to manage the
> app going forward.
Sorry; I edited out the part that would have made that more clear. What I was asking about were all of the various egg-info directores that are in the site-packages. You answered me later, but so did Ian, and you two seem to disagree. I'll make a note of this in the doc.
However, appengine-monkey and appengine-homedir also contain the
bootstrap code used to create an application. That's what I made
appengine-homedir into, by deleting the monkey code. And now it just
needs somebody to maintain it.
What/where is that bootstrap code? Is it just the ./app/config.py file? I can look in the source I guess.
Ian says only the initial application loader (Paste Deploy) uses entry
points. However, I have used Pylons in two environments that were
Setuptools-challenged: App Engine and py2exe. Both raised exceptions
beyond that point due to 'pkg_resources.require()' calls. So it's
safest to make sure that Setuptools and the egg-info directories are
uploaded to the server.
Matt Feifarek
Even if you can just keep testing the process and adding advice to the
On Thu, Oct 8, 2009 at 11:32 AM, Matt Feifarek <matt.f...@gmail.com> wrote:
> I'm not sure I'm up to that task (I use Pylons a lot, but kindof "tune-out"
> the parts about PasteDeply/Script, SetupTools, etc.
howto, that would be a big help. What we need is somebody who
regularly uses Pylons on App Engine to maintain the howto.
I don't know how long my commitment to use GAE will continue. I'm basically evaluating it now with a project. If I can help out the general community now, great. If things don't go well with the over-all experiment, I may fall out of the loop myself, but for right now, I can commit to keeping this doc up-to-date with what I discover.
I like the idea of a paster create -t pylons-appengine... that would be excellent.
Mike Orr
>> What we need is somebody who
>> regularly uses Pylons on App Engine to maintain the howto.
>
> I don't know how long my commitment to use GAE will continue. I'm basically
> evaluating it now with a project. If I can help out the general community
> now, great. If things don't go well with the over-all experiment, I may fall
> out of the loop myself, but for right now, I can commit to keeping this doc
> up-to-date with what I discover.
That sounds good.
> I like the idea of a paster create -t pylons-appengine... that would be
> excellent.
Somebody just needs to write it. If you can put together a list of
what it should contain, and how it should differ from the regular
Pylons template, we can get somebody with more experience to actually
create the template.
Application templates can also ask yes/no or text questions, and
contain if-blocks. So if you think those would be useful, just
outline what they should do.
Off the top of my head, the model.py should contain commented code for
a typical (simple) Datastore model. Not all apps will use Datastore,
but those that do will want something to start from.
--
Mike Orr <slugg...@gmail.com>
Jason S.
appengine-monkey/homedir scripts and using the stock paster template
(and an INI file). I'm not sure if I'm going to use GAE seriously, but
it's an interesting problem to work on. If I am, then personally, I
would rather not use an entirely different app template and config
system.
There were basically three steps:
1. Wrangle all the dependencies together into a 'lib' directory and
set up sys.path in my main .py file. This was the easy part. I did
have to yank the simplejson out of Django to replace the simplejson
that Pylons depends on, which is partly in C.
2. Patch Paste Deploy with a new loader class, to load a function spec
instead of an egg spec, without changing anything else about the INI
file format. In other words, my app:main looks like this:
[app:main]
use = func:myhello.config.middleware:make_app
full_stack = true
3. Hack on Mako/the render function a bit. I'm still not done with
this one. Mako's TemplateLookup uses various non-supported functions
such as os.makedirs and tempfile.mkstmp. The Template class works
fine, of course, but TemplateLookup is rather critical for any serious
usage. Has this been addressed elsewhere? I'm just now sticking my
nose into the GAE situation.
I'm sure there will be more problems down the line, but if anyone else
is interested in this approach I will write up some docs and submit
the Paste Deploy patch.
BTW, the performance I'm getting when hitting it with apache bench
(ab) is kind of awful. Haven't profiled yet.
Matt Feifarek
I just got a hello-world Pylons app running on GAE without using the
appengine-monkey/homedir scripts and using the stock paster template
(and an INI file). I'm not sure if I'm going to use GAE seriously, but
it's an interesting problem to work on. If I am, then personally, I
would rather not use an entirely different app template and config
system.
I absolutely agree. GAE is slowing me down: I can't debug, can't do print statements, and can't tell when things go wrong. It would be much preferable to keep things as stock Pylons as possible.
1. Wrangle all the dependencies together into a 'lib' directory and
set up sys.path in my main .py file. This was the easy part. I did
have to yank the simplejson out of Django to replace the simplejson
that Pylons depends on, which is partly in C.
I think that you can simply delete the speedups file from the simplejson module.
2. Patch Paste Deploy with a new loader class, to load a function spec
instead of an egg spec, without changing anything else about the INI
file format. In other words, my app:main looks like this:
[app:main]
use = func:myhello.config.middleware:make_app
full_stack = true
I'm not following completely. Does this make the GAE SDK http/app server become aware of your ini file somehow? If not, the func: you made seems to have the exact same effect as the normal pylons behavior; it still runs make_app... that's the WSGI starting point, right? (Also, middleware:make_app is the "entry point" for a pylons app).
3. Hack on Mako/the render function a bit. I'm still not done with
this one. Mako's TemplateLookup uses various non-supported functions
such as os.makedirs and tempfile.mkstmp. The Template class works
fine, of course, but TemplateLookup is rather critical for any serious
usage. Has this been addressed elsewhere?
Yep, I think that Mako can live happily on GAE. I think that commenting out cache_dir line in environment basically accomplishes what you're working on.
I'm sure there will be more problems down the line, but if anyone else
is interested in this approach I will write up some docs and submit
the Paste Deploy patch.
I'm interested.
BTW, the performance I'm getting when hitting it with apache bench
(ab) is kind of awful. Haven't profiled yet.
If you mean on the SDK, yeah, I think that's pretty standard. Hopefully in the cloud it will be good.
Mike Orr
>> 2. Patch Paste Deploy with a new loader class, to load a function spec
>> instead of an egg spec, without changing anything else about the INI
>> file format. In other words, my app:main looks like this:
>>
>> [app:main]
>> use = func:myhello.config.middleware:make_app
>> full_stack = true
>
> I'm not following completely. Does this make the GAE SDK http/app server
> become aware of your ini file somehow? If not, the func: you made seems to
> have the exact same effect as the normal pylons behavior; it still runs
> make_app... that's the WSGI starting point, right? (Also,
> middleware:make_app is the "entry point" for a pylons app).
It sounds like he has a load routine that imports
myhello.config.middleware rather than going through the entry point.
This is how you would do it "without Setuptools".
An entry point is an indirect name for a callable. This is a
Setuptools thing. A consumer package documents that it looks for a
certain entry point (a namespace). Any number of producer packages
can define entries in this namespace, which is a short name and a
string specifying "package.module:object". The user (i.e., the
application's configuration) tells the consumer to load a certain
name, using ``pkg_resources.load_entry_point()``. Setuptools goes
through the entry point to find the callable, imports it, and returns
it. In Pylons' case, PasteDeploy is the consumer, and your Pylons
application is the producer. Entry points are defined in setup.py,
and are written into the egg-info directory during install.
Even if you don't use the whole appengine-homedir infrastructure, it's
worth looking at the top-level script (runner.py, I think it's
called). It does the same thing but bypasses PasteDeploy and the INI
file. It's "different" in the front end, but once you call make_app()
it's the same.
>> 3. Hack on Mako/the render function a bit. I'm still not done with
>> this one. Mako's TemplateLookup uses various non-supported functions
>> such as os.makedirs and tempfile.mkstmp. The Template class works
>> fine, of course, but TemplateLookup is rather critical for any serious
>> usage. Has this been addressed elsewhere?
>
> Yep, I think that Mako can live happily on GAE. I think that commenting out
> cache_dir line in environment basically accomplishes what you're working on.
Mako caches compiled templates in the data directory. That won't work
on GAE because there's no filesystem. You could try making an
equivalent with Datastore or Memcached, but it would take some
programming.
--
Mike Orr <slugg...@gmail.com>
Jason S.
> I think that you can simply delete the speedups file from the simplejson
> module.
> > [app:main]
> > use = func:myhello.config.middleware:make_app
> > full_stack = true
>
> I'm not following completely. Does this make the GAE SDK http/app server
> become aware of your ini file somehow? If not, the func: you made seems to
> have the exact same effect as the normal pylons behavior; it still runs
> make_app... that's the WSGI starting point, right? (Also,
> middleware:make_app is the "entry point" for a pylons app).
soon, with code samples.
> Yep, I think that Mako can live happily on GAE. I think that commenting out
> cache_dir line in environment basically accomplishes what you're working on.
doing things I didn't have to do.
> > BTW, the performance I'm getting when hitting it with apache bench
> > (ab) is kind of awful. Haven't profiled yet.
>
> If you mean on the SDK, yeah, I think that's pretty standard. Hopefully in
> the cloud it will be good.
A stock non-Pylons, non-templated helloworld performs moderately. I've
been getting about 35-45 reqs/sec with 'ab' with various -n and -c
params, with the median time per request varying from about 400ms
under low concurrency to around 3s under high concurrency. To give
Google the benefit of the doubt, maybe things are distributed such
that concurrency from multiple hosts would be handled better.
Anything that opens/reads/lists files seems to perform worse. I might
try pre-compiling Mako templates and see how that goes. Will post more
when I know more.
Anyone else have any numbers?