I've just finished (the first draft of) an article on getting Pylons
running on GAE with a mostly "stock" setup. I've focused on using a
typical virtualenv and stock paster template and attempted to find the
minimum number of changes required to get things running, rather than
creating a new setup.
Thus, things like "paster controller" still work, you still use an INI
file for configuration, and theoretically an existing virtualenv/app
could be converted (though the necessary changes to models, etc., mean
that reusability is poor nonetheless.)
Feedback is welcome and I'll try to update the article promptly if
there are any show-stoppers or better approaches. I've done several
runs through the steps provided, though.
This seems much better than my own recent work... I think it's a mistake to
start with "appengine-homedir"... better to just start from stock Pylons and
hack it till it works on GAE. That's been my discovery over the last few
days, but I haven't patched my Pylons Wiki article yet.
I'll give you more detailed/useful feedback as soon as I can.
On Mon, Oct 12, 2009 at 8:45 PM, Jason S. <jas...@tuffmail.com> wrote:
> Hi, all,
> I've just finished (the first draft of) an article on getting Pylons
> running on GAE with a mostly "stock" setup. I've focused on using a
> typical virtualenv and stock paster template and attempted to find the
> minimum number of changes required to get things running, rather than
> creating a new setup.
> Thus, things like "paster controller" still work, you still use an INI
> file for configuration, and theoretically an existing virtualenv/app
> could be converted (though the necessary changes to models, etc., mean
> that reusability is poor nonetheless.)
> Feedback is welcome and I'll try to update the article promptly if
> there are any show-stoppers or better approaches. I've done several
> runs through the steps provided, though.
These new approaches are encouraging. The purpose of
appengine-homedir was to make it easy to get started, by handling the
virtualenv and directory structure for you. But if it turns out that
another way is easier, we can deprecate it. I do like the idea of a
pylons-appengine template for Paster.
On Wed, Oct 14, 2009 at 2:35 PM, Matt Feifarek <matt.feifa...@gmail.com> wrote:
> Very nice.
> This seems much better than my own recent work... I think it's a mistake to
> start with "appengine-homedir"... better to just start from stock Pylons and
> hack it till it works on GAE. That's been my discovery over the last few
> days, but I haven't patched my Pylons Wiki article yet.
> I'll give you more detailed/useful feedback as soon as I can.
> On Mon, Oct 12, 2009 at 8:45 PM, Jason S. <jas...@tuffmail.com> wrote:
>> Hi, all,
>> I've just finished (the first draft of) an article on getting Pylons
>> running on GAE with a mostly "stock" setup. I've focused on using a
>> typical virtualenv and stock paster template and attempted to find the
>> minimum number of changes required to get things running, rather than
>> creating a new setup.
>> Thus, things like "paster controller" still work, you still use an INI
>> file for configuration, and theoretically an existing virtualenv/app
>> could be converted (though the necessary changes to models, etc., mean
>> that reusability is poor nonetheless.)
>> Feedback is welcome and I'll try to update the article promptly if
>> there are any show-stoppers or better approaches. I've done several
>> runs through the steps provided, though.
On Wed, Oct 14, 2009 at 4:35 PM, Matt Feifarek <matt.feifa...@gmail.com> wrote: > This seems much better than my own recent work... I think it's a mistake to > start with "appengine-homedir"... better to just start from stock Pylons and > hack it till it works on GAE. That's been my discovery over the last few > days, but I haven't patched my Pylons Wiki article yet.
I personally got too annoyed with testing and general interactivity (there's a nose extension though, which I cribbed from -- but I never actually got it to work myself).
On Thu, Oct 15, 2009 at 12:10 AM, Ian Bicking <i...@colorstudy.com> wrote:
> I personally got too annoyed with testing and general interactivity > (there's a nose extension though, which I cribbed from -- but I never > actually got it to work myself).
This process has been EXTREMELY annoying. I've spent many hours trying to get things to work that I should have been spending working on my project. Mostly, its my own ignorance of current magical techniques:
I still don't get setuptools. I don't understand namespace packages, or rather their deployment. I don't get why I can't move (for example) paste, and the Paste* items to another directory that is ON sys.path and have it be import-able. All of my frustrations with GAE have been of this sort.
Basically, the challenge to getting Pylons on GAE is to manage libraries. Some need to be there for Pylons but not for GAE. Some need to be there fore GAE by not Pylons. Some need to be both. Some need to be uploaded with the "app", some cannot be, some simply should-not be. It's not clear when egg-info directories need to go up to GAE, and when they are wasting space.
And one cannot simply add things to sys.path because of setuptools. And when setuptools can't find something, I have *no idea* how to fix it. I wish I could just ditch all of this stuff and manually manage sys.path, but as is, Paste especially seems to rely on some magic to find itself. Though, I noticed that if I put an __init__.py in paste/ then it starts acting like a normal package again (duh). Basically, packaging is blowing my mind, and I wish I was just using python modules and packages.
STILL, I've made progress, and seemingly Jason has too. I think though, that it's never going to be "automagic"... at least not on all OS's. GAE is too hackety.
My status: I can take a stock pylons application in a virtualenv, wedge-in the google libraries for yaml, datastore and the like, and run the app with paster serve. If one moves the pylons-required libraries from the standard site-packages to the GAE app's lib/python and add that location to sys.path, you're almost there. Hacking in some ENVIRONMENT variables at boot-strap time and you are there.
I didn't need to patch paste like Jason did, but maybe I'm missing something. I'll be writing up my new recipe asap.
On Oct 15, 11:59 am, Matt Feifarek <matt.feifa...@gmail.com> wrote:
> On Thu, Oct 15, 2009 at 12:10 AM, Ian Bicking <i...@colorstudy.com> wrote:
> > I personally got too annoyed with testing and general interactivity
> > (there's a nose extension though, which I cribbed from -- but I never
> > actually got it to work myself).
> This process has been EXTREMELY annoying. I've spent many hours trying to
> get things to work that I should have been spending working on my project.
> Mostly, its my own ignorance of current magical techniques:
I'm not sold on the platform, either. It does seem to be a bundle of
limitations with a few features. But I suppose we have to get things
running on it to do a hands-on evaluation.
> I still don't get setuptools. I don't understand namespace packages, or
> rather their deployment. I don't get why I can't move (for example) paste,
> and the Paste* items to another directory that is ON sys.path and have it be
> import-able. All of my frustrations with GAE have been of this sort.
When you install eggs with easy_install, it adds the egg's path to
easy_install.pth. If you move the egg somewhere else, it won't work
unless it's added to the path somehow. BTW, I don't think GAE
uses .pth files, which is why sys.path manipulation is necessary.
When you install Paste packages with easy_install, they go into
separate eggs and in each one, paste/__init__.py has code that
modifies the path to make the namespacing work. It's not something
that just works because of setuptools. Took me a while to find that.
When you install them with pip, I think they overlap (i.e. one paste/
directory in site-packages/).
> And one cannot simply add things to sys.path because of setuptools.
I'm not sure what all the problems you're having with setuptools
entail, but my app.py *prepends* to sys.path, which works a little
better because it gets around installed versions of libraries in the
GAE environment.
Here's part of app.py:
sys.path = [appdir, libdir] + ['%s/%s' % (libdir, n) for n in
os.listdir(libdir) if n.endswith('.egg')] + sys.path
Performance would probably be better with a hard-coded list of
packages, but this is more flexible of course. On the other hand, I
think a switch to pip would make the listdir part unnecessary.
> My status: I can take a stock pylons application in a virtualenv, wedge-in
> the google libraries for yaml, datastore and the like, and run the app with
> paster serve. If one moves the pylons-required libraries from the standard
> site-packages to the GAE app's lib/python and add that location to sys.path,
> you're almost there. Hacking in some ENVIRONMENT variables at boot-strap
> time and you are there.
I use the GAE SDK instead of paster serve, because then all the GAE-
specific libs should be provided without moving anything. Can you use
datastore, etc, locally without the SDK?
> I didn't need to patch paste like Jason did, but maybe I'm missing
> something. I'll be writing up my new recipe asap.
Interesting. Maybe that step could be eliminated. I think I was
getting DistributionNotFound from pkg_resources and figured that it
would be better to work around eggs/the egg loader.
On Thu, Oct 15, 2009 at 12:41 PM, Jason S. <jas...@tuffmail.com> wrote: > > I still don't get setuptools. I don't understand namespace packages, or > > rather their deployment. I don't get why I can't move (for example) > paste, > > and the Paste* items to another directory that is ON sys.path and have it > be > > import-able. All of my frustrations with GAE have been of this sort.
> When you install eggs with easy_install, it adds the egg's path to > easy_install.pth. If you move the egg somewhere else, it won't work
That part I get. In fact, I've often modified easy_install.pth manually.
But I'm using pip. And pip doesn't mess with that file, nor it *seems* any other meta-data anywhere on sys.path. But still, I couldn't move Paste* around.
I was considering trying an install with easy_install and then hand-modifying easy_install.pth next.
> unless it's added to the path somehow. BTW, I don't think GAE > uses .pth files, which is why sys.path manipulation is necessary.
It *seems *like this isn't true. If I omit pth files when uploading to GAE, my app is failing with a 500 error. But I wouldn't trust that yet... can you do a test on your setup and see if you need or don't need the .pth files? (up to and including easy_install)
> When you install them with pip, I think they overlap (i.e. one paste/ > directory in site-packages/).
Yes, that's true, but I don't understand why moving those directories from one site-packages to another should matter. I can't find whatever cruft is telling python to look in that and only that sys.path location for paste.
> I'm not sure what all the problems you're having with setuptools > entail, but my app.py *prepends* to sys.path, which works a little > better because it gets around installed versions of libraries in the > GAE environment.
You mean to have the upgraded webob for example?
I use the GAE SDK instead of paster serve, because then all the GAE-
> specific libs should be provided without moving anything. Can you use > datastore, etc, locally without the SDK?
Yes. I simply add the path to the google libraries to sys.path; I'm running DataStore (well, whatever emulation that the SDK provides on my "paster serve development.ini" pylons app. Which means I can use the debugger, webtest, etc.
Can you use the debugger (weberror)?
> Interesting. Maybe that step could be eliminated. I think I was > getting DistributionNotFound from pkg_resources and figured that it > would be better to work around eggs/the egg loader.
I think I got around that by dropping an empty __init__.py file into paste/
On Thu, Oct 15, 2009 at 9:59 AM, Matt Feifarek <matt.feifa...@gmail.com> wrote: > This process has been EXTREMELY annoying. I've spent many hours trying to > get things to work that I should have been spending working on my project. > Mostly, its my own ignorance of current magical techniques:
All of us have been annoyed. That's why it has taken us 1 1/2 years to write a simple, failsafe howto, and we're still not there yet. Philip and Ian did the original work in April 2008 which led to Appengine-Monkey and patches to get Mako and Beaker to run on GAE. I came in later to do more installation and testing. All of us have spent weeks pulling our hair out trying to get various things to work.
If you just want to get going quickly with GAE, you may be better off using the mini framework that comes with it, or one of the smaller WSGI frameworks. Django has also been ported to it. (And they had similar problems to what we had, BTW.)
> I still don't get setuptools. I don't understand namespace packages, or > rather their deployment. I don't get why I can't move (for example) paste, > and the Paste* items to another directory that is ON sys.path and have it be > import-able. All of my frustrations with GAE have been of this sort.
First some vocabulary to make sure we're talking about the same things.
A "package" is a directory under sys.path containing Python modules and an __init__.py module. It's importable.
A "distribution" is a tarball or egg packaged so that it can be distributed on PyPI. Distributions thus contain packages. However, many people including me also call distributions "packages", so the term package is ambiguous.
A "namespace package" is a Setuptools trick that allows multiple distributions to be installed under the same package namespace. Paste uses this so that 'paste.deploy' and 'paste.script' can be distributed seprately from 'paste', for those who want to use one but not the others. Zope used this to split the 'zope' namespace into 100+ distributions. This was necessary to make Zope pythonic and non-monolithic.
So the answer to your question is, when you moved the packages, you did not keep everything in sync. Either you missed some egg-info directories or .pth files, or there were paths encoded in the egg-info files that were no longer valid. Or something like that.
Easy_install and Pip do manage namespace packages differently, so the techniques for moving them would be different. It's normally easier to reinstall the packages rather than to move them. If you don't want to go to the network every time, you can build your own archive of source files and install from that.
> Basically, the challenge to getting Pylons on GAE is to manage libraries. > Some need to be there for Pylons but not for GAE. Some need to be there fore > GAE by not Pylons. Some need to be both. Some need to be uploaded with the > "app", some cannot be, some simply should-not be. It's not clear when > egg-info directories need to go up to GAE, and when they are wasting space.
Yes, that is all true. The answer is complicated, and we're not sure of all the details.
> And one cannot simply add things to sys.path because of setuptools. And when > setuptools can't find something, I have no idea how to fix it. I wish I > could just ditch all of this stuff and manually manage sys.path, but as is, > Paste especially seems to rely on some magic to find itself.
Paste, and thus Pylons, was built using Setuptools' features. At the time, Setuptools was seen as the package management solution for Python, and would eventually be added to the standard library. That didn't happen due to an ongoing controversy over Setuptools and Distutils, which you can read about in the pylons-discuss archive and web-sig archive.
The Python core developers did not like Setuptools and did not use it. Some of those developers (Guido in particular) wrote Google App Engine. So, unsurprisingly, it was Setuptools-unaware. This led to the majority of problems for using Pylons on it. Basically, Pylons and Paste was written for a standard Python runtime environment (CPython and everything available in that world), and GAE is not that.
> My status: I can take a stock pylons application in a virtualenv, wedge-in > the google libraries for yaml, datastore and the like, and run the app with > paster serve. If one moves the pylons-required libraries from the standard > site-packages to the GAE app's lib/python and add that location to sys.path, > you're almost there. Hacking in some ENVIRONMENT variables at boot-strap > time and you are there.
I used Appengine-Monkey/Appengine-Homedir because my paramount goal was a simple install procedure I could explain to others. Those packages seemed to offer that, by automating things that would otherwise have to be done by hand. (Particularly installing virtualenv and setuptools.) But maybe starting from a stock Pylons installation and app is a better way to go? That's what your and Jason's research will help answer.
> > unless it's added to the path somehow. BTW, I don't think GAE
> > uses .pth files, which is why sys.path manipulation is necessary.
> It *seems *like this isn't true. If I omit pth files when uploading to GAE,
> my app is failing with a 500 error. But I wouldn't trust that yet... can you
> do a test on your setup and see if you need or don't need the .pth files?
> (up to and including easy_install)
I think it ignores pth files. I just did a test with the SDK and a
barebones directory structure:
app.py
app.yaml
test.pth
mydir/
myfile.py
test.pth contains ./mydir. app.py tries to "import myfile" and fails
to do so. Whereas if I add "mydir" to sys.path in app.py it works
immediately.
On Thu, Oct 15, 2009 at 2:21 PM, Mike Orr <sluggos...@gmail.com> wrote:
> On Thu, Oct 15, 2009 at 9:59 AM, Matt Feifarek <matt.feifa...@gmail.com> > wrote: > > This process has been EXTREMELY annoying. I've spent many hours trying to > > get things to work that I should have been spending working on my > project. > > Mostly, its my own ignorance of current magical techniques:
> All of us have been annoyed. That's why it has taken us 1 1/2 years > to write a simple, failsafe howto, and we're still not there yet. > Philip and Ian did the original work in April 2008 which led to > Appengine-Monkey and patches to get Mako and Beaker to run on GAE. I > came in later to do more installation and testing. All of us have > spent weeks pulling our hair out trying to get various things to work.
Oh I know it.
When I said "hours" for me, I knew it meant "weeks" for a few of you. I would have no hope without the work that you-all have already done.
On Oct 15, 2:01 pm, Matt Feifarek <matt.feifa...@gmail.com> wrote:
> > When you install them with pip, I think they overlap (i.e. one paste/
> > directory in site-packages/).
> Yes, that's true, but I don't understand why moving those directories from
> one site-packages to another should matter. I can't find whatever cruft is
> telling python to look in that and only that sys.path location for paste.
I just did a fresh pip install of Pylons and looked over the Paste
stuff, and it looks like there is a .pth file that comes with each
Paste package (ending with the string "nspkg.pth") that contains
executable Python (!) (I didn't know pth files could have code in
them) that jiggles with the path. Weird stuff. Anyway, it wouldn't
work without those pth files, or, as you noted, simply adding an
__init__.py to paste/
> > Interesting. Maybe that step could be eliminated. I think I was
> > getting DistributionNotFound from pkg_resources and figured that it
> > would be better to work around eggs/the egg loader.
> I think I got around that by dropping an empty __init__.py file into paste/
I don't know why yours works, but I now know why mine didn't work.
GAE's os.path.exists() pretends that a directory that (a) ends in
".egg-info" and (b) is NOT located in the deployment root, i.e. is a
subdirectory, does not exist. Naturally, this screws around with the
default paster create template, which has an .egg-info directory in
it, and anything installed using pip, which uses .egg-info
directories. EGG-INFO directories are unaffected.
With this knowledge I got a stock app working without patching Paste
Deploy, and without changing the INI file, in several different ways.
The "best" way so far is to rename the outer myapp directory to
myapp.egg and rename the inner myapp.egg-info directory to EGG-INFO.
You can also leave the app template as-is and install it inside the
virtualenv with easy_install, but you'd have to do this every time you
make a change.
On Fri, Oct 16, 2009 at 9:07 AM, Jason S. <jas...@tuffmail.com> wrote: > I just did a fresh pip install of Pylons and looked over the Paste > stuff, and it looks like there is a .pth file that comes with each > Paste package (ending with the string "nspkg.pth") that contains > executable Python (!) (I didn't know pth files could have code in > them) that jiggles with the path. Weird stuff. Anyway, it wouldn't > work without those pth files, or, as you noted, simply adding an > __init__.py to paste/
Here comes yet another approach to Pylons on GAE problem, without virtualenv:
1. Write a script, to analysis dependencies of the working Pylons project (via setup.py or .egg-info dir), and gather all the dependencies of dependencies, recursively.
2. ln -s (or copy) all the dependencies python package to the root dir, so that GAE can import them.
3. If the dependence package has a .pth associated with it, append its content to a module located in the GAE app root dir (the same dir with app.yaml), named pthes.py. At the beginning of runner.py, import pthes. So that the paste package without __init__.py can be imported, because its .pth appended 'paste' dir in sys.path.
4. ln -s (or copy) all the dependencies .egg-info dir to the root dir, so that pkg_resource.load_entry_point could work.
5. (optional) To avoid .egg-info dirs take too much quote of file count, we can package (and compress) all of them into a single file, e.g. a tar.gz file or a pickle file, and patch pkg_resource to know how to read original entry_points.txt from that file. Add some cache mechanism if need.
This approach makes all the pkg_resource needed information available in GAE environment, so that no any further modification needed for the Pylons package or project, including the .ini file.
As I'm not a setuptools guru, I'd like to hear some opinions on it or suggestions before get hands dirty.
On Thu, Oct 15, 2009 at 2:01 PM, Matt Feifarek <matt.feifa...@gmail.com> wrote: > On Thu, Oct 15, 2009 at 12:41 PM, Jason S. <jas...@tuffmail.com> wrote:
>> > I still don't get setuptools. I don't understand namespace packages, or >> > rather their deployment. I don't get why I can't move (for example) >> > paste, >> > and the Paste* items to another directory that is ON sys.path and have >> > it be >> > import-able. All of my frustrations with GAE have been of this sort.
>> When you install eggs with easy_install, it adds the egg's path to >> easy_install.pth. If you move the egg somewhere else, it won't work
> That part I get. In fact, I've often modified easy_install.pth manually.
> But I'm using pip. And pip doesn't mess with that file, nor it seems any > other meta-data anywhere on sys.path. But still, I couldn't move Paste* > around.
> I was considering trying an install with easy_install and then > hand-modifying easy_install.pth next.
pip will respect easy_install'd packages. So if you easy_install something the pip install it, pip will see the easy_install'd package and do nothing. You should be sure to use virtualenv --no-site-packages. Also pip install --ignore-installed will force a fresh install.
I remember at some point having a problem with pip installing the Paste packages, and paste/__init__.py was missing. I think I was working on an actual app so I punted and created the file manually, and didn't try to reproduce and fix it.
Note that to get .pth files respected you can't add a directory to sys.path, instead you have to do "import site; site.addsitedir(dir)" which also looks for .pth files. runner.py does this; if nothing else you should look at what runner.py does. It also goes through the necessary steps to get rid of the system webob (or other installed libraries) if you want to use an upgraded version. It's slightly different on the SDK and the live environment.
>> unless it's added to the path somehow. BTW, I don't think GAE >> uses .pth files, which is why sys.path manipulation is necessary.
> It seems like this isn't true. If I omit pth files when uploading to GAE, my > app is failing with a 500 error. But I wouldn't trust that yet... can you do > a test on your setup and see if you need or don't need the .pth files? (up > to and including easy_install)
At one time .pth files weren't being read on GAE (or maybe appengine-monkey messed something up, I can't remember). This hasn't been a problem for at least the past six months, or maybe a year.
>> When you install them with pip, I think they overlap (i.e. one paste/ >> directory in site-packages/).
> Yes, that's true, but I don't understand why moving those directories from > one site-packages to another should matter. I can't find whatever cruft is > telling python to look in that and only that sys.path location for paste.
>> I'm not sure what all the problems you're having with setuptools >> entail, but my app.py *prepends* to sys.path, which works a little >> better because it gets around installed versions of libraries in the >> GAE environment.
> You mean to have the upgraded webob for example?
I found prepending alone to be insufficient in some cases, instead I scan and remove any webob directory, in addition to prepending.
>> I use the GAE SDK instead of paster serve, because then all the GAE- >> specific libs should be provided without moving anything. Can you use >> datastore, etc, locally without the SDK?
> Yes. I simply add the path to the google libraries to sys.path; I'm running > DataStore (well, whatever emulation that the SDK provides on my "paster > serve development.ini" pylons app. Which means I can use the debugger, > webtest, etc.
I don't use paster serve, and wouldn't recommend it. You can/should use paste.deploy.loadapp('development.ini') in your main.py or runner.py or whatever script, and then use dev_appserver.
> Can you use the debugger (weberror)?
I have gotten it to work, but by overwriting environ['wsgi.multiprocess'] (to set it to False) -- without that the debugger won't work. But in the SDK generally it won't be multiprocess until you edit a file, so the debugger works fairly reliably.
Generally, I'll note one advantage to GAE that you should make use of: once a set of libraries is working, everything to make them work is in a set of portable files (.py files and directories and whatnot). After installing I check *all* the files into version control. I don't do this elsewhere because there are C extensions and other things that aren't portable, but GAE disallows all those so they aren't an issue. It would be relatively easy to create an actual tar/zip of everything for a working Pylons setup, and anyone could use it as a starting point, and it would be 100% reliable. Unfortunately that's only good enough for a specific set of libraries, and either you have to repackage a collection of different library combinations or you have to support installation in some manner. appengine-homedir is one attempt to get installation working reasonably. The intent of appengine-homedir is also that you check app/ into version control (but not the enclosing virtualenv environment). It would be kind of nice if app/ had some script included to recreate the enclosing virtualenv environment.
Another detail I guess is what your app is named -- each person might name it differently, and that effects the file locations, so makes a simple tar/zip file harder. Are app names worth it? I don't know... could you name them all "app" and be done with it? If you start combining multiple apps it might matter, but people haven't done that much (though it is possible, and several GAE tools are distributed that way). Also I mostly got rid of configuration, as GAE apps aren't generally "configured" (due to the static nature of the deployment). But that broke paster controller and paster shell (though with what sitecustomize does paster shell isn't as necessary). Restoring development.ini, or allowing it as an option, would be easy enough to do.
On Fri, Oct 16, 2009 at 1:04 PM, Ian Bicking <i...@colorstudy.com> wrote:
> I remember at some point having a problem with pip installing the > Paste packages, and paste/__init__.py was missing. I think I was > working on an actual app so I punted and created the file manually, > and didn't try to reproduce and fix it.
Yes, it seems to be necessary in all of my experiments to create that file to make paste/ into a true package. No big deal.
> I don't use paster serve, and wouldn't recommend it. You can/should > use paste.deploy.loadapp('development.ini') in your main.py or > runner.py or whatever script, and then use dev_appserver.
Would you mind expanding on that a little bit? Why don't you recommend it?
> > Can you use the debugger (weberror)?
> I have gotten it to work, but by overwriting > environ['wsgi.multiprocess'] (to set it to False) -- without that the > debugger won't work. But in the SDK generally it won't be > multiprocess until you edit a file, so the debugger works fairly > reliably.
Interesting. I'll have to try that.
Generally, I'll note one advantage to GAE that you should make use of:
> once a set of libraries is working, everything to make them work is in > a set of portable files (.py files and directories and whatnot). > After installing I check *all* the files into version control.
Cool. Good concept. But it seems like once you do something like that (or like moving your libraries into the GAE "app") you can't manage the virtualenv install anymore (like adding new distributions with pip). I suppose one can always manually add them.
> It would be relatively easy to create an actual > tar/zip of everything for a working Pylons setup, and anyone could use > it as a starting point, and it would be 100% reliable. Unfortunately > that's only good enough for a specific set of libraries, and either > you have to repackage a collection of different library combinations > or you have to support installation in some manner.
Still, a "stock" pylons app could be hosted on BitBucket; just clone and run. Cool idea.
> appengine-homedir > is one attempt to get installation working reasonably. The intent of > appengine-homedir is also that you check app/ into version control > (but not the enclosing virtualenv environment). It would be kind of > nice if app/ had some script included to recreate the enclosing > virtualenv environment.
It *seemed* to me that once appengine-homedir got running, the VE is basically moot; you no longer run the devappserver from it, right? Since your runner.py script does all of the sys.path manipulations, there's no point in keeping the VE bin and lib and what-not around, especially if you don't recommend using paster serve.
> Another detail I guess is what your app is named -- each person might > name it differently, and that effects the file locations, so makes a > simple tar/zip file harder. Are app names worth it? I don't know...
Since each GAE is insulated in its own namespace, if the whole world named their apps "app" it wouldn't matter, IMO.
> that way). Also I mostly got rid of configuration, as GAE apps aren't > generally "configured" (due to the static nature of the deployment). > But that broke paster controller and paster shell (though with what > sitecustomize does paster shell isn't as necessary). Restoring > development.ini, or allowing it as an option, would be easy enough to
Why isn't paster shell necessary? Seems like sitecustomize.py is mostly about fixing imports and shim'ing things into sys.path.
On Fri, Oct 16, 2009 at 3:57 PM, Matt Feifarek <matt.feifa...@gmail.com> wrote: > On Fri, Oct 16, 2009 at 1:04 PM, Ian Bicking <i...@colorstudy.com> wrote: >> I don't use paster serve, and wouldn't recommend it. You can/should >> use paste.deploy.loadapp('development.ini') in your main.py or >> runner.py or whatever script, and then use dev_appserver.
> Would you mind expanding on that a little bit? Why don't you recommend it?
I'm note sure if he meant "never use paster serve" or "just don't use it for App Engine apps".
> It *seemed* to me that once appengine-homedir got running, the VE is > basically moot; you no longer run the devappserver from it, right?
It's mainly used for installing packages. You may realize later you need a different package, or want to upgrade the ones you have. Without virtualenv, you'll have to have a global copy of Setuptools installed, and will have to specify the destination path every time you install something (because you're not installing into the default site-packages any more). Or create a distutils.cfg specifiying the install location.
On Fri, Oct 16, 2009 at 5:57 PM, Matt Feifarek <matt.feifa...@gmail.com> wrote: >> I don't use paster serve, and wouldn't recommend it. You can/should >> use paste.deploy.loadapp('development.ini') in your main.py or >> runner.py or whatever script, and then use dev_appserver.
> Would you mind expanding on that a little bit? Why don't you recommend it?
On GAE specifically, dev_appserver does all the fixups and security restrictions to make it a mostly representative environment. So you should just use dev_appserver.
>> Generally, I'll note one advantage to GAE that you should make use of: >> once a set of libraries is working, everything to make them work is in >> a set of portable files (.py files and directories and whatnot). >> After installing I check *all* the files into version control.
> Cool. Good concept. But it seems like once you do something like that (or > like moving your libraries into the GAE "app") you can't manage the > virtualenv install anymore (like adding new distributions with pip). I > suppose one can always manually add them.
pip install works with appengine-homedir. A couple things are installed outside of the app/ directory, but once it's setup there's a file in lib/python2.5/distutils/distutils.cfg that makes everything install into app/lib/python.
When I update a library or install something new, I just run another commit after.
>> appengine-homedir >> is one attempt to get installation working reasonably. The intent of >> appengine-homedir is also that you check app/ into version control >> (but not the enclosing virtualenv environment). It would be kind of >> nice if app/ had some script included to recreate the enclosing >> virtualenv environment.
> It *seemed* to me that once appengine-homedir got running, the VE is > basically moot; you no longer run the devappserver from it, right? Since > your runner.py script does all of the sys.path manipulations, there's no > point in keeping the VE bin and lib and what-not around, especially if you > don't recommend using paster serve.
I recommend the venv for running python interactively, running tests, and installing things.
>> that way). Also I mostly got rid of configuration, as GAE apps aren't >> generally "configured" (due to the static nature of the deployment). >> But that broke paster controller and paster shell (though with what >> sitecustomize does paster shell isn't as necessary). Restoring >> development.ini, or allowing it as an option, would be easy enough to
> Why isn't paster shell necessary? Seems like sitecustomize.py is mostly > about fixing imports and shim'ing things into sys.path.
It also sets up the data store, so you can access and use your models. It doesn't set up the entire configuration of your app, but the most important configuration is generally persistence, which sitecustomize handles. And maybe it could even handle more if it was useful to do so.
