[Plone-Users] buildout, instancemanager, and eggs
George Lee
or pointing to documentation on how to use them?
Ditto with installing products/packages that use eggs -- how does this work? (I
think this is related to using buildout and instancemanager but I'm not usre)
Peace,
George
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Plone-Users mailing list
Plone...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plone-users
Martin Aspeli
> Can someone write an e-mail describing how to use buildout and instancemanager,
> or pointing to documentation on how to use them?
The two are separate and somewhat competing. I'm pretty strongly in the
buildout camp, mainly because it's something that is used (increasingly
as a standard method) by the Zope 3 and Grok communities, and has a life
outside Zope too, and I like working with it. :)
> Ditto with installing products/packages that use eggs -- how does this work? (I
> think this is related to using buildout and instancemanager but I'm not usre)
It's obviously in my book ;-p
Seriously, though, I will be writing documentation on this, to present
at the conference and because we definitely need more of this. The very
short version is this:
First, see http://plone.org/documentation/how-to/use-paster. This tells
you about the paster command, and how to create new eggs (hint: paster
create -t plone).
Install easy_install, and get ZopeSkel:
$ wget peak.telecommunity.com/dist/ez_setup.py
$ python ez_setup.py
$ easy_install ZopeSkel
Then, create a new buildout, all-set-up for Plone 3, like so:
$ paster create -t plone3_buildout myproject
Answer its questions. The only one you need to answer is the admin password.
Also - make sure you have PIL installed! :-)
Also - if you are on Windows, read the README.txt in the buildout, it
contains important information. And be sure that Python 2.4 is in your
system PATH, and that Pythons' scripts directory (where
paster/easy_install get installed) is too.
$ cd myproject
$ python bootstrap.py
Now you're ready to use buildout. You can do that with:
$ ./bin/buildout
(You need to re-run this command each time you change buildout.cfg.)
This reads buildout.cfg. You should read that file and it tells you
where to add new things. Use the --help option to see some options. The
most important ones are -oN, which cause it to not look for updated
packages and not try to download things, making buildout much faster.
To install a new Zope 2 style product, either add it manually to the
top-level 'products' directory (note lowercase p), or edit buildout.cfg
and add a download URL to the [productdistros] section, which causes
buildout to manage the download and install for you.
To install a new egg that's in the cheesehop (or another index) just
list it in the list of eggs under the main [buildout] section.
If that egg is a package with a configure.zcml, you need to load it,
either from another product's configure.zcml with an <include
package="my.package" />, or by adding the package to the list in the
'zcml' option in the [instance] section.
If that egg is a development egg, then check it out (that is, the whole
thing, including the directory where setup.py is located!) to the src/
directory, and add the following:
[buildout]
...
develop =
src/my.package
...
eggs =
...
my.package
Look at the buildout page in the cheesehop to learn more:
http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b30. The docs here
are quite good.
For reference, here's a buildout.cfg for a project of mine. This one's
pretty complete. It will:
- Download Plone 3's products and eggs and install them
- Download a version of Zope recommended for Plone, compile and
install it (all locally, of course)
- Download various products (e.g. DocFinderTab) and install them
- Download various eggs (e.g. simplon.plone.ldap) and install them
- Create a Python interpreter in ./bin/zopepy that has all of Zope's
eggs (but not products) in its pythonpath, for testing
- Download, build and install Deliverance
- Download, build and install Varnish
You may not need all of that, of course.
[buildout]
parts =
plone
zope2
productdistros
instance
zopepy
deliverance
varnish
find-links =
http://download.zope.org/ppix
http://download.zope.org/distribution/
http://effbot.org/downloads
eggs =
clientname.theme
clientname.policy
# LDAP configuration - should be made into a proper setup.py dependency
simplon.plone.ldap
# Project workspaces - should be made into a proper setup.py dependency
borg.localrole
borg.project
develop =
src/clientname.policy
src/clientname.theme
# Deliverance - we don't have a proper release of this yet.
src/Deliverance
[plone]
recipe = plone.recipe.plone>=3.0,<3.1
[zope2]
recipe = plone.recipe.zope2install
url = ${plone:zope2-url}
[productdistros]
recipe = plone.recipe.distros
urls =
http://www.zope.org/Members/shh/DocFinderTab/1.0.1/DocFinderTab-1.0.1.tar.gz
http://plone.org/products/clouseau/releases/0.7.1/Clouseau.0.7.1.zip
http://plone.org/products/richdocument/releases/3.0/RichDocument-3.0.tar.gz
http://plone.org/products/cachefu/releases/1.1/CacheFu-1.1.tgz
http://plone.org/products/ploneldap/releases/1.0/ploneldap-bundle-1%2C0rc1.tar.gz
nested-packages =
CacheFu-1.1.tgz
version-suffix-packages =
[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
debug-mode = on
verbose-security = on
http-address = 127.0.0.1:8080
eggs =
${buildout:eggs}
${plone:eggs}
products =
${buildout:directory}/products
${plone:location}
${productdistros:location}
zcml =
clientname.policy
simplon.plone.ldap
[zopepy]
recipe = zc.recipe.egg
eggs = ${instance:eggs}
interpreter = zopepy
extra-paths = ${zope2:location}
scripts = zopepy
[deliverance]
recipe = plone.recipe.deliverance
[varnish]
recipe = plone.recipe.varnish
url =
http://puzzle.dl.sourceforge.net/sourceforge/varnish/varnish-1.1.1.tar.gz
bind = localhost:8081
backends = 127.0.0.1:8000
cache-size = 100M
--
Acquisition is a jealous mistress
George Lee
>
> George Lee wrote:
> > Can someone write an e-mail describing how to use buildout
> > and instancemanager,
> > or pointing to documentation on how to use them?
>
> Seriously, though, I will be writing documentation on this, to present
> at the conference and because we definitely need more of this. The very
> short version is this: ...
Thanks -- this is really helpful. I have some questions but first am trying to
just run buildout to see how it works. When I try to start up Zope using
./instance start, it gives me an error (traceback below) -- any idea what is
going on?
Thanks; peace,
George
Traceback (most recent call last):
File "/ywalab/parts/zope2/lib/python/Zope2/Startup/run.py", line 56, in ?
run()
File "/ywalab/parts/zope2/lib/python/Zope2/Startup/run.py", line 21, in run
starter.prepare()
File "/ywalab/parts/zope2/lib/python/Zope2/Startup/__init__.py", line 102,
in prepare
self.startZope()
File "/ywalab/parts/zope2/lib/python/Zope2/Startup/__init__.py", line 278,
in startZope
Zope2.startup()
File "/ywalab/parts/zope2/lib/python/Zope2/__init__.py", line 47, in startup
_startup()
File "/ywalab/parts/zope2/lib/python/Zope2/App/startup.py", line 59, in
startup
DB = dbtab.getDatabase('/', is_root=1)
File "/ywalab/parts/zope2/lib/python/Zope2/Startup/datatypes.py", line 280,
in getDatabase
db = factory.open(name, self.databases)
File "/ywalab/parts/zope2/lib/python/Zope2/Startup/datatypes.py", line 178,
in open
DB = self.createDB(database_name, databases)
File "/ywalab/parts/zope2/lib/python/Zope2/Startup/datatypes.py", line 175,
in createDB
return ZODBDatabase.open(self, databases)
File "/ywalab/parts/zope2/lib/python/ZODB/config.py", line 97, in open
storage = section.storage.open()
File "/ywalab/parts/zope2/lib/python/ZODB/config.py", line 135, in open
quota=self.config.quota)
File "/ywalab/parts/zope2/lib/python/ZODB/FileStorage/FileStorage.py",
line 112, in __init__
self._lock_file = LockFile(file_name + '.lock')
File "/ywalab/parts/zope2/lib/python/ZODB/lock_file.py", line 60, in __init__
self._fp = open(path, 'w+')
IOError: [Errno 13] Permission denied: '/ywalab/var/filestorage/Data.fs.lock'
George Lee
> Thanks -- this is really helpful. I have some questions but first am trying to
> just run buildout to see how it works. When I try to start up Zope using
> ./instance start, it gives me an error (traceback below) -- any idea what is
> going on?
Actually, and I get that far only if I first change zope.conf to set
effective-user; without doing this, I get the following traceback. Am I supposed
to set an "effective-user" in zope.conf?
Traceback (most recent call last):
File "/ywalab/parts/zope2/lib/python/Zope2/Startup/run.py", line 56, in ?
run()
File "/ywalab/parts/zope2/lib/python/Zope2/Startup/run.py", line 21, in run
starter.prepare()
File "/ywalab/parts/zope2/lib/python/Zope2/Startup/__init__.py", line 98, in
prepare
self.dropPrivileges()
File "/ywalab/parts/zope2/lib/python/Zope2/Startup/__init__.py", line 234, in
dropPrivileges
return dropPrivileges(self.cfg)
File "/ywalab/parts/zope2/lib/python/Zope2/Startup/__init__.py", line 403, in
dropPrivileges
raise ZConfig.ConfigurationError(msg)
ZConfig.ConfigurationError: A user was not specified to setuid to; fix this to
start as root (change the effective-user directive in zope.conf)
Peace,
George
P.S. Something weird is going on with gmane -- it's not threading new
conversations currently.
Wichert Akkerman
I suspect you ran buildout as root? If you do that all files and
directories will be owned by root. But when you start the Zope instance
it will run as the 'zope' user, who is not allowed to write in
var/filestorage in your buildout. I suggest doing a 'chown -R zope var'
to fix that.
Wichert.
--
Wichert Akkerman <wic...@wiggy.net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
Wichert Akkerman
> George Lee <georgeleejr@...> writes:
>
> > Thanks -- this is really helpful. I have some questions but first am trying to
> > just run buildout to see how it works. When I try to start up Zope using
> > ./instance start, it gives me an error (traceback below) -- any idea what is
> > going on?
>
>
> Actually, and I get that far only if I first change zope.conf to set
> effective-user; without doing this, I get the following traceback. Am I supposed
> to set an "effective-user" in zope.conf?
You are not supposed to touch anything in the parts/ directory:
everything in there is created and managed by buildout. You can set the
effective user with an "effective-user = <user" in the instance section
of your buildcout.cfg and rerun buildout. That will regenerate zope.conf
for you.
Wichert.
--
Wichert Akkerman <wic...@wiggy.net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
-------------------------------------------------------------------------
Peter Fraterdeus
>
>IOError: [Errno 13] Permission denied: '/ywalab/var/filestorage/Data.fs.lock'
>
Your user doesn't have permission to write to the filestorage directory
Start zope as sudo (or as the user defined in zope.conf)
P
Raphael Ritz
> Can someone write an e-mail describing how to use buildout and instancemanager,
> or pointing to documentation on how to use them?
>
> Ditto with installing products/packages that use eggs -- how does this work? (I
> think this is related to using buildout and instancemanager but I'm not usre)
>
In addition to all infos you've gotten already these slides
http://www.bccn-berlin.de/People/ritz/presentations/on-distributing-python-packages
from a presentation I gave recently might provide some
pointers to further background material.
Raphael
George Lee
> > > George Lee wrote:
> > When I try to start up Zope using
> > ./instance start, it gives me an error (traceback below) -- any idea what is
> > going on?
> >
> > ...
> >
> > IOError: [Errno 13] Permission denied: '/ywalab/var/filestorage/Data.fs.lock'
>
> I suspect you ran buildout as root?
Thanks, I reran buildout not as root and it works.
Do I have this right: buildout is a useful tool for setting up multiple
sandboxes with different configurations of projects - for instance, I could be
running different versions of Zope and Plone with different libraries present or
not. buildout.cfg is a single file where I can specify everything I need to and
will let buildout download the correct files into the right places very
straightforwardly. Egg-ified packages work nicely too.
Three questions:
* Is buildout only for development purposes, or also to set up production
environments?
* Suppose a project that I'm developing contains a library plone.app.hooray
and a Products/Hooray product. How would I structure the files for others to
download -- is it a single .egg file, and then when they run a setup script it
extracts the library and Product to the correct location? What existing tools
let me create such an .egg file? I've skimmed through documentation on eggs and
easy_install but it's hard to distill it to easily understand what would happen
in this case. Can anyone explain / point to documentation / point to an example
of a download place where I could look at the .egg and folder structure?
* If I understand correctly, buildout is not the only way to unpack eggs - is
that correct? What is the usual way (easy_install?) and what is the relationship
between buildout and eggs?
Thanks!
Peace,
George
Martin Aspeli
>>> IOError: [Errno 13] Permission denied: '/ywalab/var/filestorage/Data.fs.lock'
>> I suspect you ran buildout as root?
>
>
> Thanks, I reran buildout not as root and it works.
I don't think you should generally run buildout as root. You obviously
need write privileges over the buildout's directory, but it is entirely
self-contained within that directory.
> Do I have this right: buildout is a useful tool for setting up multiple
> sandboxes with different configurations of projects - for instance, I could be
> running different versions of Zope and Plone with different libraries present or
> not.
Yep. Another useful thing is to let lots of developers share the exact
same configuration, by checking in the buildout to svn. If you do that,
set an svn:ignore on the parts and var directories, so that people don't
have to get each other's compiled code (in general, it's safe to nuke
the contents of 'parts' and re-run buildout) or Data.fs or logs etc.
> buildout.cfg is a single file where I can specify everything I need to and
> will let buildout download the correct files into the right places very
> straightforwardly. Egg-ified packages work nicely too.
Definitely.
> Three questions:
>
> * Is buildout only for development purposes, or also to set up production
> environments?
Both. I tend to have a buildout.cfg that is used for development, and a
separate deployment.cfg that's used for deployment, turning off debug
mode, setting up ZEO etc. You can specify a config file with -c, and one
config file can "extend" another, making it possible to share the basic
setup.
> * Suppose a project that I'm developing contains a library plone.app.hooray
> and a Products/Hooray product. How would I structure the files for others to
> download -- is it a single .egg file, and then when they run a setup script it
> extracts the library and Product to the correct location? What existing tools
> let me create such an .egg file? I've skimmed through documentation on eggs and
> easy_install but it's hard to distill it to easily understand what would happen
> in this case. Can anyone explain / point to documentation / point to an example
> of a download place where I could look at the .egg and folder structure?
An old-style product is just a directory, so you'd typically create an
archive and upload it somewhere. People who want to use that can
download it and extract into the 'products' directory, or use the
plone.recipe.distros recipe to have it automatically downloaded and
installed.
Create new egg-aware packages with 'paster create -t plone' and make
sure setup.py reflects your package and its dependencies. You can then
upload it to the Cheese Shop/PyPI. buildout will be able to find it
there if it's referenced by name, and download and install it.
You don't generally use easy_install if you're using buildout.
easy_install does the same kind of find-download-install thing as
buildout, but for the global python environment (or a workingenv, but
let's not get confused about that right now).
Under the hood, buildout and easy_install both use setuptools.
There is comprehensive, if lengthy, documentation here:
http://peak.telecommunity.com/DevCenter/PythonEggs
> * If I understand correctly, buildout is not the only way to unpack eggs - is
> that correct? What is the usual way (easy_install?) and what is the relationship
> between buildout and eggs?
No, that's wrong. buildout can download and install any eggs you tell it
about (typically using the 'eggs' option in buildout.cfg). See above.
I recommend you read http://cheeseshop.python.org/pypi/zc.buildout/1.0.0b30.
Martin
--
Acquisition is a jealous mistress
George Lee
also helped me a good deal:
http://dev.plone.org/plone/browser/ploneout/trunk/README.txt
Peace,
George
George Lee
> also helped me a good deal:
>
> http://dev.plone.org/plone/browser/ploneout/trunk/README.txt
This is very helpful as well:
http://svn.plone.org/svn/collective/ZopeSkel/trunk/zopeskel/templates/plone3_buildout/README.txt
Ross Patterson
>> Hi -- for anyone following along to learn about buildout, reading through this
>> also helped me a good deal:
>>
>> http://dev.plone.org/plone/browser/ploneout/trunk/README.txt
>
> This is very helpful as well:
>
> http://svn.plone.org/svn/collective/ZopeSkel/trunk/zopeskel/templates/plone3_buildout/README.txt
George,
Thanks so much for starting this thread. I was very much in need of
reading this discussion.
Ross
Ross Patterson
deploy to a ZEO server and multiple ZEO clients. Currently we use svn
bundles of Zope and Plone and our own packages and products in one big
svn:externals definition. We also use some more recent versions of
certain products and packages than those included in the upstream
svn:externals. For example:
Zope svn://svn.zope.org/repos/main/Zope/branches/2.10
Zope/lib/python/zope/app/interface svn://svn.zope.org/repos/main/zope.app.interface/trunk/src/zope/app/interface
Though I haven't seen any documentation to this effect, svn seems to
treat the second svn:externals as a "svn switch" on "svn up".
So one question I have is how would I "override" the versions of some
package like this using buildout?
Another question I have is that there aren't releases for all the
dependencies our application has. I'm wondering what the best way is to
define a canonical release for deployment to our cluster. Ideally, we'd
have a buildout (or whatever) on a staging server where we're testing
the candidate for deployment and then when all tests pass that *exact*
source should be deployed to the cluster. How might one do this without
upstream releases? Is there a way to get buildout and setuptools to
play together so that I can build a source distribution from the test
passing buildout on staging? Is there a way I can have a buildout on
each cluster node such that a simple ./bin/buildout will get the latest
deployment release for that node?
Wichert Akkerman
> We have a Plone 3.0/Zope 2.10 application that we need to develop on, maintain and
> deploy to a ZEO server and multiple ZEO clients. Currently we use svn
> bundles of Zope and Plone and our own packages and products in one big
> svn:externals definition. We also use some more recent versions of
> certain products and packages than those included in the upstream
> svn:externals. For example:
>
> Zope svn://svn.zope.org/repos/main/Zope/branches/2.10
> Zope/lib/python/zope/app/interface svn://svn.zope.org/repos/main/zope.app.interface/trunk/src/zope/app/interface
>
> Though I haven't seen any documentation to this effect, svn seems to
> treat the second svn:externals as a "svn switch" on "svn up".
>
> So one question I have is how would I "override" the versions of some
> package like this using buildout?
You can enforce specific versions in your buildout configuration. For
example in a grok project I have this:
[buildout]
versions = versioninfo
[versioninfo]
grok = 0.10
zope.app.keyreference = 3.4.0a1
ZODB3 = 3.8.0b2
SQLAlchemy = 0.3.10
> Another question I have is that there aren't releases for all the
> dependencies our application has. I'm wondering what the best way is to
> define a canonical release for deployment to our cluster. Ideally, we'd
> have a buildout (or whatever) on a staging server where we're testing
> the candidate for deployment and then when all tests pass that *exact*
> source should be deployed to the cluster. How might one do this without
> upstream releases? Is there a way to get buildout and setuptools to
> play together so that I can build a source distribution from the test
> passing buildout on staging? Is there a way I can have a buildout on
> each cluster node such that a simple ./bin/buildout will get the latest
> deployment release for that node?
I would start by asking upstream developers for releases. Even if that
does not result in a release it will give them a hint that egg releases
are useful for people and they should consider making them. I think
there is also a buildout recipe that allows you to specify a subversion
url and revision pin and which will use that to do a checkout. You can
then register that with your instance as either an egg or a product.
Wichert.
--
Wichert Akkerman <wic...@wiggy.net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
Martin Aspeli
wichert wrote:
>
>> So one question I have is how would I "override" the versions of some
>> package like this using buildout?
>
> You can enforce specific versions in your buildout configuration. For
> example in a grok project I have this:
>
> [buildout]
> versions = versioninfo
>
> [versioninfo]
> grok = 0.10
> zope.app.keyreference = 3.4.0a1
> ZODB3 = 3.8.0b2
> SQLAlchemy = 0.3.10
>
Another way of doing the same thing, is to specify egg versions when you
list eggs, e.g.:
eggs =
my.package==0.3
sqlalchemy>0.3,<0.4dev
And so on.
>> Another question I have is that there aren't releases for all the
>> dependencies our application has. I'm wondering what the best way is to
>> define a canonical release for deployment to our cluster. Ideally, we'd
>> have a buildout (or whatever) on a staging server where we're testing
>> the candidate for deployment and then when all tests pass that *exact*
>> source should be deployed to the cluster. How might one do this without
>> upstream releases? Is there a way to get buildout and setuptools to
>> play together so that I can build a source distribution from the test
>> passing buildout on staging? Is there a way I can have a buildout on
>> each cluster node such that a simple ./bin/buildout will get the latest
>> deployment release for that node?
>
> I would start by asking upstream developers for releases. Even if that
> does not result in a release it will give them a hint that egg releases
> are useful for people and they should consider making them. I think
> there is also a buildout recipe that allows you to specify a subversion
> url and revision pin and which will use that to do a checkout. You can
> then register that with your instance as either an egg or a product.
>
There are definitely svn-checkout recipes - search the cheese shop. But you
can also use svn:externals to check out and manage development eggs.
Martin
--
View this message in context: http://www.nabble.com/buildout%2C-instancemanager%2C-and-eggs-tf4313762s6742.html#a12363590
Sent from the General Questions mailing list archive at Nabble.com.