[Plone-developers] Buildout recipe for installing Plone
Martin Aspeli
I've been grappling with how best to install Plone as a dependency in a
buildout, and I think I've finally settled on a solution. The main idea
here is that dist_plone.platforms.independent is the canonical place to
define which products and eggs make up a release. This means that the
recipe is probably not so useful for tracking svn, but that's not what
it's designed for.
I've added some extra metadata in dist_plone to allow us to track
versions of eggs. This is used by this recipe, and also in the Plone
egg, so that this egg doesn't need to track dependencies on its own.
The installers still use tarballs for eggs. I suspect we'll keep it that
way for 3.0, since it would be a lot of work to make them use setuptools
directly. This adds a little bit of burden, since when updating
dist_plone, we need to make sure the version there is the actual,
released, cheese-shop (or other find-link) available egg.
Hanno: It uses plone.recipe.distros to download all the tarballs. It'd
be great if you could make a new release of that, since I just fixed a
bug in it. :) There's also some XXX comments in the recipe source that
describe some difficulties in translating the feature set of dist_plone
to that of plone.recipe.distros.
At the moment, the recipe does not rely on the Plone egg, because Daniel
and I were unable to get that to install cleanly in buildout (for
whatever reason, CMFPlone and friends were not being initialised and
thus couldn't be added in the ZMI). The Plone egg also suffers a bit
because it has a static list of eggs but tracks svn for Products. We
really need tags on the Products bundle + a tagged list of eggs for each
release.
Anyway - take a look at
http://dev.plone.org/collective/browser/buildout/plone.recipe.plone/trunk/setup.py
for more documentation.
I probably won't make a cheese shop release until plone.recipe.distros
is re-released (given that it has a bug which causes the buildout to
crash, though I've fixed it I think).
I plan to keep the versions of releases of plone.recipe.plone in sync
with Plone versions, so the current version would be 3.0b3. The release
cycle would probably be:
- Update and tag dist_plone
- Switch the external in plone.recipe.plone from dist_plone trunk to
that tag
- Tag plone.recipe.plone
- Do the cheese shop dance
People should be able to control which release of Plone they get with:
[plone]
recipe = plone.recipe.plone>=3.0-dev,<3.5
or similar specifiers.
Thoughts?
Martin
--
Acquisition is a jealous mistress
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Plone-developers mailing list
Plone-de...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plone-developers
Alex Clark
Nice! This is probably a n00b error but I get: http://paste.plone.org/15215
Something about:
[Errno 2] No such file or directory:
'/private/var/zope/plone3/parts/plone/CMFActionIcons'
Which I don't understand because it looks like CMFActionIcons is in the bundle
you defined in build/lib/dist_plone/platforms/independent.py:
http://antiloop.plone.org/download/CMF-2.1.0-beta-plone-3.0b2.tar.gz
Alex
--
Alex Clark (http://aclark.net)
Hanno Schlichting
> Hi guys,
>
> I've been grappling with how best to install Plone as a dependency in a
> buildout, and I think I've finally settled on a solution. The main idea
> here is that dist_plone.platforms.independent is the canonical place to
> define which products and eggs make up a release. This means that the
> recipe is probably not so useful for tracking svn, but that's not what
> it's designed for.
>
> I've added some extra metadata in dist_plone to allow us to track
> versions of eggs. This is used by this recipe, and also in the Plone
> egg, so that this egg doesn't need to track dependencies on its own.
>
> The installers still use tarballs for eggs. I suspect we'll keep it that
> way for 3.0, since it would be a lot of work to make them use setuptools
> directly. This adds a little bit of burden, since when updating
> dist_plone, we need to make sure the version there is the actual,
> released, cheese-shop (or other find-link) available egg.
Note that at least the Windows installer, while still using package
tarballs for eggs, actually installs setuptools in the local Python
environment and installs all packages in Python's site-packages (it
calls easy_install on all of them) instead of putting them into the
instances lib/python. So at least on Windows we already use proper eggs :)
> Hanno: It uses plone.recipe.distros to download all the tarballs. It'd
> be great if you could make a new release of that, since I just fixed a
> bug in it. :) There's also some XXX comments in the recipe source that
> describe some difficulties in translating the feature set of dist_plone
> to that of plone.recipe.distros.
I just made a new release (which you could have done yourself ;) ). If
there are some new options which would be helpful for the distros
recipe, like an option to rename the downloaded folder you can add those...
> At the moment, the recipe does not rely on the Plone egg, because Daniel
> and I were unable to get that to install cleanly in buildout (for
> whatever reason, CMFPlone and friends were not being initialised and
> thus couldn't be added in the ZMI). The Plone egg also suffers a bit
> because it has a static list of eggs but tracks svn for Products. We
> really need tags on the Products bundle + a tagged list of eggs for each
> release.
Why would we need a tag on the products bundle (or better a tag which
includes svn:externals to all tags of all products)? We have tags on all
of the products and tarballs available for them, which is what is used
by the dist_plone script. I don't see much value in having a way to do
an SVN checkout of a particular release, that's what releases and
tarballs are made for.
> Anyway - take a look at
> http://dev.plone.org/collective/browser/buildout/plone.recipe.plone/trunk/setup.py
> for more documentation.
>
> I probably won't make a cheese shop release until plone.recipe.distros
> is re-released (given that it has a bug which causes the buildout to
> crash, though I've fixed it I think).
Release is out, so feel free to make a release or just a development
snapshot available of p.r.p.
> I plan to keep the versions of releases of plone.recipe.plone in sync
> with Plone versions, so the current version would be 3.0b3. The release
> cycle would probably be:
>
> - Update and tag dist_plone
> - Switch the external in plone.recipe.plone from dist_plone trunk to
> that tag
> - Tag plone.recipe.plone
> - Do the cheese shop dance
>
> People should be able to control which release of Plone they get with:
>
> [plone]
> recipe = plone.recipe.plone>=3.0-dev,<3.5
>
> or similar specifiers.
>
> Thoughts?
Really nice work!!!
Hanno
Martin Aspeli
> Martin,
>
> Nice! This is probably a n00b error but I get: http://paste.plone.org/15215
> Something about:
> [Errno 2] No such file or directory:
> '/private/var/zope/plone3/parts/plone/CMFActionIcons'
>
> Which I don't understand because it looks like CMFActionIcons is in the bundle
> you defined in build/lib/dist_plone/platforms/independent.py:
> http://antiloop.plone.org/download/CMF-2.1.0-beta-plone-3.0b2.tar.gz
This is the bug I fixed in plone.recipe.distros. Wait for a new release
or install that too as a development egg.
MArtin
--
Acquisition is a jealous mistress
Martin Aspeli
> Note that at least the Windows installer, while still using package
> tarballs for eggs, actually installs setuptools in the local Python
> environment and installs all packages in Python's site-packages (it
> calls easy_install on all of them) instead of putting them into the
> instances lib/python. So at least on Windows we already use proper eggs :)
Does that mean we *shouldn't* do this in the buildout on Windows? I
suppose it's only a problem if someone uses Python + Zope from the win
installer but a separate buildout (which is supported).
>> Hanno: It uses plone.recipe.distros to download all the tarballs. It'd
>> be great if you could make a new release of that, since I just fixed a
>> bug in it. :) There's also some XXX comments in the recipe source that
>> describe some difficulties in translating the feature set of dist_plone
>> to that of plone.recipe.distros.
>
> I just made a new release (which you could have done yourself ;) ).
Ah, right. Heh, sorry. ;-)
> If
> there are some new options which would be helpful for the distros
> recipe, like an option to rename the downloaded folder you can add those...
:-p
The code there scares me every time I look at it. My time is severely
restricted at the moment, so I'm trying not to be a perfectionist, but
if we need it, I'm sure it won't be hard to add.
>> At the moment, the recipe does not rely on the Plone egg, because Daniel
>> and I were unable to get that to install cleanly in buildout (for
>> whatever reason, CMFPlone and friends were not being initialised and
>> thus couldn't be added in the ZMI). The Plone egg also suffers a bit
>> because it has a static list of eggs but tracks svn for Products. We
>> really need tags on the Products bundle + a tagged list of eggs for each
>> release.
>
> Why would we need a tag on the products bundle (or better a tag which
> includes svn:externals to all tags of all products)? We have tags on all
> of the products and tarballs available for them, which is what is used
> by the dist_plone script. I don't see much value in having a way to do
> an SVN checkout of a particular release, that's what releases and
> tarballs are made for.
Yes, except the Plone egg pulls in Products/ (and makes it a namespace
package). It does that with svn:externals, since it doesn't happen on
the fly during install, it happens before the egg is uploaded to the
cheese shop.
If course, we could manually extract the appropriate tarball, or script
it, before making a Plone egg release. We could even pull that into the
dist_plone environment.
>> Anyway - take a look at
>> http://dev.plone.org/collective/browser/buildout/plone.recipe.plone/trunk/setup.py
>> for more documentation.
>>
>> I probably won't make a cheese shop release until plone.recipe.distros
>> is re-released (given that it has a bug which causes the buildout to
>> crash, though I've fixed it I think).
>
> Release is out, so feel free to make a release or just a development
> snapshot available of p.r.p.
Cool, will do.
>> I plan to keep the versions of releases of plone.recipe.plone in sync
>> with Plone versions, so the current version would be 3.0b3. The release
>> cycle would probably be:
>>
>> - Update and tag dist_plone
>> - Switch the external in plone.recipe.plone from dist_plone trunk to
>> that tag
>> - Tag plone.recipe.plone
>> - Do the cheese shop dance
>>
>> People should be able to control which release of Plone they get with:
>>
>> [plone]
>> recipe = plone.recipe.plone>=3.0-dev,<3.5
>>
>> or similar specifiers.
>>
>> Thoughts?
>
> Really nice work!!!
;-)
You'll notice that it's really your nice work. The recipe doesn't do
much except call zc.recipe.egg and plone.recipe.distros with parameters
sucked from dist_plone. ;-)
Martin
--
Acquisition is a jealous mistress
Martin Aspeli
>> Release is out, so feel free to make a release or just a development
>> snapshot available of p.r.p.
>
> Cool, will do.
I also made a ZopeSkel release.
You should now be able to do this:
$ easy_install ZopeSkel # try to do ZopeSkel>=0.4 if that fails
$ paster create -t plone3_buildout myproject
$ cd myproject
$ python bootstrap.py
$ ./bin/buildout
Which should install Plone 3 beta 3. And then:
$ ./bin/instance fg
To run it.
Hanno Schlichting
Martin Aspeli wrote:
>> Note that at least the Windows installer, while still using package
>> tarballs for eggs, actually installs setuptools in the local Python
>> environment and installs all packages in Python's site-packages (it
>> calls easy_install on all of them) instead of putting them into the
>> instances lib/python. So at least on Windows we already use proper eggs :)
>
> Does that mean we *shouldn't* do this in the buildout on Windows? I
> suppose it's only a problem if someone uses Python + Zope from the win
> installer but a separate buildout (which is supported).
I don't mean we need to change anything here. This was just a simple
remark, that some of our installers already handle eggs in some way.
>> If
>> there are some new options which would be helpful for the distros
>> recipe, like an option to rename the downloaded folder you can add those...
>
> :-p
>
> The code there scares me every time I look at it. My time is severely
> restricted at the moment, so I'm trying not to be a perfectionist, but
> if we need it, I'm sure it won't be hard to add.
The best thing you could do is start writing tests for these recipes. I
myself find tested code much less scary than untested code ;) But I know
that dreaded time restriction problem all to well... :(
>>> At the moment, the recipe does not rely on the Plone egg, because Daniel
>>> and I were unable to get that to install cleanly in buildout (for
>>> whatever reason, CMFPlone and friends were not being initialised and
>>> thus couldn't be added in the ZMI). The Plone egg also suffers a bit
>>> because it has a static list of eggs but tracks svn for Products. We
>>> really need tags on the Products bundle + a tagged list of eggs for each
>>> release.
>> Why would we need a tag on the products bundle (or better a tag which
>> includes svn:externals to all tags of all products)? We have tags on all
>> of the products and tarballs available for them, which is what is used
>> by the dist_plone script. I don't see much value in having a way to do
>> an SVN checkout of a particular release, that's what releases and
>> tarballs are made for.
>
> Yes, except the Plone egg pulls in Products/ (and makes it a namespace
> package). It does that with svn:externals, since it doesn't happen on
> the fly during install, it happens before the egg is uploaded to the
> cheese shop.
Hhm, that Products is a namespace package thing should really be handled
on the right level first. Zope itself should declare it a namespace
package and include the __init__.py in the skeleton for new instances.
Unless we have at least that it is a matter of luck (or better who comes
first on the PYTHONPATH) if the namespace declaration actually works.
In addition it currently isn't possible to have products distributed as
eggs register themselves as Zope2 products: five:registerPackage which
would tell Zope to load the initialize method is broken in ZEO setup's
and you also cannot put it into a product that is still delivered as a
tarball, as the initialize method is called twice than, with all kinds
of funny and not so funny side effects...
I think we need to fix that story first before advocating it in any way.
>>> Thoughts?
>> Really nice work!!!
>
> ;-)
>
> You'll notice that it's really your nice work. The recipe doesn't do
> much except call zc.recipe.egg and plone.recipe.distros with parameters
> sucked from dist_plone. ;-)
Putting it all together is sometimes harder than writing the single
pieces :)
Hanno
Martin Aspeli
> Hhm, that Products is a namespace package thing should really be handled
> on the right level first. Zope itself should declare it a namespace
> package and include the __init__.py in the skeleton for new instances.
> Unless we have at least that it is a matter of luck (or better who comes
> first on the PYTHONPATH) if the namespace declaration actually works.
True.
> In addition it currently isn't possible to have products distributed as
> eggs register themselves as Zope2 products: five:registerPackage which
> would tell Zope to load the initialize method is broken in ZEO setup's
> and you also cannot put it into a product that is still delivered as a
> tarball, as the initialize method is called twice than, with all kinds
> of funny and not so funny side effects...
>
> I think we need to fix that story first before advocating it in any way.
Hold on ... does that apply to things not in Products.* also?
We have lots of eggs in Plone 3 that need <five:registerPackage />, e.g.
because they need skins or an extension profile.
Are these broken with ZEO too?
Martin
--
Acquisition is a jealous mistress
Hanno Schlichting
> Hanno Schlichting wrote:
>
>> In addition it currently isn't possible to have products distributed as
>> eggs register themselves as Zope2 products: five:registerPackage which
>> would tell Zope to load the initialize method is broken in ZEO setup's
>> and you also cannot put it into a product that is still delivered as a
>> tarball, as the initialize method is called twice than, with all kinds
>> of funny and not so funny side effects...
>>
>> I think we need to fix that story first before advocating it in any way.
>
> Hold on ... does that apply to things not in Products.* also?
>
> We have lots of eggs in Plone 3 that need <five:registerPackage />, e.g.
> because they need skins or an extension profile.
At least those things can and should be done with zcml statements.
> Are these broken with ZEO too?
The only purpose of five:registerPackage is to call the initialize
method of a package even, if it is not in a Products folder configured
in your zope.conf. That's something you shouldn't really do in a Python
package in the first place.
The bug description is here: http://dev.plone.org/plone/ticket/6476
which also points to the underlying Zope bug report which has been
marked as wontfix.
The only thing that doesn't work is to create a new ZEO server once a
package that uses five:registerPackage is present. If you copy over the
Data.fs from a different server, all seems to work.
Still you need to expect that the code in your initialize method is
being called twice...
Hanno
Daniel Nouri
Martin Aspeli wrote:
> Hi guys,
>
> I've been grappling with how best to install Plone as a dependency in a
> buildout, and I think I've finally settled on a solution. The main idea
> here is that dist_plone.platforms.independent is the canonical place to
> define which products and eggs make up a release. This means that the
> recipe is probably not so useful for tracking svn, but that's not what
> it's designed for.
My feeling is that dist_plone.platforms.independent should be phased out and
replaced by a setup.py. Is this reasonable for 3.5?
> People should be able to control which release of Plone they get with:
>
> [plone]
> recipe = plone.recipe.plone>=3.0-dev,<3.5
>
> or similar specifiers.
I wish we could find out what the problem is with getting the Plone egg
working in buildout. Because
eggs =
Plone>=3.0-dev,<3.1
still looks nicer to me. Not being able to look at plone.recipe.plone, I
have no idea what it does, however, I can't imagine it's much more than a
workaround?
Regards
Daniel
Daniel Nouri
> Why would we need a tag on the products bundle (or better a tag which
> includes svn:externals to all tags of all products)? We have tags on all
> of the products and tarballs available for them, which is what is used
> by the dist_plone script. I don't see much value in having a way to do
> an SVN checkout of a particular release, that's what releases and
> tarballs are made for.
svn:externals in a tagged bundle is what I use in AT right now. It's super
useful to see what has changed compared to trunk versions (I can svn diff
all lines in EXTERNALS.txt against their /trunk equivalents). Also, an svn
export of that bundle actually *is* the release. Not sure how useful this
would be for Plone itself.
Martin and I originally thought that it would be useful to have for the
Plone egg, since that pulls in Products/ right now via svn:externals. But I
guess relying on dist_plone or whatever to produce that Products directory
is fine too.
Daniel
Martin Aspeli
>> I've been grappling with how best to install Plone as a dependency in a
>> buildout, and I think I've finally settled on a solution. The main idea
>> here is that dist_plone.platforms.independent is the canonical place to
>> define which products and eggs make up a release. This means that the
>> recipe is probably not so useful for tracking svn, but that's not what
>> it's designed for.
>
> My feeling is that dist_plone.platforms.independent should be phased out and
> replaced by a setup.py. Is this reasonable for 3.5?
Maybe, but it probably can't be a "simple" setup.py. It needs to
reference tarballs for products and eggs. As I understand, this is a
requirement for the installers.
>> People should be able to control which release of Plone they get with:
>>
>> [plone]
>> recipe = plone.recipe.plone>=3.0-dev,<3.5
>>
>> or similar specifiers.
>
> I wish we could find out what the problem is with getting the Plone egg
> working in buildout. Because
>
> eggs =
> Plone>=3.0-dev,<3.1
>
> still looks nicer to me.
True.
> Not being able to look at plone.recipe.plone, I
> have no idea what it does, however, I can't imagine it's much more than a
> workaround?
It does three things:
- Specify a Zope tarball. This means that (optionally) you can get a
"known good" version of Zope in the buildout environment, which is
similar to what the installers do. The nice thing about that is that if
you update Plone and the new version requires/prefers a new version of
Zope, you get that too.
- Add all of Plone's eggs to the working set, pegged at the versions
from dist_plone. I'm working on something right now which lets you
override that with develop eggs if you need to track subversion for one
or more dependent eggs (well, you can do that now of course, but I want
to make it a bit easier).
- Download and install all the product tarballs.
I'm not convinced that having the Plone egg ship with a
Products/__init__.py that declares it as a namespace package can work
reliably until Zope does that for us. Otherwise, as Hanno points out, I
believe the first thing on the PYTHONPATH will win. Also,
$SOFTWARE_HOME/Products/__init__.py does set some variables which serve
a purpose.
If we can make the Plone egg work reliably, then plone.recipe.plone
still has a purpose, but it can get a little simpler: Instead of
specifying all eggs, it can just add Plone==3.0 or whatever to the
working set and (when appropriate) stop downloading tarballs of
products. To the user of the recipe, there won't be a difference, so we
have some insulation from future changes in Plone's packaging and egg story.
Martin
--
Acquisition is a jealous mistress
Daniel Nouri
> I'm not convinced that having the Plone egg ship with a
> Products/__init__.py that declares it as a namespace package can work
> reliably until Zope does that for us. Otherwise, as Hanno points out, I
> believe the first thing on the PYTHONPATH will win. Also,
> $SOFTWARE_HOME/Products/__init__.py does set some variables which serve
> a purpose.
Oh, right. I've submitted this patch 4 months ago, but somehow never got
around to checking it in. I guess it's way too late now for Zope 2.10:
http://www.zope.org/Collectors/Zope/2276/Products.diff/view
Daniel