I've been grappling with how best to install Plone as a dependency in a buildout, and I think I've finally settled on a solution. The main idea here is that dist_plone.platforms.independent is the canonical place to define which products and eggs make up a release. This means that the recipe is probably not so useful for tracking svn, but that's not what it's designed for.
I've added some extra metadata in dist_plone to allow us to track versions of eggs. This is used by this recipe, and also in the Plone egg, so that this egg doesn't need to track dependencies on its own.
The installers still use tarballs for eggs. I suspect we'll keep it that way for 3.0, since it would be a lot of work to make them use setuptools directly. This adds a little bit of burden, since when updating dist_plone, we need to make sure the version there is the actual, released, cheese-shop (or other find-link) available egg.
Hanno: It uses plone.recipe.distros to download all the tarballs. It'd be great if you could make a new release of that, since I just fixed a bug in it. :) There's also some XXX comments in the recipe source that describe some difficulties in translating the feature set of dist_plone to that of plone.recipe.distros.
At the moment, the recipe does not rely on the Plone egg, because Daniel and I were unable to get that to install cleanly in buildout (for whatever reason, CMFPlone and friends were not being initialised and thus couldn't be added in the ZMI). The Plone egg also suffers a bit because it has a static list of eggs but tracks svn for Products. We really need tags on the Products bundle + a tagged list of eggs for each release.
I probably won't make a cheese shop release until plone.recipe.distros is re-released (given that it has a bug which causes the buildout to crash, though I've fixed it I think).
I plan to keep the versions of releases of plone.recipe.plone in sync with Plone versions, so the current version would be 3.0b3. The release cycle would probably be:
- Update and tag dist_plone - Switch the external in plone.recipe.plone from dist_plone trunk to that tag - Tag plone.recipe.plone - Do the cheese shop dance
People should be able to control which release of Plone they get with:
[plone] recipe = plone.recipe.plone>=3.0-dev,<3.5
or similar specifiers.
Thoughts?
Martin -- Acquisition is a jealous mistress
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Plone-developers mailing list Plone-develop...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers
Nice! This is probably a n00b error but I get: http://paste.plone.org/15215 Something about: [Errno 2] No such file or directory: '/private/var/zope/plone3/parts/plone/CMFActionIcons'
> I've been grappling with how best to install Plone as a dependency in a > buildout, and I think I've finally settled on a solution. The main idea > here is that dist_plone.platforms.independent is the canonical place to > define which products and eggs make up a release. This means that the > recipe is probably not so useful for tracking svn, but that's not what > it's designed for.
> I've added some extra metadata in dist_plone to allow us to track > versions of eggs. This is used by this recipe, and also in the Plone > egg, so that this egg doesn't need to track dependencies on its own.
> The installers still use tarballs for eggs. I suspect we'll keep it that > way for 3.0, since it would be a lot of work to make them use setuptools > directly. This adds a little bit of burden, since when updating > dist_plone, we need to make sure the version there is the actual, > released, cheese-shop (or other find-link) available egg.
> Hanno: It uses plone.recipe.distros to download all the tarballs. It'd > be great if you could make a new release of that, since I just fixed a > bug in it. :) There's also some XXX comments in the recipe source that > describe some difficulties in translating the feature set of dist_plone > to that of plone.recipe.distros.
> At the moment, the recipe does not rely on the Plone egg, because Daniel > and I were unable to get that to install cleanly in buildout (for > whatever reason, CMFPlone and friends were not being initialised and > thus couldn't be added in the ZMI). The Plone egg also suffers a bit > because it has a static list of eggs but tracks svn for Products. We > really need tags on the Products bundle + a tagged list of eggs for each > release.
> I probably won't make a cheese shop release until plone.recipe.distros > is re-released (given that it has a bug which causes the buildout to > crash, though I've fixed it I think).
> I plan to keep the versions of releases of plone.recipe.plone in sync > with Plone versions, so the current version would be 3.0b3. The release > cycle would probably be:
> - Update and tag dist_plone > - Switch the external in plone.recipe.plone from dist_plone trunk to > that tag > - Tag plone.recipe.plone > - Do the cheese shop dance
> People should be able to control which release of Plone they get with:
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Plone-developers mailing list Plone-develop...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers
> I've been grappling with how best to install Plone as a dependency in a > buildout, and I think I've finally settled on a solution. The main idea > here is that dist_plone.platforms.independent is the canonical place to > define which products and eggs make up a release. This means that the > recipe is probably not so useful for tracking svn, but that's not what > it's designed for.
> I've added some extra metadata in dist_plone to allow us to track > versions of eggs. This is used by this recipe, and also in the Plone > egg, so that this egg doesn't need to track dependencies on its own.
> The installers still use tarballs for eggs. I suspect we'll keep it that > way for 3.0, since it would be a lot of work to make them use setuptools > directly. This adds a little bit of burden, since when updating > dist_plone, we need to make sure the version there is the actual, > released, cheese-shop (or other find-link) available egg.
Note that at least the Windows installer, while still using package tarballs for eggs, actually installs setuptools in the local Python environment and installs all packages in Python's site-packages (it calls easy_install on all of them) instead of putting them into the instances lib/python. So at least on Windows we already use proper eggs :)
> Hanno: It uses plone.recipe.distros to download all the tarballs. It'd > be great if you could make a new release of that, since I just fixed a > bug in it. :) There's also some XXX comments in the recipe source that > describe some difficulties in translating the feature set of dist_plone > to that of plone.recipe.distros.
I just made a new release (which you could have done yourself ;) ). If there are some new options which would be helpful for the distros recipe, like an option to rename the downloaded folder you can add those...
> At the moment, the recipe does not rely on the Plone egg, because Daniel > and I were unable to get that to install cleanly in buildout (for > whatever reason, CMFPlone and friends were not being initialised and > thus couldn't be added in the ZMI). The Plone egg also suffers a bit > because it has a static list of eggs but tracks svn for Products. We > really need tags on the Products bundle + a tagged list of eggs for each > release.
Why would we need a tag on the products bundle (or better a tag which includes svn:externals to all tags of all products)? We have tags on all of the products and tarballs available for them, which is what is used by the dist_plone script. I don't see much value in having a way to do an SVN checkout of a particular release, that's what releases and tarballs are made for.
> I probably won't make a cheese shop release until plone.recipe.distros > is re-released (given that it has a bug which causes the buildout to > crash, though I've fixed it I think).
Release is out, so feel free to make a release or just a development snapshot available of p.r.p.
> I plan to keep the versions of releases of plone.recipe.plone in sync > with Plone versions, so the current version would be 3.0b3. The release > cycle would probably be:
> - Update and tag dist_plone > - Switch the external in plone.recipe.plone from dist_plone trunk to > that tag > - Tag plone.recipe.plone > - Do the cheese shop dance
> People should be able to control which release of Plone they get with:
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Plone-developers mailing list Plone-develop...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers
> Nice! This is probably a n00b error but I get: http://paste.plone.org/15215 > Something about: > [Errno 2] No such file or directory: > '/private/var/zope/plone3/parts/plone/CMFActionIcons'
This is the bug I fixed in plone.recipe.distros. Wait for a new release or install that too as a development egg.
MArtin
-- Acquisition is a jealous mistress
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Plone-developers mailing list Plone-develop...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers
> Note that at least the Windows installer, while still using package > tarballs for eggs, actually installs setuptools in the local Python > environment and installs all packages in Python's site-packages (it > calls easy_install on all of them) instead of putting them into the > instances lib/python. So at least on Windows we already use proper eggs :)
Does that mean we *shouldn't* do this in the buildout on Windows? I suppose it's only a problem if someone uses Python + Zope from the win installer but a separate buildout (which is supported).
>> Hanno: It uses plone.recipe.distros to download all the tarballs. It'd >> be great if you could make a new release of that, since I just fixed a >> bug in it. :) There's also some XXX comments in the recipe source that >> describe some difficulties in translating the feature set of dist_plone >> to that of plone.recipe.distros.
> I just made a new release (which you could have done yourself ;) ).
Ah, right. Heh, sorry. ;-)
> If > there are some new options which would be helpful for the distros > recipe, like an option to rename the downloaded folder you can add those...
:-p
The code there scares me every time I look at it. My time is severely restricted at the moment, so I'm trying not to be a perfectionist, but if we need it, I'm sure it won't be hard to add.
>> At the moment, the recipe does not rely on the Plone egg, because Daniel >> and I were unable to get that to install cleanly in buildout (for >> whatever reason, CMFPlone and friends were not being initialised and >> thus couldn't be added in the ZMI). The Plone egg also suffers a bit >> because it has a static list of eggs but tracks svn for Products. We >> really need tags on the Products bundle + a tagged list of eggs for each >> release.
> Why would we need a tag on the products bundle (or better a tag which > includes svn:externals to all tags of all products)? We have tags on all > of the products and tarballs available for them, which is what is used > by the dist_plone script. I don't see much value in having a way to do > an SVN checkout of a particular release, that's what releases and > tarballs are made for.
Yes, except the Plone egg pulls in Products/ (and makes it a namespace package). It does that with svn:externals, since it doesn't happen on the fly during install, it happens before the egg is uploaded to the cheese shop.
If course, we could manually extract the appropriate tarball, or script it, before making a Plone egg release. We could even pull that into the dist_plone environment.
>> I probably won't make a cheese shop release until plone.recipe.distros >> is re-released (given that it has a bug which causes the buildout to >> crash, though I've fixed it I think).
> Release is out, so feel free to make a release or just a development > snapshot available of p.r.p.
>> I plan to keep the versions of releases of plone.recipe.plone in sync >> with Plone versions, so the current version would be 3.0b3. The release >> cycle would probably be:
>> - Update and tag dist_plone >> - Switch the external in plone.recipe.plone from dist_plone trunk to >> that tag >> - Tag plone.recipe.plone >> - Do the cheese shop dance
>> People should be able to control which release of Plone they get with:
You'll notice that it's really your nice work. The recipe doesn't do much except call zc.recipe.egg and plone.recipe.distros with parameters sucked from dist_plone. ;-)
Martin
-- Acquisition is a jealous mistress
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Plone-developers mailing list Plone-develop...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers
Martin Aspeli wrote: >> Release is out, so feel free to make a release or just a development >> snapshot available of p.r.p.
> Cool, will do.
I also made a ZopeSkel release.
You should now be able to do this:
$ easy_install ZopeSkel # try to do ZopeSkel>=0.4 if that fails $ paster create -t plone3_buildout myproject $ cd myproject $ python bootstrap.py $ ./bin/buildout
Which should install Plone 3 beta 3. And then:
$ ./bin/instance fg
To run it.
Martin
-- Acquisition is a jealous mistress
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Plone-developers mailing list Plone-develop...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers
Martin Aspeli wrote: >> Note that at least the Windows installer, while still using package >> tarballs for eggs, actually installs setuptools in the local Python >> environment and installs all packages in Python's site-packages (it >> calls easy_install on all of them) instead of putting them into the >> instances lib/python. So at least on Windows we already use proper eggs :)
> Does that mean we *shouldn't* do this in the buildout on Windows? I > suppose it's only a problem if someone uses Python + Zope from the win > installer but a separate buildout (which is supported).
I don't mean we need to change anything here. This was just a simple remark, that some of our installers already handle eggs in some way.
>> If >> there are some new options which would be helpful for the distros >> recipe, like an option to rename the downloaded folder you can add those...
> :-p
> The code there scares me every time I look at it. My time is severely > restricted at the moment, so I'm trying not to be a perfectionist, but > if we need it, I'm sure it won't be hard to add.
The best thing you could do is start writing tests for these recipes. I myself find tested code much less scary than untested code ;) But I know that dreaded time restriction problem all to well... :(
>>> At the moment, the recipe does not rely on the Plone egg, because Daniel >>> and I were unable to get that to install cleanly in buildout (for >>> whatever reason, CMFPlone and friends were not being initialised and >>> thus couldn't be added in the ZMI). The Plone egg also suffers a bit >>> because it has a static list of eggs but tracks svn for Products. We >>> really need tags on the Products bundle + a tagged list of eggs for each >>> release. >> Why would we need a tag on the products bundle (or better a tag which >> includes svn:externals to all tags of all products)? We have tags on all >> of the products and tarballs available for them, which is what is used >> by the dist_plone script. I don't see much value in having a way to do >> an SVN checkout of a particular release, that's what releases and >> tarballs are made for.
> Yes, except the Plone egg pulls in Products/ (and makes it a namespace > package). It does that with svn:externals, since it doesn't happen on > the fly during install, it happens before the egg is uploaded to the > cheese shop.
Hhm, that Products is a namespace package thing should really be handled on the right level first. Zope itself should declare it a namespace package and include the __init__.py in the skeleton for new instances. Unless we have at least that it is a matter of luck (or better who comes first on the PYTHONPATH) if the namespace declaration actually works.
In addition it currently isn't possible to have products distributed as eggs register themselves as Zope2 products: five:registerPackage which would tell Zope to load the initialize method is broken in ZEO setup's and you also cannot put it into a product that is still delivered as a tarball, as the initialize method is called twice than, with all kinds of funny and not so funny side effects...
I think we need to fix that story first before advocating it in any way.
>>> Thoughts? >> Really nice work!!!
> ;-)
> You'll notice that it's really your nice work. The recipe doesn't do > much except call zc.recipe.egg and plone.recipe.distros with parameters > sucked from dist_plone. ;-)
Putting it all together is sometimes harder than writing the single pieces :)
Hanno
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Plone-developers mailing list Plone-develop...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers
Hanno Schlichting wrote: > Hhm, that Products is a namespace package thing should really be handled > on the right level first. Zope itself should declare it a namespace > package and include the __init__.py in the skeleton for new instances. > Unless we have at least that it is a matter of luck (or better who comes > first on the PYTHONPATH) if the namespace declaration actually works.
True.
> In addition it currently isn't possible to have products distributed as > eggs register themselves as Zope2 products: five:registerPackage which > would tell Zope to load the initialize method is broken in ZEO setup's > and you also cannot put it into a product that is still delivered as a > tarball, as the initialize method is called twice than, with all kinds > of funny and not so funny side effects...
> I think we need to fix that story first before advocating it in any way.
Hold on ... does that apply to things not in Products.* also?
We have lots of eggs in Plone 3 that need <five:registerPackage />, e.g. because they need skins or an extension profile.
Are these broken with ZEO too?
Martin
-- Acquisition is a jealous mistress
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Plone-developers mailing list Plone-develop...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers
>> In addition it currently isn't possible to have products distributed as >> eggs register themselves as Zope2 products: five:registerPackage which >> would tell Zope to load the initialize method is broken in ZEO setup's >> and you also cannot put it into a product that is still delivered as a >> tarball, as the initialize method is called twice than, with all kinds >> of funny and not so funny side effects...
>> I think we need to fix that story first before advocating it in any way.
> Hold on ... does that apply to things not in Products.* also?
> We have lots of eggs in Plone 3 that need <five:registerPackage />, e.g. > because they need skins or an extension profile.
At least those things can and should be done with zcml statements.
> Are these broken with ZEO too?
The only purpose of five:registerPackage is to call the initialize method of a package even, if it is not in a Products folder configured in your zope.conf. That's something you shouldn't really do in a Python package in the first place.
The bug description is here: http://dev.plone.org/plone/ticket/6476 which also points to the underlying Zope bug report which has been marked as wontfix.
The only thing that doesn't work is to create a new ZEO server once a package that uses five:registerPackage is present. If you copy over the Data.fs from a different server, all seems to work.
Still you need to expect that the code in your initialize method is being called twice...
Hanno
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Plone-developers mailing list Plone-develop...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers
> I've been grappling with how best to install Plone as a dependency in a > buildout, and I think I've finally settled on a solution. The main idea > here is that dist_plone.platforms.independent is the canonical place to > define which products and eggs make up a release. This means that the > recipe is probably not so useful for tracking svn, but that's not what > it's designed for.
My feeling is that dist_plone.platforms.independent should be phased out and replaced by a setup.py. Is this reasonable for 3.5?
> People should be able to control which release of Plone they get with:
I wish we could find out what the problem is with getting the Plone egg working in buildout. Because
eggs = Plone>=3.0-dev,<3.1
still looks nicer to me. Not being able to look at plone.recipe.plone, I have no idea what it does, however, I can't imagine it's much more than a workaround?
Regards Daniel
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Plone-developers mailing list Plone-develop...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers
Hanno Schlichting wrote: > Why would we need a tag on the products bundle (or better a tag which > includes svn:externals to all tags of all products)? We have tags on all > of the products and tarballs available for them, which is what is used > by the dist_plone script. I don't see much value in having a way to do > an SVN checkout of a particular release, that's what releases and > tarballs are made for.
svn:externals in a tagged bundle is what I use in AT right now. It's super useful to see what has changed compared to trunk versions (I can svn diff all lines in EXTERNALS.txt against their /trunk equivalents). Also, an svn export of that bundle actually *is* the release. Not sure how useful this would be for Plone itself.
Martin and I originally thought that it would be useful to have for the Plone egg, since that pulls in Products/ right now via svn:externals. But I guess relying on dist_plone or whatever to produce that Products directory is fine too.
Daniel
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Plone-developers mailing list Plone-develop...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers
>> I've been grappling with how best to install Plone as a dependency in a >> buildout, and I think I've finally settled on a solution. The main idea >> here is that dist_plone.platforms.independent is the canonical place to >> define which products and eggs make up a release. This means that the >> recipe is probably not so useful for tracking svn, but that's not what >> it's designed for.
> My feeling is that dist_plone.platforms.independent should be phased out and > replaced by a setup.py. Is this reasonable for 3.5?
Maybe, but it probably can't be a "simple" setup.py. It needs to reference tarballs for products and eggs. As I understand, this is a requirement for the installers.
>> People should be able to control which release of Plone they get with:
> I wish we could find out what the problem is with getting the Plone egg > working in buildout. Because
> eggs = > Plone>=3.0-dev,<3.1
> still looks nicer to me.
True.
> Not being able to look at plone.recipe.plone, I > have no idea what it does, however, I can't imagine it's much more than a > workaround?
It does three things:
- Specify a Zope tarball. This means that (optionally) you can get a "known good" version of Zope in the buildout environment, which is similar to what the installers do. The nice thing about that is that if you update Plone and the new version requires/prefers a new version of Zope, you get that too.
- Add all of Plone's eggs to the working set, pegged at the versions from dist_plone. I'm working on something right now which lets you override that with develop eggs if you need to track subversion for one or more dependent eggs (well, you can do that now of course, but I want to make it a bit easier).
- Download and install all the product tarballs.
I'm not convinced that having the Plone egg ship with a Products/__init__.py that declares it as a namespace package can work reliably until Zope does that for us. Otherwise, as Hanno points out, I believe the first thing on the PYTHONPATH will win. Also, $SOFTWARE_HOME/Products/__init__.py does set some variables which serve a purpose.
If we can make the Plone egg work reliably, then plone.recipe.plone still has a purpose, but it can get a little simpler: Instead of specifying all eggs, it can just add Plone==3.0 or whatever to the working set and (when appropriate) stop downloading tarballs of products. To the user of the recipe, there won't be a difference, so we have some insulation from future changes in Plone's packaging and egg story.
Martin -- Acquisition is a jealous mistress
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Plone-developers mailing list Plone-develop...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers
Martin Aspeli wrote: > I'm not convinced that having the Plone egg ship with a > Products/__init__.py that declares it as a namespace package can work > reliably until Zope does that for us. Otherwise, as Hanno points out, I > believe the first thing on the PYTHONPATH will win. Also, > $SOFTWARE_HOME/Products/__init__.py does set some variables which serve > a purpose.
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Plone-developers mailing list Plone-develop...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers
