setuptools integration
Christopher Lenz
I'm planning to start working on the move to setuptools again. To
kickstart it, I've posted a summary to the wiki:
<http://trac.edgewall.org/wiki/TracDev/Proposals/Setuptools>
Feedback welcome, in particular on the last part:
> Because this proposal requires moving many files around
> (causing pain when merging), I would prefer doing these
> changes in one big changeset directly on trunk instead
> of updating the branch.
Cheers,
Chris
--
Christopher Lenz
cmlenz at gmx.de
http://www.cmlenz.net/
Ilias Lazaridis
Christopher Lenz wrote:
> Hey all,
>
> I'm planning to start working on the move to setuptools again. To
> kickstart it, I've posted a summary to the wiki:
>
> <http://trac.edgewall.org/wiki/TracDev/Proposals/Setuptools>
>
> Feedback welcome,
It would be important to keep the following requirement in account:
* vendors should be able to provide a "product" (collection of plugins
and related configuration)
http://trac.edgewall.org/ticket/4446
* trac should be able to work with an simple svn checkout.
http://dev.lazaridis.com/base/wiki/TracxVsTrac
* Environment Inheritance should be multi-level, with the ability to
change the base environment manually.
http://dev.lazaridis.com/base/wiki/PlanEnvInheritance
> in particular on the last part:
>
> > Because this proposal requires moving many files around
> > (causing pain when merging), I would prefer doing these
> > changes in one big changeset directly on trunk instead
> > of updating the branch.
Many tiny changesets directly on trunk should be prefered, whenever
possible, thus development is kept trackable.
.
John Hampton
> * vendors should be able to provide a "product" (collection of plugins
> and related configuration)
Why can't a vendor simply ship additional plugins as is? As I see it,
in order for a vendor to create a "product" there are two options.
1. One, simply a set of packages that are installed along with trac
2. A trac install plus a host of plugins, plus a fully configured env
On point 1, I don't see why a vendor can't do that currently. Speaking
for gentoo experience (as that's what I'm most familiar with) I would
simply have the ebuild download and install extra plugins. Due to setup
utils, I wouldn't even have to install trac. I'd simply install a
plugin and it would automatically pull down the required version of trac.
On point 2, I don't think this is a job for trac to worry about. First,
it would require that trac take care of creating the database for the
environment. Obviously, it currently does this for sqlite, but there is
no clean way to do this for PostgreSQL (nor would many PostgreSQL admins
want it to). You're also assuming a lot about the configuration of the
web server, env file layout, etc. If a vendor really wants to do all
this, they can. Bash is a great language for this type of thing.
> * trac should be able to work with an simple svn checkout.
While I'm not averse to this ability, I don't think that it's a show
stopper, or even one that really matters. Sure, it'd be neat, and maybe
it's just a simple byproduct of the other stuff, but most programs don't
simply work from a svn checkout.
Having a checkout/compile cycle is standard practice. Even then, once
it's setuptoolized, one will be able to simply say:
easy_install http://domain.com/path/to/repos
and have it install.
> * Environment Inheritance should be multi-level, with the ability to
> change the base environment manually.
This one I'm not sure I fully understand. If you're just talking about
config options, then it appears that cmlenz's proposal is sufficient.
Templates is a decent idea, though I think one level of inheritance
would be sufficient (unless it's trivial from there to extend it to N).
Cascading bunches of templates would probably get very confusing.
Ddefining one coherent theme for a group of trac sites easily would be
nice. However, I don't really see how this is that much different than
the ability to use site_* templates or override a default template by
moving a file with said name into the templates/ dir of the local env.
Now, if you're talking about inheriting config, templates, and database
content, then I think you're crazy. Mainly because inheriting database
content is much more along the lines of multi-project support. Trying
to cascade a bunch of databases together wouldn't be fun.
> Many tiny changesets directly on trunk should be prefered, whenever
> possible, thus development is kept trackable.
I see one big merge just as trac-able[1] as many small changes. I think
the main point was that the merge is a pain, and trying to do it in
small changes would result in more breakage of trunk, or much more work
to accommodate for the small changes.
-John
[1] Yes, pun intended.
Noah Kantrowitz
Hash: SHA1
On Jan 18, 2007, at 7:59 PM, Christopher Lenz wrote:
>
> Hey all,
>
> I'm planning to start working on the move to setuptools again. To
> kickstart it, I've posted a summary to the wiki:
+1 from on everything except the tracext namespace. With the global
plugins folder in 0.10 I don't think this is needed anymore. If
someone chooses to use site-packages over the global folder, that is
their call.
- --Noah
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
iD8DBQFFsFdJA9fEp2eRsU8RAn/FAJoC9JQkfexshELyvF8vTLKOku7PzwCg6q/o
0rjTTNDiey9wc0QOHjJVxeA=
=jg/C
-----END PGP SIGNATURE-----
John Hampton
> +1 from on everything except the tracext namespace. With the global
> plugins folder in 0.10 I don't think this is needed anymore. If someone
> chooses to use site-packages over the global folder, that is their call.
I'm definitely +1 on the merge too. However, why not add tracext? I
like the idea of having third party plugins use a different namespace,
even though it's not technically *needed*. Any reason why adding
tracenv is a bad idea with the advent of the global plugins folder?
-John
Christian Boos
>
> Hey all,
>
> I'm planning to start working on the move to setuptools again. To
> kickstart it, I've posted a summary to the wiki:
>
> <http://trac.edgewall.org/wiki/TracDev/Proposals/Setuptools>
Sure, everyting looks good in there.
>
> Feedback welcome, in particular on the last part:
>
> > Because this proposal requires moving many files around
> > (causing pain when merging), I would prefer doing these
> > changes in one big changeset directly on trunk instead
> > of updating the branch.
>
I hope you're not talking about doing that *before* the other pending
branches get merged?
-- Christian
Christopher Lenz
> Christopher Lenz wrote:
>> Feedback welcome, in particular on the last part:
>>
>> > Because this proposal requires moving many files around
>> > (causing pain when merging), I would prefer doing these
>> > changes in one big changeset directly on trunk instead
>> > of updating the branch.
>>
>
> I hope you're not talking about doing that *before* the other
> pending branches get merged?
I'm not sure how much of a problem that'd be. The "moving around" is
exclusively about templates, htdocs, and the default wiki pages, i.e.
not the code. Do those branches have any/many changes in those areas?
Ilias Lazaridis
> Ilias Lazaridis wrote:
> > * vendors should be able to provide a "product" (collection of plugins
> > and related configuration)
> > * trac should be able to work with an simple svn checkout.
[...] (comments)
> > * Environment Inheritance should be multi-level, with the ability to
> > change the base environment manually.
[...] (comments)
(I apologize for not answering in detail)
The requirements result out of a real-life use-case. The resulting
requirements are of general nature - their implementation would
increase the flexibility of the trac framework.
Python is a dynamic language - thus it should work like that: dynamic,
if needed. Setuptools can become an "innovation cancer", if it is used
without care.
The Requirement:
Allow svn checkout of different trac development constellations (trac
version, several plugins and a specific config file).
Practical use-case:
svn co http://dev.lazaridis.com/base/svn/infra
(retrieves some plugins as svn externals, using a specified revision)
Manual steps are needed (collected into one script):
http://dev.lazaridis.com/base/browser/infra/prepare.py?rev=174
Most of those manual steps _can_ be _eliminated_:
http://trac.edgewall.org/ticket/4317
[PATCH] - CORE - Small modifications for loader
http://trac.edgewall.org/ticket/4383
CORE - Loader should load single-file plugins after egg plugins
http://trac.edgewall.org/ticket/4295
CORE - Provide a mechanism to use custom ini files
... (other tiny tasks, which were not filed)
(as said, this is already implemented with the "tracx" plugin, which is
essentially a trac based "product", which can be retrieved from the svn
in _different_ versions (including dependencies and configuration, thus
ready to go)
-
http://trac.edgewall.org/ticket/4446
CORE - Implement Ability to Provide Trac "Products" (plugin bundles)
It is possibly of benefit to address this ticket _after_ the setuptools
stuff is moved into trunk.
I think it's of benefit to work towards this direction, in order to
allow _developers_ to easily fetch a more complete setup directly from
svn, with the ability to switch easily between different version.
If developer A has a problem and needs a peer-review, other developers
can fetch the _complete_ product, activate it, make the tests, and then
reactivate their own version again. (3 cmd-lines).
.
Christian Boos
>
> Am 19.01.2007 um 08:56 schrieb Christian Boos:
>> Christopher Lenz wrote:
>>> Feedback welcome, in particular on the last part:
>>>
>>> > Because this proposal requires moving many files around
>>> > (causing pain when merging), I would prefer doing these
>>> > changes in one big changeset directly on trunk instead
>>> > of updating the branch.
>>>
>>
>> I hope you're not talking about doing that *before* the other
>> pending branches get merged?
>
> I'm not sure how much of a problem that'd be. The "moving around" is
> exclusively about templates, htdocs, and the default wiki pages, i.e.
> not the code. Do those branches have any/many changes in those areas?
Lots of changes in the templates, yes.
Given svn's inability to assist the merge after a move, this will be
quite painful to do manually.
I don't see the need to rush for doing that move *right now*, given that
those branches (security, timeline-link-resolver-tmp and
property-renderer-tmp) are in a "ready to merge" status.
-- Christian
Emmanuel Blot
>
> While I'm not averse to this ability, I don't think that it's a show
> stopper, or even one that really matters.
> it's setuptoolized, one will be able to simply say:
>
> easy_install http://domain.com/path/to/repos
>
> and have it install.
I would have liked to keep this feature as well: it is really nice not
to have an extra step (bypassing trac install) as it has been possible
up to now.
"installing" means that you have a copy of the actual files, which
implies that if you change the files in place - i.e. the installed
files - you cannot simply 'svn commit' them back to the repository
I'm not sure how many users do the same, but I have a customized
release of Trac (which cannot be easily implemented as plugins) that
is managed in a dedicated SVN repository. I would have liked being
able to develop and run Trac from the same location, since it's far
easier to manage than having a copy of the SVN files.
Maybe I missed something important here, and that this will still be
possible with the "python setup.py develop" thing. Please - let me
know.
Up to now, setuptools have really be a pain in the a... for me: issues
with their direct SVN internal data file parsing (move to svn 1.4),
deadlock at plugin load time caused by the site.py file, compatiblity
issues, problemS with proxy for installation over the network and many
other troubles that made me lose hours of work.
I do feel that setuptools is the way to go though: its ability to
manage dependencies is a great feature, especially for Trac and its
large and growing plugin base.
So I'm +1 for the merge, however I would have liked not to add another
trouble with the install/copy step.
Cheers,
Manu
Christian Boos
> ...
> Maybe I missed something important here, and that this will still be
> possible with the "python setup.py develop" thing. Please - let me
> know.
I see no reasons why this couldn't be supported, and, as you imply, it's
a "must have".
-- Christian
Noah Kantrowitz
Hash: SHA1
1. It is an abuse of the namespacing stuff in setuptools. All plugins
are not part of a single package.
2. Redundancy is bad. This offers no new functionality really.
3. It makes it adds one more step to making a plugin.
4. It makes it somewhat harder to interface "normal" code with
plugins, as the namespacing system in setuptools is rather weird.
- --Noah
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
iD8DBQFFsq7IA9fEp2eRsU8RAl61AKCOeeKAErIWoUgsQC6JkGGwzXlX2gCg2mfg
PhubDnWRwp9vfqdKntz2Zt0=
=qkKs
-----END PGP SIGNATURE-----
Christopher Lenz
> On Jan 19, 2007, at 1:05 AM, John Hampton wrote:
>> I'm definitely +1 on the merge too. However, why not add tracext? I
>> like the idea of having third party plugins use a different
>> namespace,
>> even though it's not technically *needed*. Any reason why adding
>> tracenv is a bad idea with the advent of the global plugins folder?
>
> 1. It is an abuse of the namespacing stuff in setuptools. All plugins
> are not part of a single package.
The idea was to avoid having hundreds of plugins using naming
conventions such as trac_xyz or tracxxx poluting the Python root
package namespace (e.g. imagine pydoc output with many plugins
installed.)
I don't actually care much about this, myself, and would be
absolutely willing to drop it from the proposal.
> 2. Redundancy is bad. This offers no new functionality really.
What does that have to do with redundancy?
> 3. It makes it adds one more step to making a plugin.
Using the namespace would of course be completely optional.
> 4. It makes it somewhat harder to interface "normal" code with
> plugins, as the namespacing system in setuptools is rather weird.
Can you elaborate on this point? What kind of problems have you
encountered with namespace packages?
Thanks,
Noah Kantrowitz
Hash: SHA1
On Jan 25, 2007, at 1:45 PM, Christopher Lenz wrote:
>
> Am 21.01.2007 um 01:07 schrieb Noah Kantrowitz:
>> On Jan 19, 2007, at 1:05 AM, John Hampton wrote:
>>> I'm definitely +1 on the merge too. However, why not add
>>> tracext? I
>>> like the idea of having third party plugins use a different
>>> namespace,
>>> even though it's not technically *needed*. Any reason why adding
>>> tracenv is a bad idea with the advent of the global plugins folder?
>>
>> 1. It is an abuse of the namespacing stuff in setuptools. All plugins
>> are not part of a single package.
>
> The idea was to avoid having hundreds of plugins using naming
> conventions such as trac_xyz or tracxxx poluting the Python root
> package namespace (e.g. imagine pydoc output with many plugins
> installed.)
The global plugins folder already does this.
>
> I don't actually care much about this, myself, and would be
> absolutely willing to drop it from the proposal.
>
>> 2. Redundancy is bad. This offers no new functionality really.
>
> What does that have to do with redundancy?
It would be redundant with the global folder as far as limiting
namespace pollution.
>
>> 3. It makes it adds one more step to making a plugin.
>
> Using the namespace would of course be completely optional.
>
>> 4. It makes it somewhat harder to interface "normal" code with
>> plugins, as the namespacing system in setuptools is rather weird.
>
> Can you elaborate on this point? What kind of problems have you
> encountered with namespace packages?
Anyone accidently putting code in the __init__.py in the namespace
folder (or anything else at level) may be confused at first if they
don't read up on how the namespace system works. Ditto for if the
forget to call the declare_namespace() function. For someone not used
to working Python, which is the case for many new plugin developers,
this all might be a bit much.
- --Noah
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
iD8DBQFFuZBGA9fEp2eRsU8RAkDyAJ9HLJ6q6d6Occ4r1MIx7jHRM++vuQCg6ZJ9
B/lFcaGAnLIuFfXBDYKFMns=
=EPuV
-----END PGP SIGNATURE-----
Matt Good
> > Can you elaborate on this point? What kind of problems have you
> > encountered with namespace packages?
>
> Anyone accidently putting code in the __init__.py in the namespace
> folder (or anything else at level) may be confused at first if they
> don't read up on how the namespace system works. Ditto for if the
> forget to call the declare_namespace() function. For someone not used
> to working Python, which is the case for many new plugin developers,
> this all might be a bit much.
Well, I've been thinking it'd be nice to work on a nice "quickstart"
tool for plugins, probably based on the PasteScript package
generator. This could put a big warning in the namespace __init__.py
that it should not be edited. I don't think people need to fully
understand the workings of namespacing here as long as they have a
good starting point.
However, if the namespace stuff is still controversial we could put
that aside temporarilly, but I'd like to see the setuptools stuff
merged soon[1]. So, are there any other issues that need resolved
before this is merged?
-- Matt Good
[1] for ticket #4382 I'd like to use simplejson to handle the encoding
of Python datastructures to JavaScript, so I'd like to add it as a
setuptools requirement
Christian Boos
> On Jan 26, 12:23 am, Noah Kantrowitz <kan...@rpi.edu> wrote:
>
>>> Can you elaborate on this point? What kind of problems have you
>>> encountered with namespace packages?
>>>
>> Anyone accidently putting code in the __init__.py in the namespace
>> folder (or anything else at level) may be confused at first if they
>> don't read up on how the namespace system works. Ditto for if the
>> forget to call the declare_namespace() function. For someone not used
>> to working Python, which is the case for many new plugin developers,
>> this all might be a bit much.
>>
>
> Well, I've been thinking it'd be nice to work on a nice "quickstart"
> tool for plugins, probably based on the PasteScript package
> generator. This could put a big warning in the namespace __init__.py
> that it should not be edited. I don't think people need to fully
> understand the workings of namespacing here as long as they have a
> good starting point.
>
Or even a sample plugin would be enough to show the right way, with
appropriate comments, "change this", "don't modify that", etc. Last
time I was using the namespace stuff for the vc-refactoring, it was
working fine.
> However, if the namespace stuff is still controversial we could put
> that aside temporarilly, but I'd like to see the setuptools stuff
> merged soon[1]. So, are there any other issues that need resolved
> before this is merged?
>
As long as the setuptools changes are easy to merge in other branches
(i.e. they don't involve any moving of files around), I'm OK for the
setuptools related enhancements to go in trunk.
-- Christian
Christopher Lenz
> Matt Good wrote:
>> However, if the namespace stuff is still controversial we could put
>> that aside temporarilly, but I'd like to see the setuptools stuff
>> merged soon[1]. So, are there any other issues that need resolved
>> before this is merged?
>
> As long as the setuptools changes are easy to merge in other branches
> (i.e. they don't involve any moving of files around), I'm OK for the
> setuptools related enhancements to go in trunk.
So that's a "no", as you *know* that integrating the setuptools stuff
will require moving files around.
What "other branches" are we talking about anyway? I've lost track,
you seem to have merged half of the security branch already, and your
other branches are already in too AFAICT, or am I missing something
that would be horribly affected by the setuptools integration?
Cheers,
Christian Boos
> Am 11.02.2007 um 11:26 schrieb Christian Boos:
>
>> As long as the setuptools changes are easy to merge in other branches
>> (i.e. they don't involve any moving of files around), I'm OK for the
>> setuptools related enhancements to go in trunk.
>>
>
> So that's a "no", as you *know* that integrating the setuptools stuff
> will require moving files around.
>
I was then referring to setuptools improvements like #3229, using
setuptools in setup.py, the namespace support, etc. I'm not sure if the
moving is required for all of these, I think it will be only needed for
the data packaging.
> What "other branches" are we talking about anyway? I've lost track,
> you seem to have merged half of the security branch already, and your
> other branches are already in too AFAICT, or am I missing something
> that would be horribly affected by the setuptools integration?
>
Well, I did that merge just _after_ sending the mail you've responded to
;-)
Now there are only 7 files in templates/, with not many changes in.
So feel free to do the required changes for the setuptools support.
-- Christian