0.12beta getting closer
Christian Boos
We're getting quite close to being able to release a beta of 0.12. While
there are still a few important tickets to be finished first (#9060 for
example), we're really getting there.
I'm going to update the Trac Guide on t.e.o in a few days, moving the
0.12/ pages to the toplevel. The current toplevel pages will be archived
below 0.11/. It would be nice to get some proof reading first, see
http://trac.edgewall.org/wiki/0.12 (for some pages it has already been
done, check the history).
Actually, that pre-release version of 0.12 could even be called 0.12rc1,
as I don't expect many changes till the final release. Also, we may
still want to release 0.11.8 in parallel, as I messed up a few things
with 0.11.7 (e.g. r9391, to be backported).
-- Christian
W. Martin Borgert
> We're getting quite close to being able to release a beta of 0.12.
> While there are still a few important tickets to be finished first
> (#9060 for example), we're really getting there.
AFAIK, 0.12 needs Genshi > 0.5.1[1], but 0.6 is not yet released[2].
For Debian (and other distribution as well, I assume) it would be
nice to have a Genshi release before Trac RCs. Are there any plans
to synchronise Genshi 0.6 and Trac 0.12 releases? Thanks in advance!
[1] http://trac.edgewall.org/milestone/0.12
[2] http://genshi.edgewall.org/wiki/Download
Btw. keep up the excellent work on Trac!
W. Martin Borgert
> Actually, that pre-release version of 0.12 could even be called
> 0.12rc1, as I don't expect many changes till the final release.
> Also, we may still want to release 0.11.8 in parallel, as I messed
> up a few things with 0.11.7 (e.g. r9391, to be backported).
Is there a plan when 0.11.8 can be released? I ask, because Debian
is about to freeze (don't hold your breath) and it would be nice
to have 0.11.8 in. Thanks in advance!
Christian Boos
> Quoting "Christian Boos" <cb...@neuf.fr>:
>> We're getting quite close to being able to release a beta of 0.12.
>> While there are still a few important tickets to be finished first
>> (#9060 for example), we're really getting there.
>
> AFAIK, 0.12 needs Genshi > 0.5.1[1], but 0.6 is not yet released[2].
> For Debian (and other distribution as well, I assume) it would be
> nice to have a Genshi release before Trac RCs.
Yes, it would be nice.
> Are there any plans to synchronise Genshi 0.6 and Trac 0.12 releases?
> Thanks in advance!
There's no plan, only hope ;-)
More seriously, I'm not really at ease with the fact that either Genshi
and Babel are more or less on life support, since Christopher has
switched to other interests. Pedro Algarvio seems our best chance here
;-) Pedro?
Related:
- http://genshi.edgewall.org/ticket/369
- http://babel.edgewall.org/ticket/212
Another solution would be to build our own packages, trac-genshi and
trac-babel.
-- Christian
Christian Boos
No, not yet.
But besides the change I was talking about (#9103), I see only #9128, +
fixing a glitch in the RELEASE note (thanks Anatoly), + a fix for a
(tiny) security issue that has been reported meanwhile. We could even
call that 0.11.7.1, as the scope is rather small. That release could
happen "soon" as well.
-- Christian
Greg Troxel
Christian Boos <cb...@neuf.fr> writes:
Speaking as someone using trac seriously, it seems scary to be talking
about a 0.12 release that depends on unreleased versions of other tools.
If that's really the case, then I'd say the other tools need to be
encouraged/helped to get to release or forked and released before 0.12
has an rc. When you think about putting trac into a packaging system,
depending on unreleased code means creating a new -beta package for that
code from svn/git/whatever, or updating the allegedly stable package to
an unreleased version.
Christian Boos
> Christian Boos<cb...@neuf.fr> writes:
>
>> On 3/22/2010 2:45 PM, W. Martin Borgert wrote:
>>
>>> Quoting "Christian Boos"<cb...@neuf.fr>:
>>>
>>>> We're getting quite close to being able to release a beta of
>>>> 0.12. While there are still a few important tickets to be finished
>>>> first (#9060 for example), we're really getting there.
>>>>
>>> AFAIK, 0.12 needs Genshi> 0.5.1[1], but 0.6 is not yet released[2].
>>> For Debian (and other distribution as well, I assume) it would be
>>> nice to have a Genshi release before Trac RCs.
>>>
>> Yes, it would be nice.
>>
>>> Are there any plans to synchronise Genshi 0.6 and Trac 0.12
>>> releases? Thanks in advance!
>>>
>> There's no plan, only hope ;-)
>>
>> More seriously, I'm not really at ease with the fact that either
>> Genshi and Babel are more or less on life support, since Christopher
>> has switched to other interests. Pedro Algarvio seems our best chance
>> here ;-) Pedro?
>>
>> Related:
>> - http://genshi.edgewall.org/ticket/369
>> - http://babel.edgewall.org/ticket/212
>>
> Speaking as someone using trac seriously, it seems scary to be talking
> about a 0.12 release that depends on unreleased versions of other tools.
>
I understand and share your concerns.
> If that's really the case, then I'd say the other tools need to be
> encouraged/helped to get to release or forked and released before 0.12
> has an rc.
We have considered that. It looks like we're going to have a Babel 0.9.5
really soon (I see the packages on ftp.edgewall.org as I'm writing
this), and perhaps a Genshi 0.6 release at some later point (keep finger
crossed).
The fact remains that those libraries are currently strong dependencies
for Trac (Genshi even more so than Babel), and neither has currently the
level of maintenance we were used to have, every sign indicates this
trend won't change. I've also stopped to hope there will be any drastic
performance improvement, and while there has been some improvements last
year (1), I rather feel that this means that all that could be
realistically done has now been done on this topic. Of course,
theoretically someone could step up anytime and bring in a radically new
idea and dramatically increase performance, but that's precisely what
I've been hoping for since the last 4 years. While there has been such
attempts, made by bright minds, none really succeeded.
So don't get me wrong, I actually like Genshi, I was really pleased to
ditch the ugly and hard to maintain Clearsilver templates and
enthusiastically converted most of the Trac templates to it. I also
helped to fix (easy) Genshi issues during the early stage of its
development, but was always intimidated by its complexity. I still enjoy
very much *using* Genshi from a developer perspective, but for the
reasons exposed above, from the point of view of Trac maintenance and
its future evolution, we really have a situation to address, lots of
users are still hesitant to move away from Trac *0.10* because of
performance concerns (Genshi is perhaps not the only factor at play
there, but it's an important one).
It seems however that there could be a way forward, by considering once
again a switch of the templating engine. This time it would be in favor
of Jinja2 (2), which seems to have a great momentum and have the
adequate capabilities and speed we need. In addition, its main developer
is also a long time Trac supporter and has shown interest in supporting
us for the switch (hello Armin!).
There has been some informal discussions on this topic on the #trac IRC
channel, so the logical next step is to discuss it more formally here on
Trac-dev. To me at least, the status quo is not really an option. I'm
aware this will cause some inconvenience, especially for the plugins
that currently depend on stream filters. However this will be mitigated
by the fact that we can continue to support Genshi, and migrate to
Jinja2 only gradually, starting with the time critical parts (like the
changeset view, the browser view and the timeline).
-- Christian
(1) - http://genshi.edgewall.org/changeset/1038 and 1052
(2) - http://dev.pocoo.org/projects/jinja/
W. Martin Borgert
> Of course, theoretically someone could step up anytime and bring in
> a radically new idea
Like building Trac 0.13 completely on top of Django? :~)
Greg Troxel
Christian Boos <cb...@neuf.fr> writes:
> On 3/23/2010 12:43 AM, Greg Troxel wrote:
>> Speaking as someone using trac seriously, it seems scary to be talking
>> about a 0.12 release that depends on unreleased versions of other tools.
>
> I understand and share your concerns.
>
>> If that's really the case, then I'd say the other tools need to be
>> encouraged/helped to get to release or forked and released before 0.12
>> has an rc.
>
> We have considered that. It looks like we're going to have a Babel
> 0.9.5 really soon (I see the packages on ftp.edgewall.org as I'm
> writing this), and perhaps a Genshi 0.6 release at some later point
> (keep finger crossed).
>
> The fact remains that those libraries are currently strong
> dependencies for Trac (Genshi even more so than Babel), and neither
> has currently the level of maintenance we were used to have, every
> sign indicates this trend won't change. I've also stopped to hope
> there will be any drastic performance improvement, and while there has
> been some improvements last year (1), I rather feel that this means
> that all that could be realistically done has now been done on this
> topic. Of course, theoretically someone could step up anytime and
> bring in a radically new idea and dramatically increase performance,
> but that's precisely what I've been hoping for since the last 4
> years. While there has been such attempts, made by bright minds, none
> really succeeded.
Thanks for the detailed explanation. Everything you say sounds quite
reasonable in term of how trac got where it is and the way forward.
But, if Genshi doesn't have a release that can be used (reasonably) with
trac, then it should be a small matter of making a tarball from Genshi's
CM ystems and calling it Genshi-0.5.80 or GenshiTrac-0.5.80 if it's
socially awkward to call it Genshi-0.6. Then packaging systems can
either update their Genshi package or make a GenshiTrac package, and
there's no more packaging issue. (I realize this doesn't address the
lack of maintainer time issue. But if something in svn is good enough
and required, it should be declared a release before 0.12 is in RC.)
Christian Boos
That's why we won't after all release a 0.12rc1 directly, in the coming
days, but first make a 0.12beta. I hope it's more acceptable to have a
"beta" version depending on an unreleased package than a RC :-)
We'll then release the 0.12rc1 version after the Genshi 0.6 release.
But if it's still not released in a month or so from now, then we could
indeed integrate the genshi/ files as tracdep/genshi.* and make sure
that the Trac plugins importing the genshi modules will actually find
our tracdep.genshi ones (hopefully a `sys.modules['genshi'] =
tracdep.genshi` will be enough).
-- Christian
Andrea Tomasini
+1 I'll start doing it tomorrow if needed... too much technology to take care of now, and given the limited development capacity, wouldn't be that bad to focus on features more than technology :-)
Ciao
ANdreaT
Tim Hatch
> indeed integrate the genshi/ files as tracdep/genshi.* and make sure
> that the Trac plugins importing the genshi modules will actually find
> our tracdep.genshi ones (hopefully a `sys.modules['genshi'] =
> tracdep.genshi` will be enough).
I'll cross my fingers for a release instead. Mucking about with imports
or sys.path is bound to cause problems, as it does with Twill. There's
a thread on the genshi and babel lists about this.
Tim
Christian Boos
Well, I was talking about a new idea to improve Genshi performance...
Rewriting Trac on top of Django is a *very* different topic. While I
also would like to be able to focus on Trac features rather than on the
supporting technology, I have the feeling that doing a switch to Django
would do the exact opposite, make us focus again on the technology (and
eventually be limited by the constraints imposed upon us by the
framework) rather than on our features... We now have a base on which we
can build on, and if there wouldn't have been the problems related to
Genshi maintenance, I would even have been willing to close the eyes a
bit longer on the performance side and not have /any/ change in the
infrastructure for the next versions (besides a switch to Python 2.5,
but now it's me who's digressing ;-) ).
And there's already Basie, for those interested by a Trac-like system on
top of Django (http://basieproject.org/).
-- Christian
Christian Boos
... in which thread Remy already replied that the situation is
different, as we're loading Genshi first, then loading the plugins. And
I think it make more sense to talk about such hacks on this list rather
than on the Genshi list, so I'd prefer we continue that discussion here.
-- Christian
Noah Kantrowitz
While I agree that this is needed, we should be clear about the
impact; partially supporting Genshi doesn''t really mitigate not
support stream filters, since Genshi or not we can't filter a non-
Genshi and the whole point of stream filters is to allow UI changes on
other peoples' pages. The other question is how to support themes on
the converted pages, and I have no good answer for that.
--Noah
W. Martin Borgert
> But if it's still not released in a month or so from now, then we
> could indeed integrate the genshi/ files as tracdep/genshi.* and
> make sure that the Trac plugins importing the genshi modules will
> actually find our tracdep.genshi ones (hopefully a
> `sys.modules['genshi'] = tracdep.genshi` will be enough).
I think, this model is not an option for Debian nor for most
other Linux distributions. Debian, and surely others as well,
work very hard on removing embedded code copies. Adding a new
copy would be a stop in the wrong direction. Embedded code
copies make QA work, especially security fixes, very hard. Also,
they bloat the archive. They constitute a policy violation in
Debian. It would be preferable to find an alternative solution.
Christopher Lenz
> On 3/23/2010 12:43 AM, Greg Troxel wrote:
>> Speaking as someone using trac seriously, it seems scary to be talking
>> about a 0.12 release that depends on unreleased versions of other tools.
>
> I understand and share your concerns.
>
>> If that's really the case, then I'd say the other tools need to be
>> encouraged/helped to get to release or forked and released before 0.12
>> has an rc.
>
> We have considered that. It looks like we're going to have a Babel 0.9.5 really soon (I see the packages on ftp.edgewall.org as I'm writing this), and perhaps a Genshi 0.6 release at some later point (keep finger crossed).
The Babel maintenance release has made it out of the door today. I will do my best to get Genshi 0.6 out by next week.
To be honest, I'm not sure what's going to happen after that, and you're certainly right to be pessimistic. I'm really sorry about how the situation has turned out for Trac.
Yes, my interests have mostly shifted elsewhere. I still believe that both Babel and Genshi are worth being further maintained and enhanced, and I'm still interested to actually do the work, but obviously I'm not able to allot anywhere enough spare time to that task right now. What's more, I've unforunately been unable to attract other developers to contribute significantly to either code base, let alone build a strong community.
That's not to say that I consider either project end-of-life. I still use them for my own stuff. But I'm the pretty much the single point of failure for both projects, and I've been failing badly and consistently at maintaining/enhancing them for some time now. Sorry.
> The fact remains that those libraries are currently strong dependencies for Trac (Genshi even more so than Babel), and neither has currently the level of maintenance we were used to have, every sign indicates this trend won't change. I've also stopped to hope there will be any drastic performance improvement, and while there has been some improvements last year (1), I rather feel that this means that all that could be realistically done has now been done on this topic. Of course, theoretically someone could step up anytime and bring in a radically new idea and dramatically increase performance, but that's precisely what I've been hoping for since the last 4 years. While there has been such attempts, made by bright minds, none really succeeded.
>
> So don't get me wrong, I actually like Genshi, I was really pleased to ditch the ugly and hard to maintain Clearsilver templates and enthusiastically converted most of the Trac templates to it. I also helped to fix (easy) Genshi issues during the early stage of its development, but was always intimidated by its complexity. I still enjoy very much *using* Genshi from a developer perspective, but for the reasons exposed above, from the point of view of Trac maintenance and its future evolution, we really have a situation to address, lots of users are still hesitant to move away from Trac *0.10* because of performance concerns (Genshi is perhaps not the only factor at play there, but it's an important one).
>
> It seems however that there could be a way forward, by considering once again a switch of the templating engine. This time it would be in favor of Jinja2 (2), which seems to have a great momentum and have the adequate capabilities and speed we need. In addition, its main developer is also a long time Trac supporter and has shown interest in supporting us for the switch (hello Armin!).
>
> There has been some informal discussions on this topic on the #trac IRC channel, so the logical next step is to discuss it more formally here on Trac-dev. To me at least, the status quo is not really an option. I'm aware this will cause some inconvenience, especially for the plugins that currently depend on stream filters. However this will be mitigated by the fact that we can continue to support Genshi, and migrate to Jinja2 only gradually, starting with the time critical parts (like the changeset view, the browser view and the timeline).
I agree that adoption of Jinja2 should be considered, it's become a very solid templating solution, and comes with both more momentum and better performance than Genshi.
But I'm not sure how a gradual transition could work. As Noah said, you can't switch some of the most important pages to Jinja and still support stream filters. Or site templates using match templates for advanced customization. You'll also need to rethink parts of the internationalization story, I guess.
If there's going to be another template engine switch, I think it's going to hurt. But it might just be worth it.
Cheers,
--
Christopher Lenz
cmlenz at gmx.de
http://www.cmlenz.net/
Remy Blank
> I agree that adoption of Jinja2 should be considered, it's become a very
> solid templating solution, and comes with both more momentum and better
> performance than Genshi.
It's already a good sign that nobody has anything *bad* to say about
Jinja2. Still, is this the only reasonable alternative? Are there other
contenders out there, and what are the pros and cons of each solution?
I'm not trying to work against Jinja2, just to keep the view broad
before making a choice.
> But I'm not sure how a gradual transition could work.
(snip)
> If there's going to be another template engine switch, I think it's going
> to hurt. But it might just be worth it.
I'm a bit wary of how this could work out. Let's say (for the sake of
argument) that we switch to Jinja for 0.13. This is a large internal
change, but mostly invisible to end-users (except for better speed,
which may or may not have a significant impact). Besides, we either
break most of the existing plugins (if we don't transition gradually) or
increase the work significantly (if we transition gradually).
So, we get a new release where:
- a big chunk of work is barely visible to end-users
- plugins stop working for no (user-understandable) reason
- themes and style customizations stop working
I do hope that if we go this way (and maybe a radical but painful switch
with limited backward compatibility is the best solution), we can at
least bundle it with a few highly visible, high value features.
Improving internals is all well and good, but at the end of the day,
it's the user-visible features that the users care about.
-- Remy
Greg Troxel
Christian Boos <cb...@neuf.fr> writes:
> That's why we won't after all release a 0.12rc1 directly, in the
> coming days, but first make a 0.12beta. I hope it's more acceptable to
> have a "beta" version depending on an unreleased package than a RC :-)
Well, I have no standing to actually complain about what people
volunteer to do - just pointing out how it feels on the packaging end.
Sure, with a beta, it won't get packaged, so there's no issue.
> We'll then release the 0.12rc1 version after the Genshi 0.6 release.
That would be just fine from my viewpoint.
> But if it's still not released in a month or so from now, then we
> could indeed integrate the genshi/ files as tracdep/genshi.* and make
> sure that the Trac plugins importing the genshi modules will actually
> find our tracdep.genshi ones (hopefully a `sys.modules['genshi'] =
> tracdep.genshi` will be enough).
What the gentleman from Debian said :-)
John Hampton
> It's already a good sign that nobody has anything *bad* to say about
> Jinja2. Still, is this the only reasonable alternative? Are there other
> contenders out there, and what are the pros and cons of each solution?
>
> I'm not trying to work against Jinja2, just to keep the view broad
> before making a choice.
Breve [1], mako [2], and cheetah [3], off the top of my head.
> I'm a bit wary of how this could work out. Let's say (for the sake of
> argument) that we switch to Jinja for 0.13. This is a large internal
> change, but mostly invisible to end-users (except for better speed,
> which may or may not have a significant impact). Besides, we either
> break most of the existing plugins (if we don't transition gradually) or
> increase the work significantly (if we transition gradually).
>
> So, we get a new release where:
>
> - a big chunk of work is barely visible to end-users
> - plugins stop working for no (user-understandable) reason
> - themes and style customizations stop working
>
> I do hope that if we go this way (and maybe a radical but painful switch
> with limited backward compatibility is the best solution), we can at
> least bundle it with a few highly visible, high value features.
> Improving internals is all well and good, but at the end of the day,
> it's the user-visible features that the users care about.
Speed of the site being one of the biggest features that users care about ;)
If we change templating engines again (for which I'm +1 atm) then I
think our only real option is doing a full switch, no backwards
compatibility for the following reasons:
* There is no indication that a gradual transition will provide any
help with plugin transitions
* Even with a compatibility layer, it's not going to be 100% and we'll
probably end up breaking a bunch of plugins anyway
* We can be very clear about which plugins are supported. i.e. Only
plugins that specifically state they work on 0.13 will work
* Breaking everything on purpose will help bring out the issues early,
rather than mask them in the compatibility layer.
-John
[1] http://breve.twisty-industries.com/
[2] http://www.makotemplates.org/
[3] http://www.cheetahtemplate.org/
Noah Kantrowitz
I'm biased given how much work I've put in on ThemeEngine, but I think this
is the kind of thing that we should try to ask users what they want. It
seems like uptake on ThemeEngine (and other large-scale changes to the trac
UI) is near zero, so even though it kills me a little inside maybe this
isn't a big feature to lose. CSS-only changes to the UI (and the big system
in ThemeEngine for generating them) would still work as-is.
The same goes for streamfilter plugins. What plugins out there are making
good use of this? I know TagsPlugin relies on a stream filter for injecting
the tag display into wiki pages. Also, could make something like stream
filters (not talking about API compat, just functional similarity) the same
way 0.10 injected XSRF tokens (render the full page and then parse it out
again)? Maybe that will still be enough of a performance win to justify the
added slowdown?
--Noah
Rowan
compatibility will just make things easier for everyone, less code =
less things that can break (and less work for the core developers).
As long as there is ample warning to all plugin developers about when
the transition will take place then they can upgrade their plugins to
the new template system then all "end users" have to do is update
their plugins along with trac. I'd suggest a long RC period if this
does happen that way plugin authors can test their plugins against a
trac core that is unlikely to change much.
~Rowan
Rowan
fields in BlackMagicTicketTweaks, although changing the layout is not
100% necessary since the validation is done by the back end, it does
help from the user perspective if they're not allowed to use a field
then if it's disabled in the browser it saves confusion. Stream
filters really do make tweaking the UI easy, I'd like something that
allows the modification of the UI from the python side (no javascript)
to be retained in Trac.
~Rowan
Felix Schwarz
> The same goes for streamfilter plugins. What plugins out there are making
> good use of this?
Depends on what you define as 'good use' but for Agilo we basically use
a lot of interfaces to their maximum extend (and beyond ;-).
We use stream filters to:
- Modify the ticket page
- inject resources in all template
Basically we're using Genshi 0.6 so heavily that the current Genshi
trunk raises an exception if we run it with the Agilo match templates ;-)
fs
Felix Schwarz
also think of all the captcha plugins - I guess every single one of them
uses streamfilters.
fs
Christian Boos
>
> Am 08.04.2010 02:20, schrieb Noah Kantrowitz:
>> The same goes for streamfilter plugins. What plugins out there are
>> making
>> good use of this?
>
> Depends on what you define as 'good use' but for Agilo we basically
> use a lot of interfaces to their maximum extend (and beyond ;-).
>
> We use stream filters to:
> - Modify the ticket page
> - inject resources in all template
Can you point us to some examples?
In Trac itself, the stream filters could all be replaced with some
"placeholders" or defined points of injection of extra xhtml content.
Stripping of access keys and injection of form token could both be
replaced by functions used in templates.
The stream filter approach is indeed very flexible, but also fragile by
design. It's all well and good if the templates don't evolve, but as
we're making changes, we can't possibly know which plugins will break,
as /any/ part of the content might be relevant for some xpath expression
used by a transform filter. On the other hand, if we're adding
"extension points" in the templates (the exact form these extensions
points could take I don't know yet - Jinja2 informed opinions welcomed),
they'll naturally be part of the template's evolution.
> Basically we're using Genshi 0.6 so heavily that the current Genshi
> trunk raises an exception if we run it with the Agilo match templates ;-)
Which I suppose is also a situation you can't be happy with...
-- Christian
>
> fs
>
Eirik Schwenke
Warning: it might be a bit to early in the day for me to be posting this
-- if anything below looks confrontational that's *not* intended...
I'm a bit confused -- why wouldn't we just slap a 0.6 on genshi, and
place it under the trac umbrella ?
As far as I understand, the alternative is for genshi to stagnate, and
not be packaged at all ? Do we really expect trac to need/introduce
incompatible changes to genshi ?
On Debian Stable I get:
> $ aptitude search '?depends(python-genshi)'
p lazygal Static web gallery generator
p python-creoleparser Parser for the Creole common wiki markup
language
p trac Enhanced wiki and issue tracking system for
software development projects
Looking at sid/unstable as well yields the following additional packages:
p python-django-genshi Django integration for Genshi
p python-relatorio Python module to create reports from Python
objects
p python-sponge Sponge is a web framework built on top of
CherryPy and Genshi
p python-turbogears2 Python web application framework
p tryton-server Tryton Application Platform (Server)
Most of these projects seem a little stale to me -- not entirely sure
about genshi and turbogears though ?
As I see this, genshi would benefit from being tested with one big,
real-world project (trac) -- and there's no real reason to *embed* it --
now that all the infrastructure needed to support it as an external egg
is available ?
I'm not saying we should "steal" genshi, but it really seems like 0.12
will need a new 0.6 release of genshi either way -- and releasing a new
"major" version seems cleaner than any other option.
Not familiar enough with pypi-dependencies to know how several version
interact (as far as I know it's not possible/advisable to simply
easy_install genshi-0.5 *and* genshi-0.6 for example?).
Anyway, I think there are three seperate issues, that all warrant
discussion:
a) Is the speed of genshi doomed due to genshis design? And if so,
should we for *speed reasons* give up on genshi ?
b) Are there resources available within the trac community to
provide adequate support for genshi for trac's needs (and
thereby better support than is available for any other
present or future project using genshi) ?
c) Does genshi's stream-filters provide enough of an advandage
to push for b) over a) ?
After looking at some of the genshi code examples on the wiki, I really
like what I see -- and think genshi have a lot going for it. If it's too
slow to be usable *due to design issues* -- I suppose the discussion
simply should be: *when* do we move to another system?
Note: I use the term "we" here -- although I've yet to contribute a
single line of code -- so take everything above with an extra grain of
salt...
Best regards,
--
.---. Eirik Schwenke <eirik.s...@nsd.uib.no>
( NSD ) Harald Hårfagresgate 29 Rom 150
'---' N-5007 Bergen tlf: (555) 889 13
GPG-key at pgp.mit.edu Id 0x8AA3392C
Christopher Lenz
> W. Martin Borgert skrev 07. april 2010 23:23:
>> On 2010-04-07 17:02, Christian Boos wrote:
>>> But if it's still not released in a month or so from now, then we
>>> could indeed integrate the genshi/ files as tracdep/genshi.* and
>>> make sure that the Trac plugins importing the genshi modules will
>>> actually find our tracdep.genshi ones (hopefully a
>>> `sys.modules['genshi'] = tracdep.genshi` will be enough).
>> I think, this model is not an option for Debian nor for most
>> other Linux distributions. Debian, and surely others as well,
>> work very hard on removing embedded code copies. Adding a new
>> copy would be a stop in the wrong direction. Embedded code
>> copies make QA work, especially security fixes, very hard. Also,
>> they bloat the archive. They constitute a policy violation in
>> Debian. It would be preferable to find an alternative solution.
>
> Warning: it might be a bit to early in the day for me to be posting this -- if anything below looks confrontational that's *not* intended...
>
>
> I'm a bit confused -- why wouldn't we just slap a 0.6 on genshi, and place it under the trac umbrella ?
Genshi pretty much *is* under the Trac umbrella, and releasing trunk as 0.6 is the plan.
[snip]
> Anyway, I think there are three seperate issues, that all warrant discussion:
>
> a) Is the speed of genshi doomed due to genshis design? And if so,
> should we for *speed reasons* give up on genshi ?
The current design is inefficient in a number of ways. The whole design choice that everything is streamed through the pipeline event by event (SAX-style) using Python generators has proved to be rather poor. That match templates are processed at render time makes their use quite expensive, and it certainly doesn't scale to a larger number of match templates.
It would be possible to move Genshi towards a more efficient implementation by:
* Dropping support for dynamic XInclude tags, i.e. Genshi would need to know what gets included at parse time.
* Moving match template processing to the parsing stage (aka static match templates); or alternatively, drop the match template idea altogether and move to the more conventional model of template inheritance where the master predefines slots that can be filled.
* Compiling templates to Python bytecode.
* Providing a fragment caching system.
* Etc.
It would still not be in the same league as highly optimized text template engine such as Mako or Jinja2, but I think it would be totally usable. As a point of reference, Genshi trunk is in some cases actually faster than Django templates in raw throughput when you don't use things like match templates (last time I tested at least), and lots of sites use Django templates. I think that doing the above changes would make Genshi consistently faster than Django templates (at least the current implementation).
But those changes also require a lot of work, and would obviously take away some features people (including Trac and Trac plugins) may have been relying on. It'd basically be Genshi2 ;)
> c) Does genshi's stream-filters provide enough of an advandage
> to push for b) over a) ?
Not sure. Stream filtering is used for a couple things, most importantly form filling and localization. Removing stream filtering leaves two alternative ways to achieve these (and similar) tasks:
* post-process the generated output (parse/transform)
* take care of everything right in the template, i.e. manually fill in form values, gettext() all strings, etc.
Stream filters are nice in that they let you implement "cross-cutting concerns" outside of the templates, and doesn't require a post-processing stage. Whether that's worth the cost I don't know.
Michael Renzmann
> The same goes for streamfilter plugins. What plugins out there are making
> good use of this?
For what it's worth, the TracHacksPlugin does use a streamfilter. It's
used as part of the HTML form validation, to inject error messages right
beside the input fields that caused the message.
Bye, Mike
Michael Renzmann
> Stream filters really do make tweaking the UI easy, I'd like something
> that allows the modification of the UI from the python side (no
> javascript) to be retained in Trac.
+1
Bye, Mike
Grzegorz Sobanski
> In Trac itself, the stream filters could all be replaced with some
> "placeholders" or defined points of injection of extra xhtml content.
> Stripping of access keys and injection of form token could both be
> replaced by functions used in templates.
We are using streamfilter in interal plugins to extend the Milestone
description with some custom reports. As you say, I always considered
the way I extend it very fragile - little change in milestone layout
could easily kill the plugin.
So I am all in favor of "placeholders". Of course that will bring
problems, because a lot of people would like them in defferent places ;)
but for me it seems a sane solution.
greets
silk
anatoly techtonik
>
>> Anyway, I think there are three seperate issues, that all warrant discussion:
>>
>> a) Is the speed of genshi doomed due to genshis design? And if so,
>> should we for *speed reasons* give up on genshi ?
>
> The current design is inefficient in a number of ways. The whole design choice that everything is streamed through the pipeline event by event (SAX-style) using Python generators has proved to be rather poor. That match templates are processed at render time makes their use quite expensive, and it certainly doesn't scale to a larger number of match templates.
This should be documented - absolutely. I would vote for a Python
conference talk describing the situation in great detail. This is a
very good example of a unfortunate architectural design that proves
that some good concepts are hard to be implemented. It should be
documented, because the problems with such designs don't lie on the
surface. Implementation for some concepts can consume too much
resources, can be inefficient for extension, maintenance and may not
scale.
It would be nice to see analysis if these essential problems could be
predicted from the beginning. What was required to be able to do so at
that time? How should we probe our new ideas to see these problems?
Genshi design resembled me XSLT. My experience with XML/XSLT proved
that this technology is inefficient, it seems to have all the same
problems, but I can't explain why, and that's bad, because some people
continue dreaming about how powerful and beautiful XSLT is. I can't
explain them why DocBook XSLT templates are bad without resorting to
example that when PHP manual turned to 12Mb XML, it took me 8 hours
and about 300Mb memory to compile it on 256Mb machine. I was debugging
XSLT templates to add new feature to Extended CHM format. After this I
ceased my work on xCHM. Now I like how Sphinx does docs for Python,
but I have 4Gb machine with several cores, and it is fast. I can't say
if it is easy to debug templates for Sphinx, but I know for sure that
XSLT debug is a nightmare even now. It is fun to learn XML, XSLT,
because nobody tell you about the complexity of debug, that writing
such templates is one way only.
It would be extremely interesting to know about Genshi mistakes,
because such reports are rare. People do not like to talk about their
mistakes. They like to share new ideas, perspective prototypes, but
knowledge about mistakes is more valuable. I'd even say that such
mistakes deserve a book. I can add some paragraphs to chapter about
XSLT. =)
> It would be possible to move Genshi towards a more efficient implementation by:
>
> * Dropping support for dynamic XInclude tags, i.e. Genshi would need to know what gets included at parse time.
> * Moving match template processing to the parsing stage (aka static match templates); or alternatively, drop the match template idea altogether and move to the more conventional model of template inheritance where the master predefines slots that can be filled.
> * Compiling templates to Python bytecode.
> * Providing a fragment caching system.
> * Etc.
>
> It would still not be in the same league as highly optimized text template engine such as Mako or Jinja2, but I think it would be totally usable. As a point of reference, Genshi trunk is in some cases actually faster than Django templates in raw throughput when you don't use things like match templates (last time I tested at least), and lots of sites use Django templates. I think that doing the above changes would make Genshi consistently faster than Django templates (at least the current implementation).
It is a pity there is no testsuite to analyze features vs performance
for various template engines using templating scenarious, examples and
performance/memory metrics.
> But those changes also require a lot of work, and would obviously take away some features people (including Trac and Trac plugins) may have been relying on. It'd basically be Genshi2 ;)
>
>> c) Does genshi's stream-filters provide enough of an advandage
>> to push for b) over a) ?
>
> Not sure. Stream filtering is used for a couple things, most importantly form filling and localization. Removing stream filtering leaves two alternative ways to achieve these (and similar) tasks:
> * post-process the generated output (parse/transform)
> * take care of everything right in the template, i.e. manually fill in form values, gettext() all strings, etc.
>
> Stream filters are nice in that they let you implement "cross-cutting concerns" outside of the templates, and doesn't require a post-processing stage. Whether that's worth the cost I don't know.
That's why metrics with examples is needed.
--
anatoly t.
Eirik Schwenke
> On 08.04.2010, at 09:23, Eirik Schwenke wrote:
>>
>> I'm a bit confused -- why wouldn't we just slap a 0.6 on genshi,
>> and place it under the trac umbrella ?
>
> Genshi pretty much *is* under the Trac umbrella, and releasing trunk
> as 0.6 is the plan.
>
Ok, that is more or less what I thought, and hence why I was confused ;-)
(...)
>> Anyway, I think there are three seperate issues, that all warrant
>> discussion:
>>
>> a) Is the speed of genshi doomed due to genshis design? And if so,
>> should we for *speed reasons* give up on genshi ?
>
> The current design is inefficient in a number of ways. The whole
> design choice that everything is streamed through the pipeline event
> by event (SAX-style) using Python generators has proved to be rather
> poor.
Hm, and "everyone" keeps claiming that SAX is pretty efficient,
certainly compared to parsing entire document trees (XLST-style).
This sort of reminds me of this article I recently came across for an
entirely different project:
http://effbot.org/zone/simple-top-down-parsing.htm
(discussing merits of a pure python top-down descent parser,
inspired by the paper by Pratt):
http://portal.acm.org/citation.cfm?doid=512927.512931
Most notably, the parser ends up being *more* efficient than the one in
python, tokenizer part aside.
It would appear that event-based parsing of templates is somewhat
similar to top-down-parsing -- but maybe we're missing something in the
implementation, due to the way functions are called ?
> That match templates are processed at render time makes their
> use quite expensive, and it certainly doesn't scale to a larger
> number of match templates.
>
> It would be possible to move Genshi towards a more efficient
> implementation by:
>
> * Dropping support for dynamic XInclude tags, i.e. Genshi would need
> to know what gets included at parse time.
> * Moving match template processing to the parsing stage (aka static
> match templates); or alternatively, drop the match template idea
> altogether and move to the more conventional model of template
> inheritance where the master
> predefines slots that can be filled.
> * Compiling templates to Python bytecode.
> * Providing a fragment caching system.
Essentially either implement caching (as bytecodes, static templates) or
redesign the templates to another design ? (includes, static/simple
templates)?
> It would still not be in the same league as highly optimized text
> template engine such as Mako or Jinja2, but I think it would be
> totally usable.
Due to the event-based design ?
We should perhaps keep in mind that the latest intel instruction set
includes operators designed for stream processing of unicode strings
(targeting xml, but the opcodes are general text/match instructions):
http://www.strchr.com/strcmp_and_strlen_using_sse_4.2
http://en.wikipedia.org/wiki/SSE4
http://software.intel.com/en-us/articles/xml-parsing-accelerator-with-intel-streaming-simd-extensions-4-intel-sse4
(the last article notes some justifications from Intel on why they
created the new instructions).
Apparently XML is so slow, by design, that Intel found the need to add
new hardware instructions. As of yet AMD hasn't implemented SSE4.2 as
far as I know.
> As a point of reference, Genshi trunk is in some
> cases actually faster than Django templates in raw throughput when
> you don't use things like match templates (last time I tested at
> least), and lots of sites use Django templates. I think that doing
> the above changes would make Genshi consistently faster than Django
> templates (at least the current implementation).
Ok. So it's definitely stacking several match-templates that is slow?
Particularly because there currently is no caching, is that correct?
From looking briefly at the code, it would appear the first step might
be moving towards generating python code (and compiling that to
bytecode), to add caching. Certainly not a trivial task.
Or maybe, if most (current) use cases transform some form of xml to
another form of xml, it would be possible to safely cache intermediate
(and final) versions ? I suppose that would lead to a lot of cached text
for pages with many variables?
That assumes that tokenizing, parsing and building the ASTs (or
equivalent) structures is the slow part?
(...)
>> c) Does genshi's stream-filters provide enough of an advandage to
>> push for b) over a) ?
>
> Not sure. Stream filtering is used for a couple things, most
> importantly form filling and localization. Removing stream filtering
> leaves two alternative ways to achieve these (and similar) tasks: *
> post-process the generated output (parse/transform) * take care of
> everything right in the template, i.e. manually fill in form values,
> gettext() all strings, etc.
>
> Stream filters are nice in that they let you implement "cross-cutting
> concerns" outside of the templates, and doesn't require a
> post-processing stage. Whether that's worth the cost I don't know.
Having worked with other template system on other projects, I definitely
like the idea of having l10n and i18n *outside* of templates, as far as
that makes sense. Feels like it would lead to easier reuse of other
peoples translations as well.
Thank you for your detailed answers!
Thomas Moschny
> The same goes for streamfilter plugins. What plugins out there are making
> good use of this?
We also have at least one internal plugin making use of streamfilters.
In general, it's a nice thing to have, as plugins can tweak the ui
with zero help from the main Trac devs. It's like having hooks
everywhere, even in places no one thought about previously. In a
hook-based model, when there's no hook where you need one, you first
have to convince others that it might be useful, and then wait for the
next Trac release. Admittedly in some cases talking to others can also
have the effect of you being convinced that the hook is indeed not
needed ;)
--
Thomas Moschny <thomas....@gmail.com>
osimons
> 2010/4/8 Noah Kantrowitz <n...@coderanger.net>:
Agreed. And, it is not just plugins - "site.html match, add & rewrite"
is also quite common for a variety of known and unknown use-cases.
I'm not fuzzed either way about how the technology works (ie. what
templating engine), and even rewriting my things (again) is acceptable
if a switch is deemed neecessary. However, depending on Trac
development for available hooks would be a big step backward, and a
big negative for me as I won't really be able to redo what I currently
have in production.
:::simon
https://www.coderesort.com
http://www.ohloh.net/accounts/osimons
Christian Boos
> On Apr 13, 2:27 pm, Thomas Moschny<thomas.mosc...@gmail.com> wrote:
>
>> 2010/4/8 Noah Kantrowitz<n...@coderanger.net>:
>>
>>
>>> The same goes for streamfilter plugins. What plugins out there are making
>>> good use of this?
>>>
>> We also have at least one internal plugin making use of streamfilters.
>>
>> In general, it's a nice thing to have, as plugins can tweak the ui
>> with zero help from the main Trac devs. It's like having hooks
>> everywhere, even in places no one thought about previously. In a
>> hook-based model, when there's no hook where you need one, you first
>> have to convince others that it might be useful, and then wait for the
>> next Trac release. Admittedly in some cases talking to others can also
>> have the effect of you being convinced that the hook is indeed not
>> needed ;)
>>
> is also quite common for a variety of known and unknown use-cases.
>
> I'm not fuzzed either way about how the technology works (ie. what
> templating engine), and even rewriting my things (again) is acceptable
> if a switch is deemed neecessary.
Well, the longer Genshi 0.6 takes to materialize, the more obvious it
will be for everyone that we *need* that change, if only from the
software maintenance point of view.
Christopher specifically asked for help for *one* ticket (#370), but
nobody seems to care so far and this tells a lot. If someone really
wants to *voice* an opposition to a migration to an other templating
system, that would be his chance. So while I still like to believe that
0.6 itself will be released at last, I'm a bit worried about the
follow-up 0.6.x releases we might need, and I don't have any illusion
about 0.7 as I witness the Genshi development grind to a halt.
But despite the above, current Genshi trunk / 0.6 sort of works for our
needs, for the few issues we have, there are workarounds. So it's kind
of OK for 0.12 and 0.12.x, and it will also be OK for 0.13, we just have
to be careful to work within the bounds imposed by the current state of
Genshi.
But precisely, those bounds are also bounds on the performance and
that's the second motivation that justifies a migration.
Now, I'm still talking about a migration and not a switch, as I think
it's perfectly possible that we introduce Jinja2 support gradually. We
could use Jinja2 templates to generate some performance critical parts
of the output, then wrap the result in a Markup object and pass that
along to Genshi, which in the first stage would still be responsible for
generating the main layout, theme, etc. as usual. Wiki pages even big
ones are currently rendered quickly precisely because they contain a lot
of content which is opaque to Genshi. The trade-off will be that most of
the complex transformations done on the overall layout, like typically
those involving site.html and documented in
http://trac.edgewall.org/wiki/TracInterfaceCustomization and
http://trac.edgewall.org/wiki/CookBook/SiteHtml will *still* work, but
those trying for example to modify each row of a rendered source file
will no longer work (and I imagine there are not a lot of such templates
anyway, precisely for performance reasons).
In a second step, we could introduce an alternative way to compose the
layout and themes using Jinja2 all the way through, and only then phase
out Genshi.
> However, depending on Trac
> development for available hooks would be a big step backward,
Not sure; as I've pointed out earlier and others have noticed as well,
not having explicit hooks is also quite fragile. Suddenly, after a
seemingly innocent change in the template, the xpath selectors of some
plugin stop working...
If you read the previous mail from Christopher where he gave some
possible ideas for an improved "Genshi2" (in his reply to Eirik
Schwenke), he was also considering getting away with the xpath matches
("alternatively, drop the match template idea altogether and move to the
more conventional model of template inheritance where the master
predefines slots that can be filled.").
> and a
> big negative for me as I won't really be able to redo what I currently
> have in production.
>
Well, I don't really know what kind of changes you have in place, but if
they are mostly about the general layout, the hybrid approach I
suggested above should give you enough room for a smooth transition.
-- Christian
Thomas Moschny
> [ ... ] but those trying for example to modify each row of a
> rendered source file will no longer work [ ... ]
Yet something like this is most interesting e.g. for a plugin
implementing a VCS backend/UI with needs to tweak the source and
changelog browsers.
>> However, depending on Trac
>> development for available hooks would be a big step backward,
>
> Not sure; as I've pointed out earlier and others have noticed as well, not
> having explicit hooks is also quite fragile. Suddenly, after a seemingly
> innocent change in the template, the xpath selectors of some plugin stop
> working...
Yes, but I can fix that in my plugin and release a new version of it
within a very short time frame, whereas getting a new hook or slot (or
whatever technique will be used) into main Trac will take at least the
time until a new release is made, if not longer, or forever if I
cannot convince you of its usefulness.
In the end this is may be a reasonable trade-off anyway: having an
actively maintained template engine with decent performance, vs making
life harder for some plugin developers.
--
Thomas Moschny <thomas....@gmail.com>
Felix Schwarz
> Christopher specifically asked for help for *one* ticket (#370), but
> nobody seems to care so far and this tells a lot.
I'd like to add that I spent 3-4 hours debugging that issue (as it was a
problem we found in Agilo). I have a couple of (too complicated) unit
tests that uncover the issue.
The problem exists in the generic strategy. The strange thing is that a
callable is returned from a method and this callable uses a local
variable of that method (basically some extra state for the callable).
This extra state is a stack which underflows due to an unknown condition.
The problem which seem to overwhelm my python foo/debugging capabilities
is that the problem was introduced in a mass-merge of GSOC work.
If someone of you plans on tackling the issue, please ping me. I'm
willing to help with code+debugging. However it's pretty frustrating
tackling the thing alone and running against wall every time...
Besides that I think I have another problematic regression in Genshi 0.6
which might be only a test ordering issue.
fs
Christian Boos
On 4/14/2010 2:13 PM, Felix Schwarz wrote:
>
> Am 14.04.2010 12:49, schrieb Christian Boos:
>> Christopher specifically asked for help for *one* ticket (#370), but
>> nobody seems to care so far and this tells a lot.
>
> I'd like to add that I spent 3-4 hours debugging that issue (as it was
> a problem we found in Agilo). I have a couple of (too complicated)
> unit tests that uncover the issue.
Yes, I can imagine. I know you created the ticket and contributed an
initial patch. Nevertheless that issue still blocks the release. If it's
only happening in a corner case, maybe you could try to find a workaround.
> The problem exists in the generic strategy. The strange thing is that
> a callable is returned from a method and this callable uses a local
> variable of that method (basically some extra state for the callable).
> This extra state is a stack which underflows due to an unknown condition.
>
> The problem which seem to overwhelm my python foo/debugging
> capabilities is that the problem was introduced in a mass-merge of
> GSOC work.
That's one of the main issue for the maintenance, Christopher is too
smart ;-)
> If someone of you plans on tackling the issue, please ping me. I'm
> willing to help with code+debugging. However it's pretty frustrating
> tackling the thing alone and running against wall every time...
I might give it a try (after the beta), but only if I get the guarantee
to have a 0.6 release after that ;-)
-- Christian
> Besides that I think I have another problematic regression in Genshi
> 0.6 which might be only a test ordering issue.
workaround, workaround ...
>
> fs
>
Noah Kantrowitz
> From: trac...@googlegroups.com [mailto:trac...@googlegroups.com] On
> Behalf Of Thomas Moschny
> Sent: Wednesday, April 14, 2010 7:33 AM
> To: trac...@googlegroups.com
> Subject: Re: Towards Jinja2 (was Re: [Trac-dev] Which genshi version
> for Trac 0.12?)
>
There is also more than one way to offer this kind of system. I suppose the
most basic version would be to have a post-render callback that gets the
full text of output and can run regexes. Thats probably yet more fragile,
but it offers the same kind of crazy.
--Noah
Christopher Lenz
> Am 14.04.2010 12:49, schrieb Christian Boos:
>> Christopher specifically asked for help for *one* ticket (#370), but
>> nobody seems to care so far and this tells a lot.
>
> I'd like to add that I spent 3-4 hours debugging that issue (as it was a problem we found in Agilo). I have a couple of (too complicated) unit tests that uncover the issue.
>
> The problem exists in the generic strategy. The strange thing is that a callable is returned from a method and this callable uses a local variable of that method (basically some extra state for the callable). This extra state is a stack which underflows due to an unknown condition.
>
> The problem which seem to overwhelm my python foo/debugging capabilities is that the problem was introduced in a mass-merge of GSOC work.
Yeah this code is a hairy mess and hell to debug. There are parts of the code developed during GSoC that I have trouble understanding myself. But to be fair, this bug was actually (luckily?) outside the GSoC work.
Anyway, I've just attached an 8 byte patch for this issue, along with your test cases:
<http://genshi.edgewall.org/attachment/ticket/370/ticket370.diff>
I would be very grateful if more people could give this change some testing in the field. All the unit tests are nice and green, but of course that might just mean that we're missing an important test case :P
Here's to 0.6 in a few days.
Remy Blank
> All the unit tests are nice and green, but of course that might just
> mean that we're missing an important test case :P
I have a failing test here (Linux, Python 2.6.4), that turned out to be:
http://genshi.edgewall.org/ticket/324
The patch suggested there (defining __length_hint__ as None) makes the
test pass.
-- Remy