Moving wiki_to_html into the templates
Christopher Lenz
Back when I was involved with porting DrProject (a Trac clone for
academic use) to use the Kid template language, one thing that I
found to be beneficial to reducing general code complexity/volume was
that we moved the generation of wiki output into the templates. I.e.
instead of having the controller prepare the HTML formatted wiki
text, it'd just pass the wiki text itself, and the template would
invoke wiki_to_html() on that where necessary.
Christian has attempted to do the same for Trac when Genshi was still
on a branch[1], but we found one problem: macros and processors in
wiki text are executed relatively late in the process, i.e. when the
page is already being generated. At that point, it is too late to add
things like style sheets to the <head> of the page, so that for
example the PatchRenderer wasn't able to tell Trac that it needed the
"diff.css" stylesheet to be rendered properly. At that point Trac
would already have rendered the <head> element, so obviously we can't
add anything to it.
There is no 100% clean way to get around that problem, unfortunately.
But I think I found an approach that's good enough. The approach
requires javascript to load stylesheets after the fact, in particular
the $.loadStyleSheet method I've added to trac.js:
<http://trac.edgewall.org/browser/trunk/htdocs/js/trac.js#L36>
A macro/processor such as the PatchRenderer would call add_stylesheet
() as it used to, however, that function would now have a return value:
if req.environ.get('trac.chrome.rendering'):
# The template is already being rendered, so return a
javascript
# snippet that can be used to load the stylesheet from the
HTML body
return tag.script(type='text/javascript')(
'$.loadStyleSheet("%s", "%s")' % (link, mimetype)
)
And note that it only returns that if that particular stylesheet
wasn't already included in some way.
In my current implementation, the macro/processor would be
responsible for checking whether the function returned a value, and
inserting that to its output stream in some way. But I'm going to
investigate whether Trac could just put any requried $.loadStyleSheet
() calls in the footer, meaning the return value wouldn't even be
needed, i.e. the implementation of add_stylesheet() might end similar
to the following:
if req.environ.get('trac.chrome.rendering'):
req.environ['trac.chrome.extrastyles'].append(href)
... and the layout.html template would be responsible for pulling in
those "late" style sheets in via $.loadStyleSheet() after the content
has been rendered.
Anyway, I suspect this whole technique may be somewhat controversial,
due to relying on javascript. However, please note that the reliance
on javascript is only for styling some kinds of content (such as
diffs, or maybe syntax highlighted source code snippets) that may
appear in wiki text. There's no missing functionality or content. The
markup is still as semantic as before, meaning that such content is
still accessible to those who don't have much use of visual styles
anyway (i.e. visually impaired users, text-browser using users,
search engines). And also accessible to anyone, just not as pretty or
colorful. I personally think that's an okay trade-off to make.
Cheers,
Chris
[1] http://trac.edgewall.org/attachment/wiki/ChristianBoos/
wiki2x_in_templates.diff
--
Christopher Lenz
cmlenz at gmx.de
http://www.cmlenz.net/
Christopher Lenz
> In my current implementation, the macro/processor would be
> responsible for checking whether the function returned a value, and
> inserting that to its output stream in some way. But I'm going to
> investigate whether Trac could just put any requried $.loadStyleSheet
> () calls in the footer, meaning the return value wouldn't even be
> needed, i.e. the implementation of add_stylesheet() might end similar
> to the following:
>
> if req.environ.get('trac.chrome.rendering'):
> req.environ['trac.chrome.extrastyles'].append(href)
>
> ... and the layout.html template would be responsible for pulling in
> those "late" style sheets in via $.loadStyleSheet() after the content
> has been rendered.
And that turned out to be very simple and completely transparent. See:
<http://trac.edgewall.org/attachment/wiki/ChristopherLenz/
wiki2x_in_templates.diff>
Cheers,
Chris
Sergey Lipnevich
pre-rendering in the code. I tried something similar (color-coding
overlapping portions of text in my own app) and decided not to do it
in template because frankly I wasn't sure how to begin. So you rule
guys :-)! But why not ask a macro if it needs to add a stylesheet to
the page? Maybe the all-or-nothing approach (render in code vs render
in template) is not the best in this case? In more complex cases, a
macro or a processor may want to change MIME format or introduce an
http-equiv="Refresh" meta keyword (in Bitten). Or this not part of the
design for macros or processors?
Consider also that JavaScript is not good when it comes to unit
testing Web page generation. It's far easier to just test if there's a
reference to a stylesheet than to test that there's a javascript that
hopefully will insert this reference later.
Sergey.
Ilias Lazaridis
Christopher Lenz wrote:
> Hey all,
>
> Back when I was involved with porting DrProject (a Trac clone for
> academic use) to use the Kid template language, one thing that I
> found to be beneficial to reducing general code complexity/volume was
> that we moved the generation of wiki output into the templates. I.e.
I am not sure if this is a positive step.
Additionally, I think that doing it java-script would be a workaround
and not a solution.
Can you possibly provide a use-case, e.g. ready code, and a wiki-page,
where interested parties can do immediately tests with?
The trac wiki is the central part of my interest, thus I would like to
look at this case.
I could possibly combine it with my current task:
http://dev.lazaridis.com/base/wiki/PlanTicketQueryMacro
.
Noah Kantrowitz
would be to have a users setting (either manually selected or have
some kind of auto-detection) for using JS. This would let any macros
that critically depended on JS (or having their CSS) to change their
behavior.
--Noah
Christopher Lenz
Am 21.10.2006 um 22:11 schrieb Sergey Lipnevich:
> Genshi is very powerful if it can render complete Wiki text without
> pre-rendering in the code. I tried something similar (color-coding
> overlapping portions of text in my own app) and decided not to do it
> in template because frankly I wasn't sure how to begin. So you rule
> guys :-)! But why not ask a macro if it needs to add a stylesheet to
> the page?
Because we don't (currently) know which macros/processors are going
to be used without actually performing the rendering. Of course, this
situation may change, and at some point we may know enough about wiki
content to be able to do this without javascript.
> Maybe the all-or-nothing approach (render in code vs render
> in template) is not the best in this case? In more complex cases, a
> macro or a processor may want to change MIME format or introduce an
> http-equiv="Refresh" meta keyword (in Bitten). Or this not part of the
> design for macros or processors?
No, macros shouldn't change the MIME type of the output, as they have
no control over where they get embedded. For meta-refresh, I don't
think it's a particularly good technique, the effect is more easily
and elegantly implemented using XHR these days (which the now
integrated jQuery makes dead easy).
I don't think there's *anything* a macro would want to add to the
page it's embedded in, apart from what we're talking about here:
requesting additional external stylesheet and script files.
> Consider also that JavaScript is not good when it comes to unit
> testing Web page generation. It's far easier to just test if there's a
> reference to a stylesheet than to test that there's a javascript that
> hopefully will insert this reference later.
That's not really "unit testing" now, is it ;-)
It's way easier to test this stuff at a lower level (using *actual*
unit tests), and then perform functional tests in the browser, either
manually, or using something like Selenium.
Cheers,
Chris
Christian Boos
> ... or using something like Selenium.
Hm, now I become scared when I hear you say "something like"
:-)
-- Christian
Christopher Lenz
> Christopher Lenz wrote:
>> Hey all,
>>
>> Back when I was involved with porting DrProject (a Trac clone for
>> academic use) to use the Kid template language, one thing that I
>> found to be beneficial to reducing general code complexity/volume was
>> that we moved the generation of wiki output into the templates. I.e.
> ...
>
> I am not sure if this is a positive step.
Erm, you would need to back that up with some reasoning, don't you
think?
> Additionally, I think that doing it java-script would be a workaround
> and not a solution.
It's a solution because it solves the problem at hand. Sure it's not
optimal, that's why I wrote "I personally think that's an okay trade-
off to make".
The alternative would be to keep all the wiki_to_xxx() invocations in
the controller code. That's an option I don't find attractive. Moving
them into templates would have the following benefits:
* (c)leaner controller code; controllers can concentrate more on
the application logic, and less on presentation concerns. We have a
couple hundreds LOC that just pack the model object in a dict
together with wiki-formatted fields (and other presentational
aspects)... that code would no longer be needed.
* slightly less tightly coupled controllers/templates; as the
controller only passes the wiki source text to the templates
* better cusomizability: every deployment can decide whether
something is to be wiki-formatted or not simply by overriding the
corresponding template snippet. Things like the
"wiki_format_messages" option for changesets would no longer be
necessary.
> Can you possibly provide a use-case, e.g. ready code, and a wiki-page,
> where interested parties can do immediately tests with?
I posted the patch. You can apply it to trunk and try it. That patch
and this mailing list thread is all I have to offer.
> I could possibly combine it with my current task:
>
> http://dev.lazaridis.com/base/wiki/PlanTicketQueryMacro
I don't see what that has to do with the topic of this thread.
Cheers,
Christian Boos
> Am 20.10.2006 um 19:17 schrieb Christopher Lenz:
>
>>
>> if req.environ.get('trac.chrome.rendering'):
>> req.environ['trac.chrome.extrastyles'].append(href)
>>
>> ... and the layout.html template would be responsible for pulling in
>> those "late" style sheets in via $.loadStyleSheet() after the content
>> has been rendered.
>>
>
> And that turned out to be very simple and completely transparent. See:
>
> <http://trac.edgewall.org/attachment/wiki/ChristopherLenz/
> wiki2x_in_templates.diff>
>
http://trac.edgewall.org/attachment/wiki/ChristopherLenz/wiki2x_in_templates.2.diff
is OK for me.
After that, I'll refresh mine:
http://trac.edgewall.org/attachment/wiki/ChristianBoos/wiki2x_in_templates.diff
and look forward to commit it later this evening.
-- Christian
Sergey Lipnevich
> It's a solution because it solves the problem at hand. Sure it's not
> optimal, that's why I wrote "I personally think that's an okay trade-
> off to make".
That's rather wide definition of a solution :-).
> The alternative would be to keep all the wiki_to_xxx() invocations in
> the controller code. That's an option I don't find attractive. Moving
> them into templates would have the following benefits:
Not necessarily. Parsing the wiki text for just macros and
preprocessors (I think it would be a fast and rather simple task) in
code and doing the rest of the work in template is also a possibility.
That's what I meant by not going the all-or-nothing route.
Sergey.
Christian Boos
>> the controller code. That's an option I don't find attractive. Moving
>> them into templates would have the following benefits:
>
> Not necessarily. Parsing the wiki text for just macros and
> preprocessors (I think it would be a fast and rather simple task) in
> code and doing the rest of the work in template is also a possibility.
> That's what I meant by not going the all-or-nothing route.
>
I also thought about this solution, but it has the disadvantage of
leaving the controller code cluttered with 'wikiparse' invocations
instead of the 'wiki_to_*' ones (the latter being moved to the
templates). Moreover that parsed wiki should be "pre-rendered" (going
through the macros, call a "prerender" method for example), which is
therefore an additional step to do in the controller. So this additional
complexity has to be compensated by some kind of benefit, and it is not
clear at this point if there's any; I discussed it with Christopher and
we agreed that until we find a compelling reason to do so, there was no
need to do things that way.
Of course if someone comes with a convincing need for this, we could
always switch to that solution at a later point.
-- Christian
Noah Kantrowitz
A possible work around for macro providers that need the old behavior
would be to also make a post-request filter that calls
add_stylesheet. That should still be executed before the template is
rendered, so it should get included in the <head>.
--Noah
Christian Boos
> ...
> A possible work around for macro providers that need the old behavior
> would be to also make a post-request filter that calls
> add_stylesheet. That should still be executed before the template is
> rendered, so it should get included in the <head>.
>
The stylesheet would then be added to *every* rendered page, even if
that macro doesn't get used. Not sure if it's a better trade-off...
-- Christian
Noah Kantrowitz
Yes, or the plugin could do the kind of page parse to see if it is
included that you talked about earlier. All aren't perfect solutions,
but I would bet that one of these (or some other way no one has
thought of yet) will cover at least most of the possible situations.
--Noah
Ilias Lazaridis
> Am 22.10.2006 um 19:38 schrieb Ilias Lazaridis:
> > Christopher Lenz wrote:
> >> Hey all,
> >>
> >> Back when I was involved with porting DrProject (a Trac clone for
> >> academic use) to use the Kid template language, one thing that I
> >> found to be beneficial to reducing general code complexity/volume was
> >> that we moved the generation of wiki output into the templates. I.e.
> > ...
> >
> > I am not sure if this is a positive step.
>
> Erm, you would need to back that up with some reasoning, don't you
> think?
I cannot (at least at this moment).
>
> > Additionally, I think that doing it java-script would be a workaround
> > and not a solution.
>
> It's a solution because it solves the problem at hand. Sure it's not
[...] - (commernts, rationales)
> > Can you possibly provide a use-case, e.g. ready code, and a wiki-page,
> > where interested parties can do immediately tests with?
>
> I posted the patch. You can apply it to trunk and try it. That patch
> and this mailing list thread is all I have to offer.
ok
> > I could possibly combine it with my current task:
> >
> > http://dev.lazaridis.com/base/wiki/PlanTicketQueryMacro
>
> I don't see what that has to do with the topic of this thread.
I don't see it, too (at least at this moment).
I will come possibly 'in touch' with this functionality, during this or
during other tasks which have to do with the wiki.
.
Sergey
> > Consider also that JavaScript is not good when it comes to unit
> > testing Web page generation. It's far easier to just test if there's a
> > reference to a stylesheet than to test that there's a javascript that
> > hopefully will insert this reference later.
>
> That's not really "unit testing" now, is it ;-)
>
> It's way easier to test this stuff at a lower level (using *actual*
> unit tests), and then perform functional tests in the browser, either
> manually, or using something like Selenium.
Hi Chris!
I'm not sure what do you mean here. If I were a macro writer, I'd
probably have a unit test with a macro rendered on a page, and I'd use
a few XPath expressions to make sure things are in place, such as
stylesheet if there's one, macro output, etc. I didn't yet look at
Trac's built-in macro tests, but I'd imagine that's how they would be
organized. Or am I off-base here?
Sergey.
Sergey
> Of course if someone comes with a convincing need for this, we could
> always switch to that solution at a later point.
Hi Christian!
What follows is not a criticism in any way, so please don't think it
is. Trac's architecture is solid, modular, maintainable, terse in code,
and scalable, so whatever you guys have in mind you have to continue
doing it. But several JavaScript-related developments introduce mild
discomfort in my personal feeling about Trac. First JQuery was used to
generate header links, now macros are required to use client-side JS to
add stylesheets. Nothing wrong with that, but I'm used to a more
conservative attitude towards JavaScript, such as that of Bugzilla
project team for example. Their motto used to be (and I think still is)
that Bugzilla must be functional without JS. Not "fully" functional of
course, but good enough to get work done. I believe that it's a good
idea to achieve maximum functionality on the server side, as that's the
side Trac controls. JS may be off, buggy, slow, or not available in a
browser. So, I as a user would feel better if Trac team were more
conservative about use of JavaScript, that's all.
At the same time, there's been a great deal of innovation coming from
you guys, and you all are experienced Web developers, so I trust that
you'll do what's best, be it adopting JS on a large scale or abandoning
it completely :-).
Thank you,
Sergey.
Christopher Lenz
> What follows is not a criticism in any way, so please don't think it
> is. Trac's architecture is solid, modular, maintainable, terse in
> code,
> and scalable, so whatever you guys have in mind you have to continue
> doing it. But several JavaScript-related developments introduce mild
> discomfort in my personal feeling about Trac. First JQuery was used to
> generate header links, now macros are required to use client-side
> JS to
> add stylesheets. Nothing wrong with that, but I'm used to a more
> conservative attitude towards JavaScript, such as that of Bugzilla
> project team for example. Their motto used to be (and I think still
> is)
> that Bugzilla must be functional without JS. Not "fully" functional of
> course, but good enough to get work done.
We're following the exact same policy here. Generating heading links,
highlighting search terms and adding stylesheets that macros need
(which is pretty rare, and just serves to make them look prettier)
does not break the functionality of the app IMO. It's mostly
decorative or usability-enhancing stuff.
Also note that search term highlighting and heading links generation
was implemented with JS ever since those features were added to Trac.
And the ticket query screen makes extensive use of JS to improve
usability, too, but falls back to doing everything server-side when
JS isn't available. While the interface is basically usable in that
mode, realistically it's rather a pain to use ... the focus of its
design has been JS-enabled users.
> I believe that it's a good
> idea to achieve maximum functionality on the server side, as that's
> the
> side Trac controls. JS may be off, buggy, slow, or not available in a
> browser.
Sure, but that's extremely (and increasingly) rare, especially for
the kind of people who typically use Trac. If someone's using a text
browser or a screen reader, I personally want Trac to work well for
them. If someone's using a really outdated browser, or have
intentionally disabled JS in a modern browser, well I'm sorry, but
they deserve no better than getting a slightly degraded look&feel. :-)
Cheers,
Chris
Christopher Lenz
That's not a unit test, because you're also testing Trac, and are
relying on the assumption that Trac doesn't shuffle stuff around so
much that your test breaks without your Macro actually being broken
itself.
If you want to *unit test* your macro, you'd test it in isolation
from (most of) the rest of Trac. You'd simply call it directly with
various parameters, and check the return value, as well as possible
side effects on the request object. You can assert whether your macro
added a stylesheet by checking whether it got added to the list in
req.environ['trac.chrome.links']['stylesheet'].
Of course the kind of tests you mean are also useful, but they're
generally referred to as "integration" or "functional"/"acceptance"
tests. So this is mostly a terminology disagreement :-)
But even in integration tests, you'd probably call wiki_to_html() on
wiki text that contains a reference to your macro, and you'd *still*
use the technique explained above to check whether the stylesheet got
added.
It's only when you move on to do functional tests where you might
care about the difference about stylesheets added normally and those
added via JS. But for functional tests, it's a good idea to use an
actual browser (manually or automated, both are possible), which
doesn't care how a stylesheet is added, both methods add <link>
elements to the <head> in the DOM, and both result in the style rules
in the stylesheet being applied.