Newline stripping in templates: the dnl way

[Tobia]

Hi all,
regarding issue #2594 "Template system should handle whitespace
better" explaining the need for some kind of (optional) whitespace
stripping from the output of templates, I've been looking at the
proposed solutions and compared them to other template and macro
engines.

In particular, a historical but still widely used and general-purpose
macro engine is m4. Its own feature for controlled newline stripping
is the "dnl" reserved word, "Discard to Next Line." It works by
"chomping" all input until and including the next newline.

For example, to call a macro foo() without copying over the newline
that appears after the macro call in the template, a m4 programmer
would write:

foo('bar`, `whatever')dnl

An equivalent feature in Django templates would enable template
developers to strip newlines from specific lines, while keeping
backwards compatibility with existing templates.

So if the general idea is well-accepted, I propose the "{#" token. The
example from the issue would become:

<ul>
{% for item in items %}{#
<li>{{ item }}</li>
{% endfor %}{#
</ul>

It is already a reserved token in the template system, so it's not
going to break anything. The existing comment syntax {# ... #} is
already restricted to single-line comments, so there are no multi-line
comments that would break. Plus, I'd wager it could be implemented
quite efficiently.

What do you think?

Tobia

[Tai Lee]

I definitely want to see a resolution to this issue. Django templates
when white space matters (e.g. plain text emails) are painful. But, I
don't think adding {# to the end of lines is easy to understand at a
glance, it doesn't even convey a hint of meaning like "dnl" does, but
either option is pretty ugly to my eyes.

I'd personally like to see a new template tag that will switch on the
new behaviour, and a deprecation path for this to become the default.

Cheers.
Tai.

[Tom Evans]

I'd be strongly -1 on anything that makes template language look more like m4!

Having said that, I can see this being useful. We use templates to
render most kinds of output. When we render text emails, the newlines
and indentation are particularly important, as everything gets
displayed 'as-is' by the client, and having a template that is
readable and also renders correctly is tricky.

Generally where this happens is where we open a block level tag, and
would then normally have a newline for readability, but cannot due to
us not wanting a newline to appear. This could be addressed by having
a different open/close tag for tags which chomp the preceeding/next
character if it is a newline. Eg:

In your folder are:
{^ for item in folder ^}
{{ item }},
{^ endfor ^}

This would render as "In your folder are: item1,item2,item3,"

Changing some of the carets:

In your folder are:
{% for item in folder ^}
{{ item }},
{^ endfor %}

This would render as "In your folder are:\nitem1,item2,item3,\n"

I'm not 100% convinced though.

Cheers

Tom

[Aron Griffis]

On Fri, Feb 24, 2012 at 9:45 AM, Tom Evans <teva...@googlemail.com> wrote:
> In your folder are:
> {^ for item in folder ^}
> {{ item }},
> {^ endfor ^}
>
> This would render as "In your folder are: item1,item2,item3,"
>
> Changing some of the carets:
>
> In your folder are:
> {% for item in folder ^}
> {{ item }},
> {^ endfor %}
>
> This would render as "In your folder are:\nitem1,item2,item3,\n"

Fwiw, I had a different solution to this problem once, which I implemented
in a project using jinja to generate configuration files:

1. Prior to processing the template, replace all blank lines with a
tag (e.g. "BLANKLINE").

2. After processing the template, remove all blank lines.

3. Remove all instances of BLANKLINE, which restores the blank lines
the author intended.

This works remarkably well, and the output is easy to predict by the
template author. For the case you mentioned above, you would still need to
join the lines, though:

{% for item in folder %}{{ item }},{% endfor %}

Aron

[mjl Martin J. Laubach]

For this (avoiding newlines) the currently discussed multiline tags would work pretty well too without adding more cruft to the template language:

foo bar {#

#}baz 

mjl 

[Aymeric Augustin]

2012/2/24 mjl Martin J. Laubach <goo...@emsi.priv.at>

For this (avoiding newlines) the currently discussed multiline tags would work pretty well too without adding more cruft to the template language:

foo bar {#

#}baz 

Martin,

You just made a strong argument against multiline tags: I wouldn't want to see them abused this way!

-- 

Aymeric.

 

[Tobia]

Tai Lee wrote:
> I don't think adding {# to the end of lines is easy to understand at a
> glance, it doesn't even convey a hint of meaning like "dnl" does

I beg to differ. {# is already recognized by template authors as
meaning "start of comment", and they know (or should know) that it
cannot extend through more than one line. Therefore I'd think it
intuitive that it will "eat" till the end of the line and not beyond.

Look:

Here are your subscriptions:
{% for thing in things %}{#
- {{ thing.name }}
You added it on {{ thing.date }}
{% endfor %}{#
you can manage your subscriptions...

Tom Evans wrote:
> I'd be strongly -1 on anything that makes template language look more
> like m4!

I'll tell you, m4 can be quite addictive once you grasp the basics! :)

> This could be addressed by having a different open/close tag for tags
> which chomp the preceeding/next character if it is a newline. Eg:

> {^ for item in folder ^}

I don't think adding new reserved characters would make the language
simpler for template authors, nor for the the template parser, nor for
the sake of backwards compatibility. {# is already part of it.

But I can see the need to chomp whitespace before a template tag, as
well as the newline after it.

Martin J. Laubach wrote:
> For this (avoiding newlines) the currently discussed multiline tags
> would work pretty well too without adding more cruft to the template
>language:
>
> foo bar {#
> #}baz

If this can be accomplished without massive performance penalties, I
agree with you.

Maybe {# #} could be made multiline at first, with special code, while
waiting for a proper implementation of generic multiline tags. This
would certainly be more forward-compatible than my proposal above
and it would solve more whitespace problems, if not all.

Tobia

[Jonathan French]

Jinja implements whitespace control by putting a minus sign after/before the % in a tag - http://jinja.pocoo.org/docs/templates/#whitespace-control - I haven't tried it myself, but it looks like {% tag -%} is equivalent to your {% tag %}{# .

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

[Florian Apolloner]

Hi,

On Friday, February 24, 2012 7:48:29 PM UTC+1, ojno wrote:

Jinja implements whitespace control by putting a minus sign after/before the % in a tag - http://jinja.pocoo.org/docs/templates/#whitespace-control - I haven't tried it myself, but it looks like {% tag -%} is equivalent to your {% tag %}{# .

Yes it is, and why nicer and more readable imo, using {# #} to achieve that effect is a horrible idea imo.

[Florian Apolloner]

s/why/way/

sry :(

[Tai Lee]

Adding more symbols to existing tags (e.g. {^ for x in y ^} or {% for
x in y -%}), multi-line comment tags that don't actually include a
comment, and half baked comment tags (where the closing tag is
assumed) are all going to make templates uglier, and harder to read.

The comment tag based solutions don't even solve the problem
completely, because leading white space is still there, and users are
still required to carefully position comment tags all around block
tags to remove whitespace.

What's better about {% for x in y -%} or {^ for x in y ^} over {%
stripwhitespace on %} at the top of your template, followed by {% for
x in y %}? I think the latter is far more readable.

My ideal solution is not to add new ways to mark each individual
template tag that should have surrounding white space stripped, but to
simply enable the removal of lines that have only block tags and no
actual content. 99% of the time this is the right thing to do, and we
just need a deprecation path and new template tag so that template
authors can opt-in to the new behaviour now.

Cheers.
Tai.

[Florian Apolloner]

Hi,

On Saturday, February 25, 2012 5:27:43 AM UTC+1, Tai Lee wrote:

Adding more symbols to existing tags (e.g. {^ for x in y ^} or {% for
x in y -%}), multi-line comment tags that don't actually include a
comment, and half baked comment tags (where the closing tag is
assumed) are all going to make templates uglier, and harder to read.

"-" is only supposed to be used in some edge cases where it's really important that you don't have whitespace, usually there is no reason to care for whitespace at all -- so it shouldn't make templates that ugly if you don't use it everywhere.

{% stripwhitespace on %} at the top of your template, followed by {% for
x in y %}? I think the latter is far more readable.

Usually I don't want to strip everything, but just at one location.

 

My ideal solution is not to add new ways to mark each individual
template tag that should have surrounding white space stripped, but to
simply enable the removal of lines that have only block tags and no
actual content. 99% of the time this is the right thing to do, and we
just need a deprecation path and new template tag so that template
authors can opt-in to the new behaviour now.

And what about the other 1%, there are even cases where whitespace is important ;) Either way, why is stripping whitespace the right thing to do? Given my experience "I don't care if there is or not" is what's 99% true of the time…

Cheers,
Florian

[Anssi Kääriäinen]

In most situations white space matters:
{{ user.lastname }} {{ user.firstname }}

- AnssiKääriäinen

[Florian Apolloner]

Hi,

On Saturday, February 25, 2012 10:04:21 AM UTC+1, Anssi Kääriäinen wrote:

In most situations white space matters:
{{ user.lastname }} {{ user.firstname }}

Right, but
"""
{{ user.lastname }}
{{ user.firstname }}
"""
would have produced exactly the same output in HTML, hence my statement that you usually don't have to care about it that much. Eg, you need to make sure that there is whitespace but it doesn't matter how much since it's collapsed anyways (and outside of tags it doesn't matter at all…)

Cheers,
Florian

[Anssi Kääriäinen]

I was just illuminating why putting {% stripwhitespace on %} in the
beginning of the template might not be that great an idea. Even with
the line change between the first and lastname they would be
concatenated if all whitespace is stripped.

I think we agree on this issue. Maybe I should have quoted the
original poster of the idea instead...

- Anssi

[Tai Lee]

I think this discussion is focusing on template tags, not template variables. Maybe even a subset of template tags (e.g. block level tagsif, for, block, filter, etc). Template variables and inline tags (e.g. now) shouldn't have white space stripped.

In 100% of cases I can think of I either wouldn't care (in HTML templates) or I would want lines that only contain block level tags and no actual content removed from the rendered output of the template.

The only time I would explicitly NOT want this to happen is when I have an existing template where white space matters and it has been carefully crafted to work around Django's white space issues. I think SmileyChris' solution from the ticket is the way to go, but in a backwards compatible way. Make it opt-in with a new template tag, similar to the way auto escaping is handled, and optionally a deprecation path that will make this the default eventually.

I don't think anyone has suggested stripping ALL white space around ALL template tags, or even ANY template variables.

Cheers.
Tai.

[mjl Martin J. Laubach]

foo bar {#

#}baz 

You just made a strong argument against multiline tags: I wouldn't want to see them abused this way!

  Ah, I fully expected to be shot down on "aesthetic" reasons. I still think it's better to have some (maybe slightly ugly) way to do things than no way and revelling in perceived beauty, and if it's as easy as the above, well, then so be it.

[Jonathan French]

Let me make sure I've got this right --- The situation being discussed is not where whitespace is insignificant and can be stripped, but where whitespace is important and you want to control the exact amount of it, e.g. plain text emails. In this case, just using stripwhitespace is not enough. Right?

[Tobia]

Jonathan French wrote:
> Let me make sure I've got this right --- The situation being discussed is
> not where whitespace is insignificant and can be stripped, but where
> whitespace is important and you want to control the exact amount of it,
> e.g. plain text emails. In this case, just using stripwhitespace is not
> enough. Right?

Yes, that is the main problem.

Django templates are useful (and used) for all sorts of text-based
formats. For some of these, an exact control over whitespace is needed
throughout the entire file (eg. text/plain), while for others, control
is needed in some sensitive places (eg. element attributes in html and
xml, which according to standars are not allowed to contain literal
newlines.)

The only option currently available to control whitespace is *not to
use any* for the template syntax itself, in effect writing long one-
liners. The admin site contains several examples of such:

<div class="inline-related{% if forloop.last %} empty-form last-
related{% endif %}" id="{{ inline_admin_formset.formset.prefix }}-{%
if not forloop.last %}{{ forloop.counter0 }}{% else %}empty{% endif
%}">

Yes, in this case whitespace could be added between the class="" and
id="" attributes, but the main issue remains there.

Multiline comments {# #} and/or generic multiline tags would offer a
more workable alternative than the present situation, allowing for
syntax-only whitespace in templates. But they admittedly suffer from
some serious ugliness and "denaturation" of Django templates.

After reading all the above, I can see the arguments against this:

<div class="inline-related{#
#}{% if forloop.last %}{#
#} empty-form last-related{#
#}{% endif %}{#
#}" id="{{
...

or even this:

<div class="inline-related{%
if forloop.last
%} empty-form last-related{%
endif
%}" id="{{
...

Ugh!!

This *will* be abused the instant it gets pushed, even if it's done on
good intentions, such as allowing for multiline {% trans %}, in effect
taking the beauty and readability out of the template language.

But the main issue is still there, with its huge one-liners and other,
even less readable workarounds. I think a new idea is needed,
something more fine-grained than stripwhitespace and less denaturating
than multiline comments. (Yes, I'm officially recognizing my original
idea as flawed. Sorry for that. :-)

Tobia

[Anssi Kääriäinen]

On Feb 27, 1:01 pm, Tobia <tobia.confo...@gruppo4.eu> wrote:
> After reading all the above, I can see the arguments against this:
>
> <div class="inline-related{#
>     #}{% if forloop.last %}{#
>         #} empty-form last-related{#
>     #}{% endif %}{#
> #}" id="{{
>     ...
>
> or even this:
>
> <div class="inline-related{%
>     if forloop.last
>         %} empty-form last-related{%
>     endif
> %}" id="{{
>     ...
>
> Ugh!!

Agreed. Seeing that just changed me from +1 to -0 for multiline tags.
Although the multi-line comment syntax isn't that bad, it is at least
clear what it does and why.

Now that I look at it, the main problem is that you want the <div>
appear to be on one line in the HTML. So, it should be one-liner in
the template too, or otherwise it will look strange. However, dealing
with div tags isn't the hard case, it is the email template. Django
doesn't have a good answer for that. Multi-variable include/with/
blocktrans tags are a problem too.

When reading the latest template-related threads on this list it is
pretty clear that there is no right answer for template design
philosophy questions. For example I use mainly Jinja2 for two reasons.
I need to generate largish reports, and Jinja2 beats Django's template
engine in performance by an order of magnitude. In addition, Jinja's
design philosophy fits my needs better than Django's. But, I do not
claim Jinja2 is the right answer for everybody.

We will just need to accept that it is impossible to create a template
language that fits everybody's needs perfectly. If you want something
different, use something different. I think good ideas how to make
life easier for external template engine users are welcome.

- Anssi

[Leon Matthews]

> Django templates are useful (and used) for all sorts of text-based
> formats. For some of these, an exact control over whitespace is needed
> throughout the entire file (eg. text/plain),

Agreed. When I started using Django templates I was very surprised at
the output I was seeing.

After a while I realised that the thing that suprised me was that
newlines were being output for what I considered to be just
'directives' -- that is, lines that contained tags, not content. For
example, {% if %} and {% endif %} tags on lines by themselves.

Honestly, my use case then was to make my HTML look pretty, so was
able to let it go pretty easily. Recently however I used the template
system to produce an email with a textual pricing table, and there the
lack of easy whitespace control hurt there.

Would it be feasible to add some logic, something along the lines of:

"Template lines containing just tags with no literal content do not
produce a line in the output (unless of course the tag itself
produces one)"

After all, new lines are easy to add, but currently painful to suppress...

Cheers,

Leon

--
Leon Matthews BSc
Technical Director, Messiah Ltd.
work: http://messiah.co.nz/
home: http://lost.co.nz/

[Tai Lee]

On Feb 29, 8:15 am, Leon Matthews <pyt...@lost.co.nz> wrote:
> Would it be feasible to add some logic, something along the lines of:
>
> "Template lines containing just tags with no literal content do not
> produce a line in the output  (unless of course the tag itself
> produces one)"

I believe that is what has been suggested, and tickets produced by
SmileyChris and worked on by others to that end, at
https://code.djangoproject.com/ticket/2594

The ticket is still accepted (and is now 6 years old), and the
consensus from this thread now appears to be that the "dnl" and multi-
line comment solutions put forward here are not preferred.

I think the next step is for any interested parties to look at the
latest patches on that ticket, update them for trunk and make sure
they still work, have the required tests and/or docs, and try to get
it committed.

Cheers.
Tai.

[Leon Matthews]

> I believe that is what has been suggested, and tickets produced by
> SmileyChris and worked on by others to that end, at
> https://code.djangoproject.com/ticket/2594

That's... quite fantastic. A actual patch for the exactly the
behaviour I had in mind.

Why was it not commited for 1.3 or 1.4? From my perspective it's a
big win, and reading the bug notes, the issues -- backwards
combatibility, docs, tests -- seem to have been addressed.

It looks like a lot of work looks went into this patch, why did it stall?

Leon

[Carl Meyer]

On 03/01/2012 01:28 PM, Leon Matthews wrote:
> It looks like a lot of work looks went into this patch, why did it stall?

Same reason any ticket stalls - it seems that nobody felt strongly
enough about it to put the time into reviewing and thoroughly testing
the patch and marking it Ready for Checkin. If you'd like to see it in
(post 1.4 at this point, of course), feel free to do that!

Carl

[Leon Matthews]

> Same reason any ticket stalls - it seems that nobody felt strongly
> enough about it to put the time into reviewing and thoroughly testing
> the patch and marking it Ready for Checkin. If you'd like to see it in
> (post 1.4 at this point, of course), feel free to do that!

Fair enough. I realised after posting that my question was
particularly poorly timed, given where we are in the release cycle. I
apologise for that -- I was overcome with excitement to find that
somebody had already written the patch I wanted!

[Leon Matthews]

On 2 March 2012 09:45, Carl Meyer <ca...@oddbird.net> wrote:
> Same reason any ticket stalls - it seems that nobody felt strongly
> enough about it to put the time into reviewing and thoroughly testing
> the patch and marking it Ready for Checkin. If you'd like to see it in
> (post 1.4 at this point, of course), feel free to do that!

Done! :-)

I've created a new version of the patch against Django 1.5-dev, which
passes all tests, etc...

I've attached it to the ticket directly:
https://code.djangoproject.com/ticket/2594

Cheers,

   Leon

[Stephen Kelly]

Oh, interesting that this is getting interest again. I think I emailed the
list annually about it without getting a reply :).

I'll review again soon too.

Thanks,

Steve.