Templates: short comments {##} eats text

[tonnzor]

Short templates comments {# #} are odd if used incorrectly - it eats
text.

This template (invalid comments):

Hello
{# this
comment
has
newlines #} World! {{ "some more text" }}

Now produce this output (text "World!" is eaten):

Hello
some more text

Yes, the comments are invalid, but in this case it should be ignored
and printed as-is:

Hello
{# this
comment
has
newlines #} World! some more text

I'm sure that things must work or do not work, not "work sometimes" or
"do not work and have some side effects".

See also ticket #3888 (http://code.djangoproject.com/ticket/3888)

[Jonas Maurus]

If it can be changed, I'd also like to see only one comment style: {#
[text] #} with the {##}-notation supporting multiple-line comments. {%
comment %} can either get continued support or be deprecated.

I don't agree with the comment on #3888 that {# ... #} is like "//" in
C-like languages as "//" doesn't require a terminator (like "*/") and
therefor always effects the full line. Thus, "{# ... #}" is much more
like "/* ... */" or "<!-- -->". A "start-comment" - text - "end
comment" style notation should treat all included text as a comment,
including newline and carriage-return characters. Under no
circumstances should it affect anything coming *after* it, so that is
certainly a valid bug. At least an error message has to be generated
in this case, at which point we could also just allow multi-line {##}-
style comments.

I'd like that functionality especially as it would allow for the easy
construction of well-formatted templates that produce spaceless mark-
up like:
<div class="blah">{# eliminate the newline
#}<span>xyz</span>{#
#}</div>

at least until the spaceless-filter is changed so it actually produces
spaceless mark-up :-) and it would allow for well formatted comments
like:
{#
# multi-line comment
#
#}

which I find more aesthetically pleasing than the {% comment %} - {%
endcomment %} blocks.

At least a consistent design decision is needed for the 1.0-release.

cheers and happy easter
Jonas

[SmileyChris]

tonnzor wrote:
> Yes, the comments are invalid, but in this case it should be ignored
> and printed as-is:
>
> Hello
> {# this
> comment
> has
> newlines #} World! some more text

This is a bug with the template lexer. Any token starting with open
tag text is incorrectly treated as a node of that type, even if it is
a multi-line token.

[Gabriel Farrell]

On Apr 6, 3:31 pm, "Jonas Maurus" <jonas.mau...@gmail.com> wrote:
>
> If it can be changed, I'd also like to see only one comment style: {#

> [text] #} with the {##}-notation supportingmultiple-linecomments. {%

> comment %} can either get continued support or be deprecated.
>
> I don't agree with the comment on #3888 that {# ... #} is like "//" in
> C-like languages as "//" doesn't require a terminator (like "*/") and
> therefor always effects the full line. Thus, "{# ... #}" is much more
> like "/* ... */" or "<!-- -->". A "start-comment" - text - "end
> comment" style notation should treat all included text as a comment,
> including newline and carriage-return characters. Under no
> circumstances should it affect anything coming *after* it, so that is
> certainly a valid bug. At least an error message has to be generated
> in this case, at which point we could also just allow multi-line {##}-
> stylecomments.
>

Big +1 on this. "{# ... #}" looks like "/* ... */", and that's the
expected behavior. It should be quick and easy to do multiline
comments -- even quicker and easier than "{% comment %} ... {%
endcomment %}".

[Lakin Wecker]

+1 on this as well.  Another reason to allow newlines inside tags.  Whitespace is whitespace is whitespace.

Lakin

[Simon G.]

A patch, some docs and a few tests would be most welcome.

--Simon

[tonnzor]

I reviewed templates engine and found the problem. Here are the
patches:

1. template.Lexer incorrectly recognizes ALL tokens - it check only
the beginning of the token, ignoring the end of it.

Example:

{{ my_var }} {{ other_var ##<EOF> : variable other_var will be printed

Current code (template/__init__.py, line 203):

if token_string.startswith(VARIABLE_TAG_START):
token = Token(TOKEN_VAR,
token_string[len(VARIABLE_TAG_START):-len(VARIABLE_TAG_END)].strip())
elif token_string.startswith(BLOCK_TAG_START):
token = Token(TOKEN_BLOCK,
token_string[len(BLOCK_TAG_START):-len(BLOCK_TAG_END)].strip())
elif token_string.startswith(COMMENT_TAG_START):

Fixed code:

if token_string.startswith(VARIABLE_TAG_START) and
token_string.endswith(VARIABLE_TAG_END):
token = Token(TOKEN_VAR,
token_string[len(VARIABLE_TAG_START):-len(VARIABLE_TAG_END)].strip())
elif token_string.startswith(BLOCK_TAG_START) and
token_string.endswith(BLOCK_TAG_END):
token = Token(TOKEN_BLOCK,
token_string[len(BLOCK_TAG_START):-len(BLOCK_TAG_END)].strip())
elif token_string.startswith(COMMENT_TAG_START) and
token_string.startswith(COMMENT_TAG_START):

2. If we want to enable multi-line comments, we need to allow newlines
in tag's content:

Current code (template/__init__.py, line 91):

tag_re = re.compile('(%s.*?%s|%s.*?%s|%s.*?%s)' %
(re.escape(BLOCK_TAG_START), re.escape(BLOCK_TAG_END),

re.escape(VARIABLE_TAG_START), re.escape(VARIABLE_TAG_END),

re.escape(COMMENT_TAG_START), re.escape(COMMENT_TAG_END)))

Fixed code:

tag_re = re.compile('(%s.*?%s|%s.*?%s|%s[\s\S]*?%s)' %
(re.escape(BLOCK_TAG_START), re.escape(BLOCK_TAG_END),

re.escape(VARIABLE_TAG_START), re.escape(VARIABLE_TAG_END),

re.escape(COMMENT_TAG_START), re.escape(COMMENT_TAG_END)))

3. If we want enable newlines for any tags, we should allow newlines
in all tags content:

Example:

{{ "this_is_a_" }}{%
if not short
%}long{%
else
%}short{% endif
%}_spaceless_{%
for keyword in keywords
%}{{ keyword }}{%
endfor %}_text

I'm sure this will make templates much more clear and readable and
having no ugly spaces in output!

Current code (template/__init__.py, line 91):

tag_re = re.compile('(%s.*?%s|%s.*?%s|%s.*?%s)' %
(re.escape(BLOCK_TAG_START), re.escape(BLOCK_TAG_END),

re.escape(VARIABLE_TAG_START), re.escape(VARIABLE_TAG_END),

re.escape(COMMENT_TAG_START), re.escape(COMMENT_TAG_END)))

Fixed code:

tag_re = re.compile('(?s)(%s.*?%s|%s.*?%s|%s.*?%s)' %
(re.escape(BLOCK_TAG_START), re.escape(BLOCK_TAG_END),

re.escape(VARIABLE_TAG_START), re.escape(VARIABLE_TAG_END),

re.escape(COMMENT_TAG_START), re.escape(COMMENT_TAG_END)))

[Malcolm Tredinnick]

On Mon, 2007-04-23 at 17:46 +0000, tonnzor wrote:
> I reviewed templates engine and found the problem. Here are the
> patches:
>
> 1. template.Lexer incorrectly recognizes ALL tokens - it check only
> the beginning of the token, ignoring the end of it.
>

[... code snipped ...]

Please put any patches in Trac, rather than here. That way they won't
get lost.

>
> 2. If we want to enable multi-line comments, we need to allow newlines
> in tag's content:

We don't want to allow newlines in tags. That decision has already been
made and reaffirmed a number of times.

Regards,
Malcolm

[Lakin Wecker]

> 2. If we want to enable multi-line comments, we need to allow newlines
> in tag's content:

We don't want to allow newlines in tags. That decision has already been
made and reaffirmed a number of times.

I know that the devs probably don't want anyone to bring this up again, so feel free to ignore me, but I really need to get this off my chest.

I'm extremely frustrated by the newlines decision.  The django devs had admitted that the decision is an aesthetic one.   This means that they have decreed something that could be left up to aesthetics.   If the django devs don't like newlines in their tags they can make that a django svn code-formatting rule.  But please don't force it down the throat of everyone who uses your template language.  

Additionally, phrasing it as an aesthetic decision makes it seem as if valid use cases will be ignored.  If there are valid uses cases which add value, like this one and possibly others, then they should be accumulated and the decision revisited.  

I've had a few use cases that I thought were valid.  The ticket regarding newlines has been effectively shut-down and we are told to bring it up here.  I did, and was ignored.  Which means that the list of possibly valid use cases are now scattered across the group archives.

Lakin

[Michael Radziej]

On Wed, Apr 25, Lakin Wecker wrote:

> I know that the devs probably don't want anyone to bring this up again, so
> feel free to ignore me, but I really need to get this off my chest.
>
> I'm extremely frustrated by the newlines decision.

I understand you, but can only recommend what I usually recommend when
someone disagrees with the project leaders' decisions: patch it. It's open
source, and this particular patch shouldn't be a hassle to maintain and
merge with future changes in Django.

My own patch stack currently comprises 22 patches ... and it's still ok and
quite manageable. I'm still up to date with svn.

I learned that it simply isn't worth all the discussions that lead to
nowhere. It's more work than to do it on your own. It's not my project, and
I'll have to accept what comes. (And that's why I involve myself now only in
things that look fruitful).

Patch it, be happy. :-)

Michael

--
noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg -
Tel +49-911-9352-0 - Fax +49-911-9352-100
http://www.noris.de - The IT-Outsourcing Company

Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel, Hansjochen Klenk -
Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689

[Andrew Durdin]

On Apr 25, 4:46 pm, Michael Radziej <m...@noris.de> wrote:
>
> My own patch stack currently comprises 22 patches ... and it's still ok and
> quite manageable. I'm still up to date with svn.

I've been using a patched version of Django for some time, but it's
highly inconvenient (I'm using a "vendor branch" strategy as described
in the subversion book); so I'm looking for a different way to manage
this sort of thing.

What tools do you use to manage your patched version?

Andrew

[Michael Radziej]

You need a tool that works on a patch level, and not on a branch level. If
you cannot find out to which patch a merge conflict belongs, you're on the
way to madness.

I use pg ("patchy git"), which is a patch system built on top of git. But if you are not
familiar with git (and don't want to learn it), take a look at quilt. quilt
only organizes patches, and you can then put the result or the patches in
another repository, if you like. I haven't used it, though.

pg: http://www.spearce.org/category/projects/scm/pg/
quilt: http://savannah.nongnu.org/projects/quilt

Unfortunately, the pg site is currently broken, but you can still get a git
clone of the project (docs included) with

git --clone http://www.spearce.org/projects/scm/pg.git

[Ramiro Morales]

On 4/25/07, Andrew Durdin <adu...@gmail.com> wrote:
> I've been using a patched version of Django for some time, but it's
> highly inconvenient (I'm using a "vendor branch" strategy as described
> in the subversion book); so I'm looking for a different way to manage
> this sort of thing.
>
> What tools do you use to manage your patched version?

I know there is people using git and quilt for this task, and
this Gary Wilson blog entry about using bzr:

http://gdub.wordpress.com/2007/01/11/hacking-django-how-bazaar/

I documented the scheme I use (it involves Darcs, but I think
something similar can be implemented using any distributed
revision system) some weeks ago to django-users:

http://groups.google.com/group/django-users/browse_frm/thread/31402bfa5c86976d/f51629dc75692593

Regards,

--
Ramiro Morales

[David Danier]

> What tools do you use to manage your patched version?

I use svk for getting my patches updated and regenerated if changes
happened to the trunk and some lame gentoo-ebuild to install django
including my patches.

I pasted my ebuild here:
http://dpaste.com/hold/9042/

Some script to rebuild the patches:
http://dpaste.com/hold/9043/

svk patch help:
http://dpaste.com/hold/9044/

svk Homepage:
http://svk.bestpractical.com/

Greetings, David Danier

[Lakin Wecker]

Thanks Michael,

Now that you actually said it, I realize that this is the conclusion I've been slowly coming too.  Thanks for listening.

Lakin

[Malcolm Tredinnick]

On Wed, 2007-04-25 at 08:38 -0600, Lakin Wecker wrote:
>
> > 2. If we want to enable multi-line comments, we need to
> allow newlines
> > in tag's content:
>
> We don't want to allow newlines in tags. That decision has
> already been
> made and reaffirmed a number of times.
>
> I know that the devs probably don't want anyone to bring this up
> again, so feel free to ignore me, but I really need to get this off my
> chest.
>
> I'm extremely frustrated by the newlines decision. The django devs
> had admitted that the decision is an aesthetic one.

That was one of the reasons. Another reason is that the first reason
doesn't preclude any functionality and there weren't any really good
use-cases mentioned that warranted making the change. There are also
technical reasons to not make the change (for example, error trapping
can occur much earlier).

Please don't reframe our reasoning as selecting one particular point to
the exclusion of all others. Trying to portray the core developers as
not considering other viewpoints is not particularly fair or
constructive.

No functionality is lost whichever decision is made and at times we just
have to make a decision and move on.

Regards,
Malcolm

[Andrew Durdin]

Thanks Michael, Ramiro, and David. I'll read up on the things you've
suggested.

[tonnzor]

> > 2. If we want to enable multi-line comments, we need to allow newlines
> > in tag's content:
>
> We don't want to allow newlines in tags. That decision has already been
> made and reaffirmed a number of times.

Comments are not actually a tag, it's a special case. And it is very
useful to have multiline comments (see comments above).

So why can't we enable newlines for {##} only?

[Malcolm Tredinnick]

On Thu, 2007-04-26 at 14:12 +0000, tonnzor wrote:
> > > 2. If we want to enable multi-line comments, we need to allow newlines
> > > in tag's content:
> >
> > We don't want to allow newlines in tags. That decision has already been
> > made and reaffirmed a number of times.
>
> Comments are not actually a tag, it's a special case. And it is very
> useful to have multiline comments (see comments above).

We already have multiline comments. They are written as {% comment %}
and {% endcomment %}.

> So why can't we enable newlines for {##} only?

This tag is parsed just as normal tags, so introducing newlines in this
would affect everything.

Please let this thread die.

Regards,
Malcolm

[tonnzor]

> We already have multiline comments. They are written as {% comment %}
> and {% endcomment %}.

This inconveniently to use. Short variant {##} is much better and very
similar to C /* */.

> > So why can't we enable newlines for {##} only?
>
> This tag is parsed just as normal tags, so introducing newlines in this
> would affect everything.

This is not true. If you see the code, you would see that variable
print, tag and comment have separate pattern.

Here's the listing (template/__init__.py, line 90:)

# match a variable or block tag and capture the entire tag, including
start/end delimiters

tag_re = re.compile('(?s)(%s.*?%s|%s.*?%s|%s.*?%s)' %
(re.escape(BLOCK_TAG_START), re.escape(BLOCK_TAG_END),

re.escape(VARIABLE_TAG_START), re.escape(VARIABLE_TAG_END),

re.escape(COMMENT_TAG_START), re.escape(COMMENT_TAG_END)))

So, we can change pattern for each of type separately.

To enable newlines in comments ONLY:
tag_re = re.compile('(?s)(%s.*?%s|%s.*?%s|%s[\s\S]*?%s)' %

(re.escape(BLOCK_TAG_START), re.escape(BLOCK_TAG_END),

re.escape(VARIABLE_TAG_START), re.escape(VARIABLE_TAG_END),

re.escape(COMMENT_TAG_START), re.escape(COMMENT_TAG_END)))

This feature is demanded and it is easy to implement w/o side effects.
We should do it.

[Thomas Steinacher]

On Apr 26, 2007, at 4:40 PM, tonnzor wrote:

>
> This feature is demanded and it is easy to implement w/o side effects.
> We should do it.

+1

[James Bennett]

On 4/26/07, tonnzor <ton...@gmail.com> wrote:
> This feature is demanded and it is easy to implement w/o side effects.
> We should do it.

The ticket for this has been closed multiple times and Adrian has
asked that it not be reopened. Malcolm has asked that this thread be
allowed to die. That seems to indicate that a decision has been made
and that the proposed changes will not be implemented; regardless of
whether you agree with it or not, please respect that and allow the
list to move on to discussing other things.

--
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."

[Thomas Steinacher]

But then it should be at least mentioned in the documentation that
this tag works only on a single line and will cause unexpected
results when used on multiple lines (or better: raise an exception
when using multiple lines).

Thomas

[tonnzor]

OK. If the decision is made (forbid newlines in comments), then
newlines should be _really_ forbidden (currently not).

Opened a ticket for it - http://code.djangoproject.com/ticket/4171