Feature proposal: escape hatch for colliding template syntax in django templates

[David Gouldin]

As client-side templates become more popular, it is increasingly
likely that django's template language will not be the only one
present in a template. (jQuery's new template language makes frequent
use of curly braces in its syntax.) At the same time, the assumption
that all template syntax should be parsed and rendered on the server
should be questioned.

I've created an effective albeit gross hack of a template tag to
accomplish this goal:

http://gist.github.com/634534

I understand this can already be accomplished with {% ssi %}, but the
absolute path requirement and need for a separate file for even the
smallest bit of client template syntax is a turn-off to me. A
template tag in the same vein as {% autoescape %} and {% comment %}
seems a much more straightforward way to tackle the problem.

Thoughts/opinions?

[flo...@gmail.com]

I agree that something like this is becoming more necessary, as I ran
into this a few days ago myself.
Funny how similar our strategies were (David's is more robust):
http://gist.github.com/629508

-Eric Florenzano

[Jacob Kaplan-Moss]

On Tue, Oct 19, 2010 at 12:24 PM, David Gouldin <dgou...@gmail.com> wrote:
> Thoughts/opinions?

Looks like a good idea to me. I've certainly used ssi as a hack for
this before, so getting a noparse/verbatim tag into Django sounds
great.

Jacob

[David Gouldin]

Given 2 +1s, I've added a ticket:

http://code.djangoproject.com/ticket/14502

On Oct 19, 6:49 pm, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:

[James Bennett]

On Tue, Oct 19, 2010 at 1:49 PM, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:
> Looks like a good idea to me. I've certainly used ssi as a hack for
> this before, so getting a noparse/verbatim tag into Django sounds
> great.

If we're going to do this, could we also look at deprecating the
'templatetag' template tag? There are a couple cases a 'verbatim' tag
wouldn't cover that 'templatetag' wouldn't, but I'm kinda hard-pressed
to think of when they'd ever come up in reality.

--
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."

[Jacob Kaplan-Moss]

On Tue, Oct 19, 2010 at 4:31 PM, James Bennett <ubern...@gmail.com> wrote:
> If we're going to do this, could we also look at deprecating the
> 'templatetag' template tag? There are a couple cases a 'verbatim' tag
> wouldn't cover that 'templatetag' wouldn't, but I'm kinda hard-pressed
> to think of when they'd ever come up in reality.

Good idea.

Maybe combine them? {% verbatim "{{{" %} or {% verbatim %} ... {% endverbatim %}

Jacob

[Łukasz Rekucki]

We could also add something like a "boundary" argument just in case
someone will want to escape "{% endverbatim %}":

{% verbatim boundary="my_random_string" %}{% endverbatim %}{%
endverbatim_my_random_string %}

--
Łukasz Rekucki

[Stephen Kelly]

Łukasz Rekucki wrote:

> On 19 October 2010 23:41, Jacob Kaplan-Moss
> <ja...@jacobian.org> wrote:
>> On Tue, Oct 19, 2010 at 4:31 PM, James Bennett
>> <ubern...@gmail.com> wrote:
>>> If we're going to do this, could we also look at deprecating the
>>> 'templatetag' template tag? There are a couple cases a 'verbatim' tag
>>> wouldn't cover that 'templatetag' wouldn't, but I'm kinda hard-pressed
>>> to think of when they'd ever come up in reality.
>>
>> Good idea.
>>
>> Maybe combine them? {% verbatim "{{{" %} or {% verbatim %} ... {%
>> endverbatim %}

I'm pretty sure the current lexer for django templates can't handle this
very well.

[Stephen Kelly]

Stephen Kelly wrote:

> Łukasz Rekucki wrote:
>
>> On 19 October 2010 23:41, Jacob Kaplan-Moss
>> <ja...@jacobian.org> wrote:
>>> On Tue, Oct 19, 2010 at 4:31 PM, James Bennett
>>> <ubern...@gmail.com> wrote:
>>>> If we're going to do this, could we also look at deprecating the
>>>> 'templatetag' template tag? There are a couple cases a 'verbatim' tag
>>>> wouldn't cover that 'templatetag' wouldn't, but I'm kinda hard-pressed
>>>> to think of when they'd ever come up in reality.
>>>
>>> Good idea.
>>>
>>> Maybe combine them? {% verbatim "{{{" %} or {% verbatim %} ... {%
>>> endverbatim %}
>
> I'm pretty sure the current lexer for django templates can't handle this
> very well.

Sorry. Sent too early. All thumbs today. Consider these examples:

{% verbatim "%} %}" %}

(That is, "%} %}" in a verbatim-no-end tag)

{% verbatim %} %} %} {% endverbatim %}

(That is, " %} %} " wrapped in verbatim tags)

The current lexer uses regexps to find tokens like that. It would need to be
completely rewritten/redesigned to handle these cases.

All the best,

Steve.

[Stephen Kelly]

Stephen Kelly wrote:

> Stephen Kelly wrote:
>
>> Łukasz Rekucki wrote:
>>
>>> On 19 October 2010 23:41, Jacob Kaplan-Moss
>>> <ja...@jacobian.org> wrote:
>>>> On Tue, Oct 19, 2010 at 4:31 PM, James Bennett
>>>> <ubern...@gmail.com> wrote:
>>>>> If we're going to do this, could we also look at deprecating the
>>>>> 'templatetag' template tag? There are a couple cases a 'verbatim' tag
>>>>> wouldn't cover that 'templatetag' wouldn't, but I'm kinda hard-pressed
>>>>> to think of when they'd ever come up in reality.
>>>>
>>>> Good idea.
>>>>
>>>> Maybe combine them? {% verbatim "{{{" %} or {% verbatim %} ... {%
>>>> endverbatim %}
>>
>> I'm pretty sure the current lexer for django templates can't handle this
>> very well.
>
> Sorry. Sent too early. All thumbs today. Consider these examples:
>
> {% verbatim "%} %}" %}
>
> (That is, "%} %}" in a verbatim-no-end tag)
>
> {% verbatim %} %} %} {% endverbatim %}
>
> (That is, " %} %} " wrapped in verbatim tags)
>
> The current lexer uses regexps to find tokens like that. It would need to
> be completely rewritten/redesigned to handle these cases.

One option might be to extend the template syntax with {$ verbatim $}

(I know that's not a popular suggestion)

Normal text {$ Verbatim {% text {% {{ %} $}

It would be simple to add, but of course it still leaves the same issue of
using the '$}' inside the tokens.

[David Danier]

> If we're going to do this, could we also look at deprecating the
> 'templatetag' template tag? There are a couple cases a 'verbatim' tag
> wouldn't cover that 'templatetag' wouldn't, but I'm kinda hard-pressed
> to think of when they'd ever come up in reality.

+1 for this. I for one don't even use {% templatetag %}, as {{ "{{" }}
is so much easier.

Besides adding some non-rendering-tag I would love to see a
rendering-tag to allow two way rendering for caching/performance
reasons, example:
Usage:
{% render %}{% cache ... %}
this will be cached content: FOOBAR {{ some_var }}
this will not be cached content: {% verbatim %}{{ user }}{% verbatim
%}
{% endcache %}{% endrender %}

I could even provide some code for a {% render %}-tag, as I use
something like this in production.

David

[Andrew Godwin]

On 20/10/10 02:40, Stephen Kelly wrote:
> Sorry. Sent too early. All thumbs today. Consider these examples:
>
> {% verbatim "%} %}" %}
>
> (That is, "%} %}" in a verbatim-no-end tag)
>
> {% verbatim %} %} %} {% endverbatim %}
>
> (That is, " %} %} " wrapped in verbatim tags)
>
> The current lexer uses regexps to find tokens like that. It would need to be
> completely rewritten/redesigned to handle these cases.
>
> All the best,
>
> Steve.
>
>

Are you sure? There's no nesting here, so I'm reasonably sure this could
be a regular language, though I don't want to sit down and prove that.

Instead, as an engineer, a regex that can distinguish the two:

In [38]: re.split(re.compile(r'{% \s* ( (?: [\w\-_\s]+ ) (?: \s* \"
[\w\-_\s%}{]+ \" \s*)* ) \s* %}', re.VERBOSE), '{% foo "%}" %} {% endfoo
%}')
Out[38]: ['', 'foo "%}" ', ' ', 'endfoo ', '']

In [39]: re.split(re.compile(r'{% \s* ( (?: [\w\-_\s]+ ) (?: \s* \"
[\w\-_\s%}{]+ \" \s*)* ) \s* %}', re.VERBOSE), '{% foo %} %} {% endfoo %}')
Out[39]: ['', 'foo ', ' %} ', 'endfoo ', '']

(the key here is asserting the even number of quote marks, something a
regular language is capable of expressing)

It's a bit early in the morning for in-depth regexes, but that seems to
show that it is _probably_ possible. Whether we should be continuing to
use the regex-based parser or moving to a proper lexing/tokenising one
is a different question, but if we did a parser rewrite it wouldn't be
able to land until 1.4 now, I imagine.

Andrew

[Johannes Dollinger]

Am 20.10.2010 um 10:40 schrieb Andrew Godwin:

> On 20/10/10 02:40, Stephen Kelly wrote:
>> Sorry. Sent too early. All thumbs today. Consider these examples:
>>
>> {% verbatim "%} %}" %}
>>
>> (That is, "%} %}" in a verbatim-no-end tag)
>>
>> {% verbatim %} %} %} {% endverbatim %}
>>
>> (That is, " %} %} " wrapped in verbatim tags)
>>
>> The current lexer uses regexps to find tokens like that. It would need to be
>> completely rewritten/redesigned to handle these cases.
>>
>> All the best,
>>
>> Steve.
>>
>>
>
> Are you sure? There's no nesting here, so I'm reasonably sure this could be a regular language, though I don't want to sit down and prove that.

A verbatim tag would at least introduce ambiguity:

{% verbatim %}{% endverbatim %}{% verbatim %}{% endverbatim %}

This could be parsed as [Verbatim(""), Verbatim("")] or [Verbatim("{% endverbatim %}{% verbatim %}")].

For the inline case, e.g. {% verbatim "%}" %}, it might be cleaner to just allow template tag delimiters in strings inside var nodes - the lexer would have to be modified anyway:

{{ '%}' }} or even {{ foo|prefix:'{%'|suffix:'%}' }}

__
Johannes

[Mantas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I propose this solution:

{!x x}

Or at least:

{% verbatim x %}
{% x %}

Where 'x' can be replaced by any other string:

{!! {!x x} !}

{!xx {!x x} xx}

{!maystring {!x x} maystring}

{% verbatim - %}

{% verbatim x %}
{% x %}

{% - %}

For example, this::

{!!
{% block content %}
{% endblock %}
!}

or

{% verbatim - %}

{% block content %}
{% endblock %}

{% - %}

Will be parsed as::

{% block content %}
{% endblock %}

This proposal is very similar to heredoc [1]:

<< EOF
some content
EOF

[1]: http://en.wikipedia.org/wiki/Here_document

- --
Mantas aka sirex
__o /\
_ \<,_ -- launchpad.net/~sirex -- /\/ \
___(_)/_(_)_____________________________/_/ \
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1N228ACgkQrRc2Mrd7P8X/OACfSXLl7/Z8z0CULLIFsDcUXoZW
x40AoJZK4s5IfEeTG6u3nE/Uk+ekYEII
=/QkB
-----END PGP SIGNATURE-----