[Django] #23424: Verbatim tag fails to render curly braces
Django
---------------------------------+-----------------------------------
Reporter: stephband | Owner: nobody
Type: Bug | Status: new
Component: Template system | Version: 1.7-rc-2
Severity: Normal | Keywords: verbatim template tag
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+-----------------------------------
The {% verbatim %} tag fails to render curly braces under certain
conditions - notably when there are two verbatim tags on the same line.
This works:
{% verbatim %}{{{% endverbatim %} – correctly renders as – {{
{% verbatim %}}}{% endverbatim %} – correctly renders as – }}
However, this fails:
{% verbatim %}{{% endverbatim %}text{% verbatim %}}{% endverbatim %} –
incorrectly renders as – {{% endverbatim %}text{% verbatim %}}
{% verbatim %}{{{% endverbatim %}text{% verbatim %}}}{% endverbatim %} –
incorrectly renders as – {{{% endverbatim %}text{% verbatim %}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23424>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: stephband | Owner: nobody
Type: Bug | Status: new
Component: Template system | Version: 1.7-rc-2
Keywords: verbatim template | Triage Stage:
tag | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by stephband):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Old description:
> The {% verbatim %} tag fails to render curly braces under certain
> conditions - notably when there are two verbatim tags on the same line.
>
> This works:
>
> {% verbatim %}{{{% endverbatim %} – correctly renders as – {{
> {% verbatim %}}}{% endverbatim %} – correctly renders as – }}
>
> However, this fails:
>
> {% verbatim %}{{% endverbatim %}text{% verbatim %}}{% endverbatim %} –
> incorrectly renders as – {{% endverbatim %}text{% verbatim %}}
> {% verbatim %}{{{% endverbatim %}text{% verbatim %}}}{% endverbatim %}
> – incorrectly renders as – {{{% endverbatim %}text{% verbatim %}}}
New description:
The {% verbatim %} tag fails to render curly braces under certain
conditions - notably when there are two verbatim tags on the same line.
This works:
{% verbatim %}{{{% endverbatim %} – correctly renders as – {{
{% verbatim %}}}{% endverbatim %} – correctly renders as – }}
{% verbatim %}{ {% endverbatim %}text{% verbatim %} }{% endverbatim %} –
correctly renders as – { text }
However, this fails:
{% verbatim %}{{% endverbatim %}text{% verbatim %}}{% endverbatim %} –
incorrectly renders as – {{% endverbatim %}text{% verbatim %}}
{% verbatim %}{{{% endverbatim %}text{% verbatim %}}}{% endverbatim %} –
incorrectly renders as – {{{% endverbatim %}text{% verbatim %}}}
{% verbatim %}{{ {% endverbatim %}text{% verbatim %} }}{% endverbatim %}
– incorrectly renders as – {{ {% endverbatim %}text{% verbatim %} }}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:1>
Django
Reporter: stephband | Owner: nobody
Type: Bug | Status: new
Component: Template system | Version: 1.7-rc-2
Keywords: verbatim template | Triage Stage:
tag | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Description changed by stephband:
Old description:
> The {% verbatim %} tag fails to render curly braces under certain
> conditions - notably when there are two verbatim tags on the same line.
>
> This works:
>
> {% verbatim %}{{{% endverbatim %} – correctly renders as – {{
> {% verbatim %}}}{% endverbatim %} – correctly renders as – }}
> {% verbatim %}{ {% endverbatim %}text{% verbatim %} }{% endverbatim %}
> – correctly renders as – { text }
>
> However, this fails:
>
> {% verbatim %}{{% endverbatim %}text{% verbatim %}}{% endverbatim %} –
> incorrectly renders as – {{% endverbatim %}text{% verbatim %}}
> {% verbatim %}{{{% endverbatim %}text{% verbatim %}}}{% endverbatim %}
> – incorrectly renders as – {{{% endverbatim %}text{% verbatim %}}}
> {% verbatim %}{{ {% endverbatim %}text{% verbatim %} }}{% endverbatim %}
> – incorrectly renders as – {{ {% endverbatim %}text{% verbatim %} }}
New description:
The {% verbatim %} tag fails to render curly braces under certain
conditions - notably when there are two verbatim tags on the same line.
This works:
{% verbatim %}{{{% endverbatim %} – correctly renders as – {{
{% verbatim %}}}{% endverbatim %} – correctly renders as – }}
{% verbatim %}{ {% endverbatim %}text{% verbatim %} }{% endverbatim %} –
correctly renders as – { text }
However, this fails:
{% verbatim %}{{% endverbatim %}text{% verbatim %}}{% endverbatim %} –
incorrectly renders as – {{% endverbatim %}text{% verbatim %}}
{% verbatim %}{{{% endverbatim %}text{% verbatim %}}}{% endverbatim %} –
incorrectly renders as – {{{% endverbatim %}text{% verbatim %}}}
{% verbatim %}{{ {% endverbatim %}text{% verbatim %} }}{% endverbatim %}
– incorrectly renders as – {{ {% endverbatim %}text{% verbatim %} }}
(Note that in the second example above the bug reporter is messing with
the example as it interprets three curly braces as a code block).
--
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:2>
Django
Reporter: stephband | Owner: nobody
Type: Bug | Status: new
Component: Template system | Version: 1.7-rc-2
Keywords: verbatim template | Triage Stage:
tag | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Description changed by timgraham:
Old description:
> The {% verbatim %} tag fails to render curly braces under certain
> conditions - notably when there are two verbatim tags on the same line.
>
> This works:
>
> {% verbatim %}{{{% endverbatim %} – correctly renders as – {{
> {% verbatim %}}}{% endverbatim %} – correctly renders as – }}
> {% verbatim %}{ {% endverbatim %}text{% verbatim %} }{% endverbatim %}
> – correctly renders as – { text }
>
> However, this fails:
>
> {% verbatim %}{{% endverbatim %}text{% verbatim %}}{% endverbatim %} –
> incorrectly renders as – {{% endverbatim %}text{% verbatim %}}
> {% verbatim %}{{{% endverbatim %}text{% verbatim %}}}{% endverbatim %}
> – incorrectly renders as – {{{% endverbatim %}text{% verbatim %}}}
> {% verbatim %}{{ {% endverbatim %}text{% verbatim %} }}{% endverbatim %}
> – incorrectly renders as – {{ {% endverbatim %}text{% verbatim %} }}
>
> (Note that in the second example above the bug reporter is messing with
> the example as it interprets three curly braces as a code block).
New description:
The {% verbatim %} tag fails to render curly braces under certain
conditions - notably when there are two verbatim tags on the same line.
This works:
{{{
{% verbatim %}{{{% endverbatim %} – correctly renders as – {{
{% verbatim %}}}{% endverbatim %} – correctly renders as – }}
{% verbatim %}{ {% endverbatim %}text{% verbatim %} }{% endverbatim %} –
correctly renders as – { text }
}}}
However, this fails:
{{{
{% verbatim %}{{% endverbatim %}text{% verbatim %}}{% endverbatim %} –
incorrectly renders as – {{% endverbatim %}text{% verbatim %}}
{% verbatim %}{{{% endverbatim %}text{% verbatim %}}}{% endverbatim %} –
incorrectly renders as – {{{% endverbatim %}text{% verbatim %}}}
{% verbatim %}{{ {% endverbatim %}text{% verbatim %} }}{% endverbatim %}
– incorrectly renders as – {{ {% endverbatim %}text{% verbatim %} }}
}}}
--
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:3>
Django
Reporter: stephband | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: verbatim template | Triage Stage: Accepted
tag | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by timgraham):
* version: 1.7-rc-2 => master
* stage: Unreviewed => Accepted
Comment:
Not sure if it will be easy/possible to fix. If not, we can document the
limitation.
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:4>
Django
Reporter: stephband | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: verbatim template | Triage Stage: Accepted
tag | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by mountainpaul):
This occurs because the {{ and }} are the starting and ending delimiters
for variables. This also affects block tag delimters, {% and %}, and
comment tag delimiters, {# and #}. In all cases they must be matching.
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:5>
Django
Reporter: stephband | Owner:
Type: Bug | mountainpaul
Component: Template system | Status: assigned
Severity: Normal | Version: master
Keywords: verbatim template | Resolution:
tag | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by mountainpaul):
* owner: nobody => mountainpaul
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:6>
Django
Reporter: stephband | Owner:
Type: Bug | mountainpaul
Component: Template system | Status: assigned
Severity: Normal | Version: master
Keywords: verbatim template | Resolution:
tag | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by shayneoneill):
Possibly a better example why this is a really huge headache is this
{{{
{%verbatim%} {{ {%endverbatim%} computer.{{field}} {%verbatim%} }}
{%endverbatim%}
}}}
From some auditing software where we are using django views to render
lists of fields , which however are being fetched via AngularJS from a
third party REST source. We simply can't find a simple way around this. :(
An alternative might even be some simple tags that provide the missing
character Ie {% special left_double_brace %} or whatever.
Whilst documentation would at least warn the bug is there, this is a show-
stopper error for a lot of use cases.
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:7>
Django
Reporter: stephband | Owner:
Type: Bug | mountainpaul
Component: Template system | Status: assigned
Severity: Normal | Version: master
Keywords: verbatim template | Resolution:
tag | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by aaugustin):
Shayne, aren't you looking for
[https://docs.djangoproject.com/en/dev/ref/templates/builtins/#templatetag
the templatetag tag]?
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:8>
Django
Reporter: stephband | Owner:
Type: Bug | mountainpaul
Component: Template system | Status: assigned
Severity: Normal | Version: master
Keywords: verbatim template | Resolution:
tag | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by shayneoneill):
Oh wow. I didn't even realise that existed. Thanks! Still I do think that
allowing multiple verbatim/endverbatim instances on a line would solve a
lot of headaches.
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:9>
Django
Reporter: stephband | Owner:
Type: Bug | mountainpaul
Component: Template system | Status: assigned
Severity: Normal | Version: master
Keywords: verbatim template | Resolution:
tag | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by aaugustin):
This may require switching to a proper lexer that understands literals
instead of the current regex-based implementation.
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:10>
Django
Reporter: stephband | Owner: Paul
| Egges
Type: Bug | Status: assigned
Component: Template system | Version: master
Severity: Normal | Resolution:
Keywords: verbatim template | Triage Stage: Accepted
tag |
Easy pickings: 0 | UI/UX: 0
Comment (by Alexey):
Probably related:
{{{
{% verbatim %}{%{% endverbatim %}
}}}
With curly bracket and percentage sign (open block tag) template throws
exception of unclosed verbatim tag. Django 1.11.22
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:11>
Django
Reporter: stephband | Owner: (none)
Type: Bug | Status: new
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: verbatim template | Triage Stage: Accepted
tag |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Chris Jerdonek):
* cc: Chris Jerdonek (added)
* owner: Paul Egges => (none)
* status: assigned => new
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:12>
Django
Reporter: stephband | Owner: Atul
| Bhouraskar
Type: Bug | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: verbatim template | Triage Stage: Accepted
tag |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Atul Bhouraskar):
* owner: (none) => Atul Bhouraskar
* status: new => assigned
* has_patch: 0 => 1
Comment:
I've created a pull request that allows all the examples in this ticket to
be correctly rendered - https://github.com/django/django/pull/14786
Test cases for the examples have been added and pass. More test cases (for
examples not yet reported) may need to be added.
Note that this pull request is based off my proposed fix for #23356 where
all verbatim tag logic is isolated into a `Parser.parse_verbatim()` method
and removes all special case logic currently in the
`Lexer.create_token()`.
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:13>
Django
Reporter: stephband | Owner: Atul
| Bhouraskar
Type: Bug | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: verbatim template | Triage Stage: Accepted
tag |
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* needs_better_patch: 0 => 1
Comment:
See [https://code.djangoproject.com/ticket/23356#comment:15 comment].
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:14>
Django
Reporter: stephband | Owner: Atul
| Bhouraskar
Type: Bug | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: verbatim template | Triage Stage: Accepted
tag |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by Chris Jerdonek):
The way I would try solving this today is, after
[https://github.com/django/django/pull/14739 PR #14739] is merged, make
[https://github.com/django/django/blob/196a99da5d9c4c33a78259a58d38fb114a4d2ee8/django/template/base.py#L364
Lexer.create_token()] raise a custom exception called something like
`VerbatimTagStarting` with the end string as the argument if a verbatim
tag is encountered. (The line where this happens will be clear after
[https://github.com/django/django/pull/14739 PR #14739].) Then, in
[https://github.com/django/django/blob/196a99da5d9c4c33a78259a58d38fb114a4d2ee8/django/template/base.py#L359
Lexer.tokenize()], handle `VerbatimTagStarting` being raised by searching
for the new end string, and then resuming the `tag_re.split()` iteration
at the new location in the template string afterwards. This will be much
easier to implement now that ticket #33002 is resolved. One side benefit
of this is that `Lexer.create_token()` should become simpler (though the
complexity will move to `tokenize()`). However, `Lexer.create_token()`
probably shouldn't be having much logic anyways.
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:15>
Django
Reporter: stephband | Owner: Atul
| Bhouraskar
Type: Bug | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: verbatim template | Triage Stage: Accepted
tag |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by Chris Jerdonek):
Here are a few comments to add to my previous comment, after thinking a
little more about this:
I believe that finding the end tag for a custom `verbatim` tag can be done
efficiently and without having to dynamically compile any new regular
expressions. (The regex pattern I have in mind here would be one that ends
with `endverbatim`.)
I also think this could be done somewhat more efficiently if Django didn't
have to support pathological
[https://docs.djangoproject.com/en/3.2/ref/templates/builtins/#verbatim
custom closing verbatim tags] like this:
{{{
{% verbatim {% %}
Avoid template rendering via the {% verbatim %}{% endverbatim %} block.
{% endverbatim {% %}
}}}
(In the above, `{%` is being used as the custom `myblock` string below.)
{{{
{% verbatim myblock %}
Avoid template rendering via the {% verbatim %}{% endverbatim %} block.
{% endverbatim myblock %}
}}}
If closing tags like this didn't have to be supported, the approach I have
in mind could I think be done with fewer operations since the regex
pattern could end with `%}` instead of `endverbatim`. (The Python `regex`
module only supports searching for non-overlapping patterns, and whether
closing tags like this are allowed affects whether the regex pattern used
to search for a closing verbatim tag will be overlapping or not.) But as I
said, I believe it can still be done efficiently even if they are allowed.
By the way, would I be allowed to work on this? Atul, I know you assigned
yourself, but I'm not sure if that was only on the assumption that this
ticket would be resolved using the PR you posted for #23356.
PS - one other thing I noticed is that care needs to be taken to ensure
that the regex will match `{% endverbatim` with any Unicode whitespace
between `{%` and `endverbatim` (so not just ASCII whitespace), but not
`\n`. That means the regex sequence `\s` can't be used as is since it
includes `\n`.
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:16>
Django
Reporter: stephband | Owner: Atul
| Bhouraskar
Type: Bug | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: verbatim template | Triage Stage: Accepted
tag |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by Atul Bhouraskar):
Hi Chris and Mariusz,
Sorry for the delay in responding, was a bit too busy at work.
Let me add in a few general comments about this ticket:
* The `{% verbatim %}` tags are only found in a very small subset of
django templates (I could be very wrong here but I've never actually come
across usage out in the wild yet).
* The conditions in this ticket are only found in a small subset of all
verbatim usages - ie combined with the above this is an ''extremely''
small use case.
* Every example provided here can be implemented with either the
`templatetag tag` as pointed out in
https://code.djangoproject.com/ticket/23424#comment:8 or via a custom tag.
As older comments have said we could note this in the documentation.
* Adding complexity to `create_token()` and/or `tokenise()` - both of
which are in the critical path for ''all'' template tags (ie. every single
template out there is impacted), does not seem (IMHO) a good enough
tradeoff to fix this.
* My strong feeling is that any solution to this ticket could always
either be considered inefficient or have special cases/ limitations which
will need to be spelt out in documentation anyway.
* It would be best to simply acknowledge the current behaviour in the
documentation and provide workarounds there - happy to create a PR.
Now coming to my proposed fix (if we are to proceed with the fixing
anyway):
* The multiple re-tokenisation ''only'' happens if the conditions for this
ticket are met (multiple verbatim tags on the same line containing matched
var/tag braces within).
* As I've said it is only a proof of concept fix and of course the regex
etc definitely can improve.
* The inefficiency will only be encountered if the template being parsed
contains a verbatim tag (as `parse_verbatim()` will never get called
otherwise).
* The inefficiency is restricted to the parsing stage which should only
ever happen once per template as long as the cached loader is used (which
is the default). Users concerned with performance should really not opt
out of this.
To summarise:
* I would strongly support leaving the current behaviour as-is and
updating the documentation as simple workarounds exist.
PS. On the other hand there is no workaround for creating "verbatim like"
tags (#23356) and the proposed fix there has ''zero'' impact on all other
template tags (and actually reduces complexity in `create_token()` with a
possible small boost in performance).
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:17>
Django
Reporter: stephband | Owner: Atul
| Bhouraskar
Type: Bug | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: verbatim template | Triage Stage: Accepted
tag |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by Atul Bhouraskar):
Replying to [comment:16 Chris Jerdonek]:
> By the way, would I be allowed to work on this? Atul, I know you
assigned yourself, but I'm not sure if that was only on the assumption
that this ticket would be resolved using the PR you posted for #23356.
>
Sure thing :)
I can de-assign myself if you want.
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:18>
Django
Reporter: stephband | Owner: Atul
| Bhouraskar
Type: Bug | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: verbatim template | Triage Stage: Accepted
tag |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Changes (by Sarah Boyce):
* cc: Sarah Boyce (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/23424#comment:19>