[Django] #35816: Django Template Language doesn't support all float literals

[Django]

#35816: Django Template Language doesn't support all float literals
----------------------------+-----------------------------------------
Reporter: Lily Foote | Type: Uncategorized
Status: new | Component: Uncategorized
Version: 5.0 | Severity: Normal
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------+-----------------------------------------
When parsing arguments to filters, Django converts numeric literals to
`float` or `int`, but in the case of scientific notation for `float` this
is incomplete:

{{{
>>> from django.template.base import Template
>>> from django.template.engine import Engine
>>> 5.2e3
5200.0
>>> Template("{{ foo|default:5.2e3 }}", engine=e)
<Template template_string="{{ foo|default:5.2e3...">
>>> 5.2e-3
0.0052
>>> Template("{{ foo|default:5.2e-3 }}", engine=e)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/lily/.local/share/lilyenv/virtualenvs/django-rusty-
templates/3.12/lib/python3.12/site-packages/django/template/base.py", line
154, in __init__
self.nodelist = self.compile_nodelist()
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lily/.local/share/lilyenv/virtualenvs/django-rusty-
templates/3.12/lib/python3.12/site-packages/django/template/base.py", line
196, in compile_nodelist
nodelist = parser.parse()
^^^^^^^^^^^^^^
File "/home/lily/.local/share/lilyenv/virtualenvs/django-rusty-
templates/3.12/lib/python3.12/site-packages/django/template/base.py", line
489, in parse
raise self.error(token, e)
File "/home/lily/.local/share/lilyenv/virtualenvs/django-rusty-
templates/3.12/lib/python3.12/site-packages/django/template/base.py", line
487, in parse
filter_expression = self.compile_filter(token.contents)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lily/.local/share/lilyenv/virtualenvs/django-rusty-
templates/3.12/lib/python3.12/site-packages/django/template/base.py", line
605, in compile_filter
return FilterExpression(token, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lily/.local/share/lilyenv/virtualenvs/django-rusty-
templates/3.12/lib/python3.12/site-packages/django/template/base.py", line
706, in __init__
raise TemplateSyntaxError(
django.template.exceptions.TemplateSyntaxError: Could not parse the
remainder: '-3' from 'foo|default:5.2e-3'
}}}

I couldn't actually find the documentation for Django's handling of
numeric literals in templates, so this may be an undocumented
implementation detail. If so, perhaps deprecating scientific notation
would be appropriate. Otherwise, we should support this case.
--
Ticket URL: <https://code.djangoproject.com/ticket/35816>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35816: Django Template Language doesn't support all float literals

-------------------------------+--------------------------------------
Reporter: Lily Foote | Owner: (none)
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------+--------------------------------------
Changes (by Lily Foote):

* version: 5.0 => 5.1

--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:1>

[Django]

#35816: Django Template Language doesn't support all float literals

-------------------------------------+-------------------------------------
Reporter: Lily Foote | Owner:
| aryamanpathak2022
Type: Uncategorized | Status: assigned

Component: Uncategorized | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by aryamanpathak2022):

* owner: (none) => aryamanpathak2022
* status: new => assigned

Comment:

I am beginner in os , Giving this a try
--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:2>

[Django]

#35816: Django Template Language doesn't support all float literals

---------------------------------+------------------------------------
Reporter: Lily Foote | Owner: Aryaman
Type: Uncategorized | Status: assigned
Component: Template system | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

---------------------------------+------------------------------------
Changes (by Sarah Boyce):

* component: Uncategorized => Template system
* stage: Unreviewed => Accepted

Comment:

Thank you! We try to support scientific numbers so we should fix this.
I believe their is an issue with the regex, this might work
{{{#!diff
diff --git a/django/template/base.py b/django/template/base.py
index ee2e145c04..e0d032f688 100644
--- a/django/template/base.py
+++ b/django/template/base.py
@@ -640,7 +640,7 @@ filter_raw_string = r"""
)?
)""" % {
"constant": constant_string,
- "num": r"[-+.]?\d[\d.e]*",
+ "num": r"[-+.]?\d[\d.e-]*",
"var_chars": r"\w\.",
"filter_sep": re.escape(FILTER_SEPARATOR),
}}}

Will need a test
--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:3>

[Django]

#35816: Django Template Language doesn't support all float literals
---------------------------------+------------------------------------
Reporter: Lily Foote | Owner: Aryaman

Type: Bug | Status: assigned

Component: Template system | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------
Changes (by Sarah Boyce):

* type: Uncategorized => Bug

--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:4>

[Django]

#35816: Django Template Language doesn't support all float literals
---------------------------------+------------------------------------
Reporter: Lily Foote | Owner: Aryaman
Type: Bug | Status: assigned
Component: Template system | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Comment (by IronJam):

Yeah it looks like a regex error. I can code a regression test and make a
pr.
--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:5>

[Django]

#35816: Django Template Language doesn't support all float literals
---------------------------------+------------------------------------

Reporter: Lily Foote | Owner: IronJam

Type: Bug | Status: assigned
Component: Template system | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Changes (by IronJam):

* owner: Aryaman => IronJam

--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:6>

[Django]

#35816: Django Template Language doesn't support all float literals
---------------------------------+------------------------------------
Reporter: Lily Foote | Owner: IronJam
Type: Bug | Status: assigned
Component: Template system | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Comment (by IronJam):

PR: - https://github.com/django/django/pull/19146
--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:7>

[Django]

#35816: Django Template Language doesn't support all float literals
---------------------------------+------------------------------------
Reporter: Lily Foote | Owner: IronJam

Type: Bug | Status: closed

Component: Template system | Version: 5.1

Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Changes (by IronJam):

* resolution: => fixed
* status: assigned => closed

--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:8>

[Django]

#35816: Django Template Language doesn't support all float literals
---------------------------------+------------------------------------
Reporter: Lily Foote | Owner: IronJam

Type: Bug | Status: new

Component: Template system | Version: 5.1
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------
Changes (by IronJam):

* resolution: fixed =>
* status: closed => new

--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:9>

[Django]

#35816: Django Template Language doesn't support all float literals
---------------------------------+------------------------------------
Reporter: Lily Foote | Owner: IronJam
Type: Bug | Status: new
Component: Template system | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Changes (by Antoliny):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:10>

[Django]

#35816: Django Template Language doesn't support all float literals
---------------------------------+------------------------------------
Reporter: Lily Foote | Owner: IronJam
Type: Bug | Status: new
Component: Template system | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Changes (by Sarah Boyce):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:11>

[Django]

#35816: Django Template Language doesn't support all float literals
---------------------------------+------------------------------------
Reporter: Lily Foote | Owner: IronJam
Type: Bug | Status: new
Component: Template system | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------

Changes (by haileyajohnson):

* needs_better_patch: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:12>

[Django]

#35816: Django Template Language doesn't support all float literals

-------------------------------------+-------------------------------------
Reporter: Lily Foote | Owner:

| haileyajohnson
Type: Bug | Status: assigned

Component: Template system | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by haileyajohnson):

* owner: IronJam => haileyajohnson

* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:13>

[Django]

#35816: Django Template Language doesn't support all float literals
-------------------------------------+-------------------------------------
Reporter: Lily Foote | Owner:
| haileyajohnson
Type: Bug | Status: assigned
Component: Template system | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by haileyajohnson):

https://github.com/django/django/pull/19213
--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:14>

[Django]

#35816: Django Template Language doesn't support all float literals
-------------------------------------+-------------------------------------

Reporter: Lily Foote | Owner: Hailey
| Johnson

Type: Bug | Status: assigned
Component: Template system | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* needs_better_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:15>

[Django]

#35816: Django Template Language doesn't support all float literals
-------------------------------------+-------------------------------------
Reporter: Lily Foote | Owner: Hailey
| Johnson
Type: Bug | Status: assigned
Component: Template system | Version: 5.1
Severity: Normal | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Sarah Boyce):

* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:16>

[Django]

#35816: Django Template Language doesn't support all float literals
-------------------------------------+-------------------------------------
Reporter: Lily Foote | Owner: Hailey
| Johnson

Type: Bug | Status: closed

Component: Template system | Version: 5.1

Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by GitHub <noreply@…>):

* resolution: => fixed
* status: assigned => closed

Comment:

In [changeset:"5183f7c287a9a5d61ca1103b55166cda52d9c647" 5183f7c2]:
{{{#!CommitTicketReference repository=""
revision="5183f7c287a9a5d61ca1103b55166cda52d9c647"
Fixed #35816 -- Handled parsing of scientific notation in DTL. (#19213)

* Refs #35816 -- Improved test coverage of FilterExpression.

* Fixed #35816 -- Made FilterExpression parse scientific numbers.

---------

Co-authored-by: Sarah Boyce <42296566+...@users.noreply.github.com>
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:17>

[Django]

#35816: Django Template Language doesn't support all float literals
-------------------------------------+-------------------------------------
Reporter: Lily Foote | Owner: Hailey
| Johnson
Type: Bug | Status: closed
Component: Template system | Version: 5.1
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sarah Boyce):

Note that despite selecting "rebase and merge" on the PR, this appears to
be squashed merged. I have now disabled the squash merge on the project.
--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:18>

[Django]

#35816: Django Template Language doesn't support all float literals
-------------------------------------+-------------------------------------
Reporter: Lily Foote | Owner: Hailey
| Johnson
Type: Bug | Status: closed
Component: Template system | Version: 5.1
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jon Banafato):

I have a question about template behavior that was changed by this feature
(please let me know if I should post this as a new ticket instead of
continuing conversation here). While [https://github.com/jazzband/django-
hosts/issues/176#issuecomment-2961158554 investigating CI test failures
for django-hosts], I observed that Django no longer throws a
TemplateSyntaxError for template variable names containing dashes and
instead appears to treat them as regular template variable names. I did
not see any notes in the
[https://docs.djangoproject.com/en/dev/ref/templates/language/#variables
template language documentation] or the templates section of the
[https://docs.djangoproject.com/en/dev/releases/6.0/#templates release
notes] for 6.0. I believe this change was introduced by the commit
associated with this issue, and I've included an example of the behavioral
change below.

On commit 8df5ce80d26824ce72af41edc03275d435de9432, the following code
fails with a TemplateSyntaxError as expected.

{{{
>>> from django.template import Template, Context
>>> template = Template("Hello, {{ first-name }}")

Traceback (most recent call last):

...

django.template.exceptions.TemplateSyntaxError: Could not parse the

remainder: '-name' from 'first-name'
}}}

On commit 5183f7c287a9a5d61ca1103b55166cda5, the following code parses the
template and renders the `first-name` provided.

{{{
>>> from django.template import Template, Context
>>> template = Template("Hello, {{ first-name }}")
>>> template.render(Context({"first-name": "Jon"}))
'Hello, Jon'
}}}

Does this warrant a new ticket for a behavior and / or documentation
change? I'd be happy to help where possible, but I'm not very familiar
with this change or the Django development process in general, so I am
looking for guidance. Thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:19>

[Django]

#35816: Django Template Language doesn't support all float literals
-------------------------------------+-------------------------------------
Reporter: Lily Foote | Owner: Hailey
| Johnson
Type: Bug | Status: closed
Component: Template system | Version: 5.1
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Hailey Johnson):

Hi, you're right that this ticket did change that behavior!

I'm happy to add documentation if we decide to keep it as is, I would
argue that if we want to revert the behavior, it shouldn't be handled
where it was before because we don't differentiate from variable names and
numeric values at that point.
--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:20>

[Django]

#35816: Django Template Language doesn't support all float literals
-------------------------------------+-------------------------------------
Reporter: Lily Foote | Owner: Hailey
| Johnson
Type: Bug | Status: closed
Component: Template system | Version: 5.1
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jon Banafato):

Thanks for the quick response! The template variable name behavior seems
like a rather impactful change compared to the state scope of supporting
float literals. I wanted to comment here mostly so that the change didn't
go unintended and unnoticed.
--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:21>

[Django]

#35816: Django Template Language doesn't support all float literals
-------------------------------------+-------------------------------------
Reporter: Lily Foote | Owner: Hailey
| Johnson
Type: Bug | Status: closed
Component: Template system | Version: 5.1
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Hailey Johnson):

That makes sense! I'd be glad to follow-up either way. I think the '-'
character was neither explicitly included or excluded in valid variable
naming (unlike, e.g. whitespace characters are explicitly prohibiting and
underscores are explicitly allowed).
--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:22>

[Django]

#35816: Django Template Language doesn't support all float literals
-------------------------------------+-------------------------------------
Reporter: Lily Foote | Owner: Hailey
| Johnson
Type: Bug | Status: closed
Component: Template system | Version: 5.1
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Baptiste Mispelon):

Thanks for flagging this Jon! It's always great when bugs get noticed
**ahead** of a release 😅

This looks like a bug to me, so I've opened #36465 (which I marked as a
release blocker since it's a breaking change in behavior).
--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:23>

[Django]

#35816: Django Template Language doesn't support all float literals
-------------------------------------+-------------------------------------
Reporter: Lily Foote | Owner: Hailey
| Johnson
Type: Bug | Status: closed
Component: Template system | Version: 5.1
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"22506b2c162b34c4c45f748cf11ede75824a40cd" 22506b2]:
{{{#!CommitTicketReference repository=""
revision="22506b2c162b34c4c45f748cf11ede75824a40cd"
Fixed #36465, Refs #35816 -- Disallowed '+' and '-' characters in template
variable names.

Regression in 5183f7c287a9a5d61ca1103b55166cda52d9c647.

Thank you to Jon Banafato and Baptiste Mispelon for the report.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35816#comment:24>