Re: [Django] #35735: For python 3.9+ class property may not be accessible by Django's template system

[Django]

#35735: For python 3.9+ class property may not be accessible by Django's template
system
---------------------------------+----------------------------------------
Reporter: Fabian Braun | Owner: Fabian Braun
Type: Bug | Status: assigned
Component: Template system | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+----------------------------------------
Changes (by Fabian Braun):

* has_patch: 0 => 1


Old description:

> Before python 3.9 class properties were always available through the
> template system. If you had a class
> {{{
> class MyClass:
> in_template = True
> }}}
> you could access the class property in the template through (if it was
> returned by a callable) `{{ get_my_class.in_template }}`.
>
> The template system first checks if the class is subscriptable (i.e.
> tries `context_value["in_template"]`), will fail and then will get the
> in_template property.
>
> As of python 3.9 some classes actually are subscriptable and trying to
> get the item will not fail: Typing shortcuts introduced syntax like
> `list[int]`. This hides class properties from the template system.
>
> Here's a test (that might go into
> tests/template_tests/syntax_tests/tests_basic.py) which passes on Python
> 3.9 and fails on Python 3.10+:
> {{{
> @setup({"basic-syntax19b": "{{ klass.in_template }}"})
> def test_access_class_property(self):
> class MyClass(list):
> in_template = True
>
> def get_my_class():
> return MyClass
>
> # Pass the callable to return the class, or it would be resolved
> by the template
> # engine
> output = self.engine.render_to_string(
> "basic-syntax19b",
> {"klass": get_my_class}
> )
> self.assertEqual(output, "True")
> }}}
>
> I'd be happy to propose a fix that will not call a classes'
> `__class_getitem__` method.
>
> Thanks to [https://github.com/benzkji Ben Stähli] and [https://github.com
> /last-partizan Serhii Tereshchenko] for figuring out this issue.
>
> References:
>
> * https://github.com/django-cms/django-cms/issues/7948

New description:

Before python 3.9 class properties were always available through the
template system. If you had a class
{{{
class MyClass:
in_template = True
}}}
you could access the class property in the template through (if it was
returned by a callable) `{{ get_my_class.in_template }}`.

The template system first executes the callable `get_my_class` which we
assume returns `MyClass`. Then it checks if the class is subscriptable
(i.e. tries `MyClass["in_template"]`), will fail and then will get the
in_template property.

As of python 3.9 some classes actually are subscriptable and trying to get
the item will not fail: Typing shortcuts introduced syntax like
`list[int]`. These hide class properties from the template system.

Here's a test (that might go into
tests/template_tests/syntax_tests/tests_basic.py) which passes on Python
3.9 and fails on Python 3.10+:
{{{
@setup({"basic-syntax19b": "{{ klass.in_template }}"})
def test_access_class_property(self):
class MyClass(list):
in_template = True

def get_my_class():
return MyClass

# Pass the callable to return the class, or it would be resolved
by the template
# engine
output = self.engine.render_to_string(
"basic-syntax19b",
{"klass": get_my_class}
)
self.assertEqual(output, "True")
}}}

I'd be happy to propose a fix that will not call a classes'
`__class_getitem__` method.

Thanks to [https://github.com/benzkji Ben Stähli] and [https://github.com
/last-partizan Serhii Tereshchenko] for figuring out this issue.

References:

* https://github.com/django-cms/django-cms/issues/7948

--
--
Ticket URL: <https://code.djangoproject.com/ticket/35735#comment:3>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35735: For python 3.9+ class property may not be accessible by Django's template
system
---------------------------------+----------------------------------------
Reporter: Fabian Braun | Owner: Fabian Braun
Type: Bug | Status: assigned
Component: Template system | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+----------------------------------------

Description changed by Fabian Braun:

Old description:

> Before python 3.9 class properties were always available through the
> template system. If you had a class
> {{{
> class MyClass:
> in_template = True
> }}}
> you could access the class property in the template through (if it was
> returned by a callable) `{{ get_my_class.in_template }}`.
>

New description:

Before python 3.9 class properties were always available through the
template system. If you had a class
{{{
class MyClass:
in_template = True

do_not_call_in_templates = True # prevent instantiation

@classmethod
def render_all_objects(cls):
...

}}}
you could access the class property in the template through (if it was

contained in the context) `{{ MyClass.in_template }}` or `{{ MyClass.
render_all_objects }}`.

The template system first gets the class `MyClass` (and does not
instantiate it) or gets it as a result of a callable `get_my_class`. Then

it checks if the class is subscriptable (i.e. tries
`MyClass["in_template"]`), will fail and then will get the in_template
property.

As of python 3.9 some classes actually are subscriptable and trying to get
the item will not fail: Typing shortcuts introduced syntax like

`list[int]`. These hide class properties or methods from the template

system.

Here's a test (that might go into
tests/template_tests/syntax_tests/tests_basic.py) which passes on Python
3.9 and fails on Python 3.10+:
{{{
@setup({"basic-syntax19b": "{{ klass.in_template }}"})
def test_access_class_property(self):
class MyClass(list):
in_template = True

do_not_call_in_templates = True # prevent instantiation

output = self.engine.render_to_string("basic-syntax19b", {"klass":

MyClass})

self.assertEqual(output, "True")
}}}

I'd be happy to propose a fix that will not call a classes'
`__class_getitem__` method.

Thanks to [https://github.com/benzkji Ben Stähli] and [https://github.com
/last-partizan Serhii Tereshchenko] for figuring out this issue.

References:

* https://github.com/django-cms/django-cms/issues/7948

--
--

Ticket URL: <https://code.djangoproject.com/ticket/35735#comment:4>

[Django]

#35735: For python 3.9+ class property may not be accessible by Django's template
system
---------------------------------+----------------------------------------
Reporter: Fabian Braun | Owner: Fabian Braun
Type: Bug | Status: assigned
Component: Template system | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+----------------------------------------

Comment (by Natalia Bidart):

Hi Fabian, thank you for taking the time to create this report!

(Before your last edit) I have tried to reproduce the issue described and
I wasn't able to, I then analyzed your test and noticed that in the test
(but not in the ticket description) `MyClass` is a child of `list`. For
that case, and for children of `dict` or `set`, the test indeed fail; but
for children of `object`, `str`, `int`, the test do not fail.

So on one hand, the issues seems less generic than presented in the title
and description. On the other hand, I'm not sure what you mean with:

> As of python 3.9 some classes actually are subscriptable

My first thought is that your `MyClass` is subscriptable because it's a
child of `list`... so I'm having a hard time understanding how Django is
at fault here. Could you please elaborate?

Also for this sentence:

> I'd be happy to propose a fix that will not call a classes'
`__class_getitem__` method.

I have grepped all the Django source code and nothing other than a few
classes in the ORM implement `__class_getitem__`, so what do you mean
exactly?

{{{
$ grep -nR __class_getitem__
django/db/models/fields/related.py:999: def __class_getitem__(cls,
*args, **kwargs):
django/db/models/manager.py:39: def __class_getitem__(cls, *args,
**kwargs):
django/db/models/query.py:435: def __class_getitem__(cls, *args,
**kwargs):
}}}

I'm closing as `needsinfo` but please reopen when you can provide further
clarifications. Thanks again!
--
Ticket URL: <https://code.djangoproject.com/ticket/35735#comment:5>

[Django]

#35735: For python 3.9+ class property may not be accessible by Django's template
system
---------------------------------+----------------------------------------
Reporter: Fabian Braun | Owner: Fabian Braun

Type: Bug | Status: closed
Component: Template system | Version: dev
Severity: Normal | Resolution: needsinfo

Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+----------------------------------------

Changes (by Natalia Bidart):

* resolution: => needsinfo
* status: assigned => closed
* version: 5.0 => dev

--
Ticket URL: <https://code.djangoproject.com/ticket/35735#comment:6>

[Django]

#35735: For python 3.9+ class property may not be accessible by Django's template
system
---------------------------------+----------------------------------------
Reporter: Fabian Braun | Owner: Fabian Braun
Type: Bug | Status: closed
Component: Template system | Version: dev
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+----------------------------------------

Description changed by Fabian Braun:

Old description:

> Before python 3.9 class properties were always available through the
> template system. If you had a class
> {{{
> class MyClass:
> in_template = True

> I'd be happy to propose a fix that will not call a classes'
> `__class_getitem__` method.
>

> Thanks to [https://github.com/benzkji Ben Stähli] and [https://github.com
> /last-partizan Serhii Tereshchenko] for figuring out this issue.
>
> References:
>
> * https://github.com/django-cms/django-cms/issues/7948

New description:

Before python 3.9 class properties were always available through the
template system. If you had a class
{{{

class MyClass(list):
in_template = True
do_not_call_in_templates = True # prevent instantiation

I'd be happy to propose a fix that will not call a classes'
`__class_getitem__` method.

Thanks to [https://github.com/benzkji Ben Stähli] and [https://github.com
/last-partizan Serhii Tereshchenko] for figuring out this issue.

References:

* https://github.com/django-cms/django-cms/issues/7948

--

--
Ticket URL: <https://code.djangoproject.com/ticket/35735#comment:7>

[Django]

#35735: For python 3.9+ class property may not be accessible by Django's template
system
---------------------------------+----------------------------------------
Reporter: Fabian Braun | Owner: Fabian Braun

Type: Bug | Status: new

Component: Template system | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+----------------------------------------

Changes (by Fabian Braun):

* resolution: needsinfo =>
* status: closed => new

Comment:

Hi Natalia!

Thanks for taking time to look into this.

Sorry, indeed the example needs the `list` parent class. Sorry, I missed
that on the ticket. [https://github.com/django/django/pull/18547/files The
fix] has three tests that work with python 3.8 and not with python 3.9+. I
wanted the test code to be as short as possible and that took me more
iterations than anticipated.

While these tests subclass `list`, **any class that implements some sort
of type hinting using `__class_get_item__` will have its properties or
methods shadowed**. `list` is just a built-in example. Having said this,
you will have to assume that projects subclass Django classes, shadowing
for example a model's manager:
{{{
{% for obj in MyModel.objects.all %}
<li>{{ obj }}</li>
{% endfor %}
}}}
will not work, if the custom `MyModel` for some reason implements
`__class_get_item__` - a thing one might do to annotate, for example, what
model a generic foreign key might refer to, or what sort of data is stored
in a JSON field, or ....

At the time the template system was designed, classes were never
subscriptable. Python has changed, and that's not Django's fault. But I
believe it is time that Django changes with python. The fix I propose
avoids the issue because it does not run `MyClass["property_name"]` in the
first place. This restores the original template variable resolution
design and order of how template references were resolved before python
3.9.

With type hinting getting more and more popular, I expect this issue to
become more and more important. That's what I mean with "class properties
may not be accessible". Sorry, if it felt like I was overstating the
issue.

I guess, already now, the issue might be important: The latest version of
django-modeltranslation [https://github.com/deschler/django-
modeltranslation/blob/0cafcee3f273ac9d01c3321bae2fcc96cc0a83c4/modeltranslation/_typing.py#L26-L40
adds `__class_getitem__` to all admin classes]. This has unforeseen side
effects on all admin classes of all projects using django-
modeltranslation. (And to prevent the discussion if django-
modeltranslation should fix this: (a) they've worked their way around it
and (b) it is not a specific issue for django-modeltranslation but for
**all classes that make their way into Django's template system**.)

I hope I could clarify your questions. Please keep asking if there is need
for more information. Since I am still convinced this should be fixed,
hence I reopen the ticket.
--
Ticket URL: <https://code.djangoproject.com/ticket/35735#comment:8>

[Django]

#35735: For python 3.9+ class property may not be accessible by Django's template
system
---------------------------------+----------------------------------------
Reporter: Fabian Braun | Owner: Fabian Braun
Type: Bug | Status: new
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+----------------------------------------

Comment (by Carlton Gibson):

This seems correct to me. (The test cases aren’t quite as clear as the
explanation here… I’m not sure I’d see quickly — at all 😅 — the
connection to `__class_getitem__` without an explicit example, even if
only for documentation’s sake.)
--
Ticket URL: <https://code.djangoproject.com/ticket/35735#comment:9>

[Django]

#35735: For python 3.9+ class property may not be accessible by Django's template
system
---------------------------------+----------------------------------------
Reporter: Fabian Braun | Owner: Fabian Braun
Type: Bug | Status: new
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+----------------------------------------

Comment (by Fabian Braun):

Python (since 3.9) resolves
[https://docs.python.org/3/reference/datamodel.html#object.__class_getitem__
MyClass["something"]] this way (taken from the docs linked - adapted for a
class):
{{{
def subscribe(cls, x):
"""Return the result of the expression 'cls[x] if cls is a class'"""

metaclass = type(cls)

# If the metaclass of cls defines __getitem__, call
metaclass.__getitem__(cls, x)
if hasattr(metaclass, '__getitem__'): # This is also true for python
pre-3.9
return metaclass.__getitem__(cls, x)

# New in Python 3.9:
# Else, if obj is a class and defines __class_getitem__, call
obj.__class_getitem__(x)
elif hasattr(cls, '__class_getitem__'):
# Instead of TypeError the __class_getitem__ class method returns
a GenericAlias object
return cls.__class_getitem__(x)

# Else, raise an exception - this will let Django's template system
try a property next
else:
raise TypeError(
f"'{cls.__name__}' object is not subscriptable"
)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35735#comment:10>

[Django]

#35735: For python 3.9+ class property may not be accessible by Django's template
system
---------------------------------+----------------------------------------
Reporter: Fabian Braun | Owner: Fabian Braun
Type: Bug | Status: new
Component: Template system | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+----------------------------------------

Changes (by Sarah Boyce):

* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/35735#comment:11>

[Django]

#35735: For python 3.9+ class property may not be accessible by Django's template
system

-------------------------------------+-------------------------------------

Reporter: Fabian Braun | Owner: Fabian
| Braun
Type: Bug | Status: new
Component: Template system | Version: dev
Severity: Normal | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Sarah Boyce):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/35735#comment:12>

[Django]

#35735: For python 3.9+ class property may not be accessible by Django's template
system
-------------------------------------+-------------------------------------
Reporter: Fabian Braun | Owner: Fabian
| Braun

Type: Bug | Status: closed

Component: Template system | Version: dev

Severity: Normal | Resolution: fixed

Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce <42296566+sarahboyce@…>):

* resolution: => fixed
* status: new => closed

Comment:

In [changeset:"d2c97981fb40e0b324b064cd3561de8f5f1887b2" d2c97981]:
{{{#!CommitTicketReference repository=""
revision="d2c97981fb40e0b324b064cd3561de8f5f1887b2"
Fixed #35735 -- Enabled template access to methods and properties of
classes with __class_get_item__.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35735#comment:13>