Django template 'if ... is' feature design

[Stephen Kelly]

Hello,

I reviewed

https://github.com/django/django/commit/c00ae7f5

while updating the features of Grantlee, and I have the following notes:

* The unit tests added both have the same name ("template"). They should
have different names of the form 'if-tag-isNN'
* The feature does not work the same way as django 'is' works. For example:

In: t = e.from_string(
"{% if \"True\" is \"True\" %}yes{% else %}no{% endif %}")
In: t.render(c)

Out: 'no'

The feature 'if ... is' feature looks like it needs more design
consideration before it is released.

Thanks,

Steve.

[Ryan Hiebert]

The `is` operator in Python checks for identical objects. A string is not guaranteed to be exactly the same object as another string (in this example "True" is not the same object (bytes in memory) as the second "True", so Python rightly sees that they are not the same object.

True, False, and None are singletons, that are guaranteed to always point to the exact same object in memory, and `is` is the appropriate operator to use there. It is for those singletons that this feature is being added.

> --
> You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ne6k90%249b3%241%40ger.gmane.org.
> For more options, visit https://groups.google.com/d/optout.

[Stephen Kelly]

Ryan Hiebert wrote:

> The `is` operator in Python checks for identical objects. A string is not
> guaranteed to be exactly the same object as another string (in this
> example "True" is not the same object (bytes in memory) as the second
> "True", so Python rightly sees that they are not the same object.

I'm sorry, I don't understand what you're saying here.

In [81]: "True" is "True"
Out[81]: True

In [82]: a = "anything"

In [83]: b = "anything"

In [84]: a is b
Out[84]: True

In [85]: a = "any" + "thing"

In [86]: a is b
Out[86]: True

In [87]: a = "any"

In [88]: a += "thing"

In [89]: a is b
Out[89]: False

> True, False, and None are singletons, that are guaranteed to always point
> to the exact same object in memory, and `is` is the appropriate operator
> to use there. It is for those singletons that this feature is being added.

Ok. I'm not usually a python programmer, so I don't know what kind of
expectations people have of 'if ... is' in templates. Given the behavior of
python, I find the behavior surprising.

Perhaps it should be documented. The documentation doesn't currently say
that 'if ... is' only works for True, False and None. If that is the case,
then it probably should be documented.

(and the unit tests should be renamed :))

Thanks,

Steve.

[Daniel Chimeno]

The is comparator is quite error prone for misunderstanding it (as we could observed some times)

maybe it worths to link python docs about it:

https://docs.python.org/3.5/reference/expressions.html#is

Well, that Python docs doesn't say to much (lack of examples)

>>> a = 'pub'

>>> b = ''.join(['p','u','b'])

>>> a == b

True

>>> a is b

False

a == b it's the same as 

id(a) == id(b)

I think we should give an emphasis on  this because are going to cause problems to some people.

Hope it helps. 

[Stephen Kelly]

Daniel Chimeno wrote:

> I think we should give an emphasis on this because are going to cause
> problems to some people.

Yes, this is why I suggested in my original mail that the feature needs more
design consideration.

* What should work / what should not work? Is the documentation clear?
* Why is there no 'is not' operator? ie '{% if a is not True %}'

Thanks,

Steve.

[Ryan Hiebert]

> On Apr 7, 2016, at 5:13 PM, Stephen Kelly <stev...@gmail.com> wrote:
>
> Daniel Chimeno wrote:
>
>> I think we should give an emphasis on this because are going to cause
>> problems to some people.
>
> Yes, this is why I suggested in my original mail that the feature needs more
> design consideration.
>
> * What should work / what should not work? Is the documentation clear?

I don't know about the docs, but it _does_ work exactly like the Python version. A while back I had this same confusion regarding the `is` operator. You can see all about it in my Stack Overflow question: http://stackoverflow.com/q/8638880/206349

> * Why is there no 'is not' operator? ie '{% if a is not True %}'

`is not` is probably logical addition, if somebody wants to put in the time to make it happen.

[Stephen Kelly]

Stephen Kelly wrote:

> * The unit tests added both have the same name ("template"). They should
> have different names of the form 'if-tag-isNN'

FYI, I made a pull request which fixes this and some other minor issues in
the if-tag tests:

https://github.com/django/django/pull/6430

If it's not trivial to integrate in master, I can file a trac ticket soon to
track it.

Thanks,

Steve.

[Stephen Kelly]

Ryan Hiebert wrote:

>> * What should work / what should not work? Is the documentation clear?
>
> I don't know about the docs, but it _does_ work exactly like the Python
> version.

I suppose you are not responding to what you quoted.

Anyway, that makes at least two people telling me that

In [81]: "True" is "True"
Out[81]: True

and

In: t = e.from_string(
"{% if \"True\" is \"True\" %}yes{% else %}no{% endif %}")
In: t.render(c)

Out: 'no'

is 'the same behavior' and seems to be what you expect.

> A while back I had this same confusion regarding the `is`
> operator. You can see all about it in my Stack Overflow question:
> http://stackoverflow.com/q/8638880/206349

Your question doesn't relate to the django template system, so I'm afraid I
don't understand how it is 'the same confusion' or relates to what this
thread is about.

Thanks!

Steve.

[Carl Meyer]

Hi Steve,

On 04/07/2016 04:36 PM, Stephen Kelly wrote:
> Anyway, that makes at least two people telling me that
>
> In [81]: "True" is "True"
> Out[81]: True
>
> and
>
> In: t = e.from_string(
> "{% if \"True\" is \"True\" %}yes{% else %}no{% endif %}")
> In: t.render(c)
>
> Out: 'no'
>
> is 'the same behavior' and seems to be what you expect.

The point is that you shouldn't ever expect any consistent behavior when
using `is` with strings (whether in Python code or in Django templates),
because that behavior is not defined by the Python language, it's
implementation dependent. It happens that the CPython implementation
interns short string literals, so they are the same object in memory,
but this doesn't happen for longer strings, or strings that aren't
generated from literals. Those "string literals" in the Django template
are not string literals from Python's perspective, so they are not
interned, so they are not the same object.

There's no reasonable way to "fix" this, because there is no reliable
Python behavior to try to match. Any fix would just be encouraging use
of `is` with strings; the right answer is "never use `is` with strings"
(or with integers, for exactly the same reasons).

In other words, the value of `"string" is "string"` in Python is
effectively accidental and almost certainly useless (it does correctly
and reliably tell you whether the two string objects are the same
object, but for immutable objects this information is
performance-optimization-dependent and useless), so there is no
"correct" behavior here for the DTL to match.

Carl

[Stephen Kelly]

Carl Meyer wrote:

> The point is that you shouldn't ever expect any consistent behavior when
> using `is` with strings (whether in Python code or in Django templates),
> because that behavior is not defined by the Python language, it's
> implementation dependent. It happens that the CPython implementation
> interns short string literals, so they are the same object in memory,
> but this doesn't happen for longer strings, or strings that aren't
> generated from literals. Those "string literals" in the Django template
> are not string literals from Python's perspective, so they are not
> interned, so they are not the same object.
>
> There's no reasonable way to "fix" this, because there is no reliable
> Python behavior to try to match. Any fix would just be encouraging use
> of `is` with strings; the right answer is "never use `is` with strings"
> (or with integers, for exactly the same reasons).
>
> In other words, the value of `"string" is "string"` in Python is
> effectively accidental and almost certainly useless (it does correctly
> and reliably tell you whether the two string objects are the same
> object, but for immutable objects this information is
> performance-optimization-dependent and useless), so there is no
> "correct" behavior here for the DTL to match.

I see, and I understand now. Thanks for the explanation and sorry for being
slow on the uptake on this one!

Am I right to think that using 'is' with numbers is expected to work like
this in the DTL? :

In [120]: c["a"] = 1

In [121]: t = e.from_string("{% if 1 is 1 %}yes{% else %}no{% endif %}")

In [122]: print t.render(c)
yes

In [123]: t = e.from_string("{% if 1 is a %}yes{% else %}no{% endif %}")

In [124]: print t.render(c)
yes

In [125]: c["a"] = 1.0

In [126]: t = e.from_string("{% if 1 is a %}yes{% else %}no{% endif %}")

In [127]: print t.render(c)
no

In [128]: t = e.from_string("{% if 1 is 1.0 %}yes{% else %}no{% endif
%}")

In [129]: print t.render(c)
no

In [130]: t = e.from_string("{% if 1.0 is 1.0 %}yes{% else %}no{% endif
%}")

In [131]: print t.render(c)
no

Would it be useful to add those as unit tests and document what can be
expected of 'if ... is'?

Thanks,

Steve.

[Carl Meyer]

Hi Steve,

On 04/07/2016 05:09 PM, Stephen Kelly wrote:
> Am I right to think that using 'is' with numbers is expected to work like
> this in the DTL? :

The situation with integers is roughly the same as with strings, except
that I believe CPython interns _all_ small integers, not just those that
are literals in the code. This happens to make DTL appear a bit more
"consistent" with CPython for integers, but the correct advice is still
the same: don't ever use `is` with integers, whether in Python code or DTL.

The `is` operator tells you whether two variables refer to the same
object. This is sometimes a meaningful question to ask for instances of
classes with mutable state, or for singletons like None, True, and
False, but it's not a useful or meaningful question to ask about
immutable objects like strings and integers, because there is no
perceptible difference (other than from `is`, `id`, and perhaps memory
usage) resulting from whether the implementation happened to choose to
reuse the same object or not.

> In [120]: c["a"] = 1
>
> In [121]: t = e.from_string("{% if 1 is 1 %}yes{% else %}no{% endif %}")
>
> In [122]: print t.render(c)
> yes
>
> In [123]: t = e.from_string("{% if 1 is a %}yes{% else %}no{% endif %}")
>
> In [124]: print t.render(c)
> yes
>
> In [125]: c["a"] = 1.0
>
> In [126]: t = e.from_string("{% if 1 is a %}yes{% else %}no{% endif %}")
>
> In [127]: print t.render(c)
> no
>
> In [128]: t = e.from_string("{% if 1 is 1.0 %}yes{% else %}no{% endif
> %}")
>
> In [129]: print t.render(c)
> no
>
> In [130]: t = e.from_string("{% if 1.0 is 1.0 %}yes{% else %}no{% endif
> %}")
>
> In [131]: print t.render(c)
> no
>
> Would it be useful to add those as unit tests and document what can be
> expected of 'if ... is'?

No to tests, because we would be adding tests for undefined and
unreliable behavior.

It might be worth adding a short documentation note. We largely want to
avoid documenting Python's behavior in the Django docs, but a short note
in the template `is` docs reminding people not to ever use `is` with
strings or numbers might be worthwhile.

Carl

[Stephen Kelly]

Carl Meyer wrote:

> No to tests, because we would be adding tests for undefined and
> unreliable behavior.

Yes, makes sense. Thanks again for the explanation.

> It might be worth adding a short documentation note. We largely want to
> avoid documenting Python's behavior in the Django docs, but a short note
> in the template `is` docs reminding people not to ever use `is` with
> strings or numbers might be worthwhile.

Or to only use it with None, True or False.

However, as you say it doesn't make sense to document python in the django
docs. If it's only me who would be confused about this, there may be no need
to change anything in the docs either.

Thanks,

Steve.

[Alasdair Nicol]

I've created a ticket #26479 to add `is not`.

[Tobias Kunze]

On 08/04/16 01:40, Stephen Kelly wrote:

> Carl Meyer wrote:
>
>> It might be worth adding a short documentation note. We largely want to
>> avoid documenting Python's behavior in the Django docs, but a short note
>> in the template `is` docs reminding people not to ever use `is` with
>> strings or numbers might be worthwhile.
>
> Or to only use it with None, True or False.

I'd be in favor of a documentation note like that, or a link to Python
documentation of `is`. I agree about not duplicating Python
documentation, but a short note like that could help many people who
aren't that familiar with Python in general.

Tobias

[Sven R. Kunze]

Good evening everybody. That's my first post here, so let's how this works. :)

This particular discussion caught my sight as it might introduce something really "low-level" functionality into the template engine.

I can contribute here to the "design consideration" that from my experience with other (less experienced) developers it that the difference of "is" and "==" is not quite easy to explain (as illustrated by this thread as well). So, I am not 100% convinced that "is" and "is not" is a valid use-case for a template language but I may miss some important piece here.

However, given the docs and tests added in the patch, it seems the intended use-case for this kind of feature is to check for "is None", "is not None", "is True", "is not True" etc. Would it makes sense to hard-wire those specific use-cases instead?

(Admittedly, even those can be implemented using "==".)

[Stephen Kelly]

Sven R. Kunze wrote:

> Good evening everybody. That's my first post here, so let's how this
> works. :)
>
> This particular discussion caught my sight as it might introduce something
> really "low-level" functionality into the template engine.

As far as I understand from

https://github.com/django/django/pull/6442

it it deliberate that the if tag exposes the entire implementation detail.

That is, if the current if <operator> tests didn't exist today and they were
submitted, they would be rejected as unnecessary because the existing tests
are really just testing the python if operator behavior. That's what I
understand, but I might have misinterpreted something.

Thanks,

Steve.

[Tim Graham]

Hi Sven,

Here are some examples of when the "is" operator isn't equivalent to "==".
http://stackoverflow.com/questions/3647692/when-is-the-operator-not-equivalent-to-the-is-operator-python

Preston (author of the patch to add the "is" operator [0]) indicated support for "is" was necessary for template-based widget rendering [1]. I haven't studied that patch closely, but maybe if you take a look you'll understand why it's needed.

I don't see much benefit to your proposal of limiting its usage to comparing boolean/None values.

[0] https://code.djangoproject.com/ticket/26118
[1] https://github.com/django/django/pull/4848

[Sven R. Kunze]

Am Donnerstag, 14. April 2016 21:04:04 UTC+2 schrieb Stephen:

As far as I understand from

 https://github.com/django/django/pull/6442

it it deliberate that the if tag exposes the entire implementation detail.

Thanks for clarifying.

I didn't know that exactly this was a deliberate choice. I always got the impression that Django tries to boil down the feature set of its template engine for very good reasons (which I agree with).

Sven

 

[Sven R. Kunze]

Hi Tim,

due to Stephen's response, my concerns are not valid. :)

Still, my comments among your lines:

Am Freitag, 15. April 2016 02:30:46 UTC+2 schrieb Tim Graham:

Here are some examples of when the "is" operator isn't equivalent to "==".
http://stackoverflow.com/questions/3647692/when-is-the-operator-not-equivalent-to-the-is-operator-python

It's true that "==" is not the same as "is". I just wanted to say that most sane cases are probably YAGNI. Especially in the light of the increased complexity for template users.

 

Preston (author of the patch to add the "is" operator [0]) indicated support for "is" was necessary for template-based widget rendering [1]. I haven't studied that patch closely, but maybe if you take a look you'll understand why it's needed.

Mmm, I couldn't see a good use-case, though. :(

 

I don't see much benefit to your proposal of limiting its usage to comparing boolean/None values.

Neither do I, given that "== True", "== False" and  "== None" will do the trick.

Sven