Request to reconsider ticket #27910: using an Enum class in model Field choices

[Shai Berger]

Hi all,

Lately I've run into ticket 27910[1], which asks for Python Enums to be
usable for generating choices in Django model (and form) fields. This
ticket was closed Wontfix because the original suggestion did not offer
any way to handle translated labels. However, after the ticket was
closed, Tom Forbes brought up a suggestion for a suitable API;
regretfully, AFAICT this suggestion was never discussed here, and (at
least on the ticket) Tom hasn't shared his implementation.

Inspired by his description, I came up with an implementation[2] that is
simple -- less than 20 lines of executable code -- and I think has a
lot of nice properties. I think this can make choices-related code
significantly more elegant.

May we please reopen the ticket?

Thanks,
Shai.

[1] https://code.djangoproject.com/ticket/27910
[2]https://github.com/django/django/compare/master...shaib:ticket_27910?expand=1

[James Bennett]

FWIW I'm pretty strongly -1 on this feature because Python's enums don't behave the way people want them to for many use cases.

The basic problem is that, to take the example in the ticket, if I were to issue a request like "/students/?year=FR", and the view were to read that "year" param and try to filter on it, it would fail -- the string "FR" will not compare equal to the enum member "FR". So I couldn't easily use that to filter for students whose year is Student.YearInSchool.Freshman.

This is a deliberate design decision in Python's enums which, unfortunately, makes them unsuitable for the kinds of things people would commonly use them for in Django; only the special one-off enum.IntEnum has members that actually are comparable to a base Python type, and writing special-case enum variants for other types to try to enable comparability would be a pain.

[Tom Forbes]

Hey Shai,

I have not had a chance to look at the implementation, but I was describing my django-choice-object library[1] and forgot to link it!

I would be +1 on re-opening it, I've used enums for this before and I find it much more DRY than the more 'standard' Django way. Perhaps we could reduce some boilerplate here by somehow automatically adding the key names to gettext?

1. https://github.com/orf/django-choice-object

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20181231201915.6709633b.shai%40platonix.com.
For more options, visit https://groups.google.com/d/optout.

[Shai Berger]

Hi James,

On Mon, 31 Dec 2018 10:24:39 -0800
James Bennett <ubern...@gmail.com> wrote:

> The basic problem is that, to take the example in the ticket, if I
> were to issue a request like "/students/?year=FR", and the view were
> to read that "year" param and try to filter on it, it would fail --
> the string "FR" will not compare equal to the enum member "FR". So I
> couldn't easily use that to filter for students whose year is
> Student.YearInSchool.Freshman.

Like so:

>>> class SchoolYear(Enum):
... FRESHMAN = 'FR'
... JUNIOR = 'JR'
...
>>> SchoolYear.FRESHMAN == 'FR'
False

>
>
> This is a deliberate design decision in Python's enums which,
> unfortunately, makes them unsuitable for the kinds of things people
> would commonly use them for in Django; only the special one-off
> enum.IntEnum has members that actually are comparable to a base
> Python type, and writing special-case enum variants for other types
> to try to enable comparability would be a pain.
>

The actual definition of IntEnum is

class IntEnum(int, Enum):
"""Enum where members are also (and must be) ints"""

(it turns out that you don't need a 'pass' statement when you have a
docstring). And indeed,

>>> class SchoolYear(str, Enum):
... FRESHMAN = 'FR'
... JUNIOR = 'JR'
...
>>> SchoolYear.FRESHMAN == 'FR'
True

All it takes is adding the base Python type as a base of your
enumeration type.

Shai.

[Shai Berger]

Hi Tom,

On Mon, 31 Dec 2018 18:26:14 +0000
Tom Forbes <t...@tomforb.es> wrote:

> I was describing my django-choice-object library[1] and forgot to
> link it!
>

Thanks, I took a look -- the library looks nice, but notably, Choice
objects are not Python enums, although they share some of their traits.

> I would be +1 on re-opening it, I've used enums for this before and I
> find it much more DRY than the more 'standard' Django way.

Thanks.

> Perhaps we could reduce some boilerplate here by somehow automatically adding
> the key names to gettext?
>

This, I'm afraid, won't fly. The way xgettext (the engine used by
makemessages) generates the translation files is by scanning the text
of source files, looking for specific function calls, and picking their
arguments with a simple text transformation. So, for the key name to
somehow be identified, it must be a parameter in a function call (not
the target of assignment), and the form to be translated must be the
form that's in the source -- no changing capitalization, or
underscores-to-spaces, or anything of the sort. So, the best we could
do to minimize repetitions would create Choice classes looking like

class SchoolYear(Choices):
choice('Freshman', 'FR')
choice('Junior', 'JR')

where the choice() calls add members to the class, and the name of the
member is generated by some transformation over the label. That,
IMO, would be unacceptably unpythonic.

Have fun,
Shai.

> 1. https://github.com/orf/django-choice-object
>

[Tom Forbes]

Unfortunately I made this for a python 2 app before enums where in the standard library. I think it has some useful ideas though especially around categories, which I find I needed a lot. I'd be interested to know if anyone else has written any workflow-driven apps in Django that require groups of choice values (i.e "if value in Choices.SOME_GROUP)?

Its a shame about the way xgetext worked, but I guess it's not too bad to duplicate this if you require it. I'm not too familiar with it and assumed it worked by importing modules and recording calls to the function.

In any case, we could make the display text optional if you do not require translations and generate them from the keys?

The way Django handles choices has always irked me as it's really not DRY, you need to create sort-of enums using class or global variables and tuples. It's repetitive and boring, and honestly enums made this much more usable.

We currently have a -1 and a +1, does anyone else have any input on this?

Tom

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20190101001534.29c8c673.shai%40platonix.com.

[Ryan Hiebert]

I would also love to see a decent way to use enums as choices. The translation issues seem the stickiest to me, but I'm kinda hoping we can come up with something reasonable.

I'm a heavy user, but mostly a lurker on this list, but +1 from me for what it's worth.

Ryan

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJPJcqi7bELk_1QFOp3ZMZV1njYd9nWSc_bumymkzta1BQ%40mail.gmail.com.

[Shai Berger]

On Mon, 31 Dec 2018 23:40:53 +0000
Tom Forbes <t...@tomforb.es> wrote:
>
> We currently have a -1 and a +1, does anyone else have any input on
> this?
>

So, the tally is +3, -1, and the -1 seems to be based on a technical
argument which I believe I refuted; and more than a week has passed
with no further comment.

Looks to me like at least a rough consensus of "eh, ok, if you really
want it..."

[Luke Plant]

I've also tried to come up with something better than Django's current
default, using Enums, and found them lacking when it comes to the
caption, especially when you need translations.

I would be happy with re-opening the ticket, but in terms of committing
a solution I would be -1 unless we can find something which is
definitely significantly better than the status quo, which the majority
of people would want to use, rather than just an alternative - something
that we would be happy to be become the recommended default and present
in the docs as such. Otherwise we just have 2 ways to do it.

I quite like Tom's suggestion here:

https://code.djangoproject.com/ticket/27910#comment:6

It's not perfect but seems a reasonable compromise given the constraints.

Luke

[James Bennett]

Shai, your rebuttal still misses an important use case, which is containment.

To continue with your example of a 'SchoolYear(str, Enum)', the following will both be False:

'FR' in SchoolYear

'FR' in SchoolYear.__members__

The first of those is also becoming illegal soon -- attempting an 'in' comparison on an Enum, using an operand that isn't an instance of Enum, will become a TypeError in Python 3.8.

Instead you have to do something like:

'FR' in [choice.value for choice in SchoolYear]

to get a containment check. And you need a containment check to perform validation.

There are other ways to do it, but they're all clunky, and this is going to be code that people have to write (in some form) over and over again, unless we build our own choice abstraction that hides this by wrapping an Enum and implementing a __contains__ that does what people want. And you'd basically have to do it in a wrapper, because Enum does metaclass stuff that interferes with a typical "just subclass and override" approach.

So I still don't think this is going to work for the model-choice use case without a lot of fiddling (and I'm still not a fan of the enum module in general, largely because it seems to have gone out of its way to make this use case difficult to implement).

[Tom Forbes]

While I agree that Enum’s are a bit clunky (and IMO removing in is a poor choice), is this going to really take be that hard to work with or that difficult to validate?

Enums are types that raise ValueError, like any other, so it’s not that confusing and fits in with Django’s existing validation code. Once we have coerced the input into an enum member then validation against choices is simple no?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAL13Cg_KEtgmNzRo%3DC8cnCiNsukr5YOCTv3MF8AybV%3D%3DiUXP0g%40mail.gmail.com.

[rap...@makeleaps.com]

I saw this thread so wanted to share a bit of a gotcha we had with enums internally for anyone trying to handle this stuff

Internally, we have a wrapper around CharField to work with enums (we go further than just choices and actually have the values be enums), but there's unfortunately still a good amount of issues in certain use cases that make it not a great fit to be put into general usage for Django. I do think it's possible to get things right, but you have to establish certain rules because the way enums work in Django can sometimes be surprising.

We notably had an issue with translations, and I believe the problem would occur with other context-sensitive values. Because database operations will often copy values before using them to save to the database, you're beholden to (what I believe to be) slightly busted semantics on `copy.copy` for enums. Probably defining a django-y enum subclass that requires identity/lookup helpers would make this more usable for the general public (much like what other people have said).

In [10]: from enum import Enum
In [11]: from django.utils.translation import ugettext_lazy, override
In [12]: class C(Enum):
    ...:     a = ugettext_lazy("Some Word")

In [13]: with override('en'):
    ...:     elt = C.a
    ...: with override('ja'):
    ...:     copy.copy(elt)
    ...:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-13-9c36cbd82121> in <module>
      2     elt = C.a
      3 with override('ja'):
----> 4     copy.copy(elt)
      5

/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py in copy(x)
    104     if isinstance(rv, str):
    105         return x
--> 106     return _reconstruct(x, None, *rv)
    107
    108

/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    272     if deep and args:
    273         args = (deepcopy(arg, memo) for arg in args)
--> 274     y = func(*args)
    275     if deep:
    276         memo[id(x)] = y

/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py in __call__(cls, value, names, module, qualname, type, start)
    289         """
    290         if names is None:  # simple value lookup
--> 291             return cls.__new__(cls, value)
    292         # otherwise, functional API: we're creating a new Enum type
    293         return cls._create_(value, names, module=module, qualname=qualname, type=type, start=start)

/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py in __new__(cls, value)
    531                     return member
    532         # still not found -- try _missing_ hook
--> 533         return cls._missing_(value)
    534
    535     def _generate_next_value_(name, start, count, last_values):

/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py in _missing_(cls, value)
    544     @classmethod
    545     def _missing_(cls, value):
--> 546         raise ValueError("%r is not a valid %s" % (value, cls.__name__))
    547
    548     def __repr__(self):

ValueError: 'Japanese Version of Some Word' is not a valid C

[Shai Berger]

Back to this issue for the DjangoCon EU sprints...

James' objection about the inclusion testing needed for validation
seems moot, because validation can be done by conversion to the enum
type (as noted by Tom).

Raphael's warning about the woes of translation are already handled by
the suggested implementation -- because it isn't the enum value that
we make translatable, but rather an attribute on it.

I should point that the suggested implementation uses IntEnum and
StrEnum. The Python documentation recommends against using these,
except in the case that one needs to account for compatibility with an
existing protocol -- which, I submit to you, is exactly our case (the
"protocol" being the types available for database fields).

As a reminder, the relevant links are:

https://code.djangoproject.com/ticket/27910
https://github.com/django/django/compare/master...shaib:ticket_27910?expand=1

Thanks,
Shai.

[Markus Holtermann]

Thanks for the proposal, Shai. I quite like it.

As discussed at the DCEU sprints I think I'd like to be able to omit the display form of an item and have it auto generated from an item's name, i.e. turning "FOO_BAR" into "Foo Bar" (`key.replace("_", " ").title()`)

Further, we could also simplify the Fields API more and do this:

class Card(models.Model):
suit = models.IntegerField(choices=Suit)

Cheers,

/Markus

> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-developers/20190413113323.1342ed7d.shai%40platonix.com.

[Tom Forbes]

I really like this implementation, it seems really clean and simple.

I would suggest that it might be worth seeing it’s simple to make the display text optional, I think in the general case capitalising the key name would be acceptable. Users can always override it if required or if they need translation support.

Even if we decide to make the display names required in all cases, it’s a great improvement over the current way of defining choices IMO.

Tom

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20190413113323.1342ed7d.shai%40platonix.com.

[Shai Berger]

On Sat, 13 Apr 2019 08:22:11 -0400
"Markus Holtermann" <in...@markusholtermann.eu> wrote:

>
> As discussed at the DCEU sprints I think I'd like to be able to omit
> the display form of an item and have it auto generated from an item's
> name, i.e. turning "FOO_BAR" into "Foo Bar" (`key.replace("_", "
> ").title()`)
>

Yes, that's an improvement.

> Further, we could also simplify the Fields API more and do this:
>
> class Card(models.Model):
> suit = models.IntegerField(choices=Suit)
>

I have doubts about this, because enum classes are iterable -- so at
first look, it might not be totally obvious of

suit = models.IntegerField(choices=Suit)

actually means

suit = models.IntegerField(choices=list(Suit))

or

suit = models.IntegerField(choices=Suit.choices())

although the repetition of the word "choices" in the last version is a
bit jarring. I could be convinced to change my mind.

Thanks,
Shai.

[Curtis Maloney]

On 4/13/19 10:22 PM, Markus Holtermann wrote:
> Thanks for the proposal, Shai. I quite like it.
>
> As discussed at the DCEU sprints I think I'd like to be able to omit the display form of an item and have it auto generated from an item's name, i.e. turning "FOO_BAR" into "Foo Bar" (`key.replace("_", " ").title()`)
>

For reference, this is what my project does ( labeled-enums:
https://pypi.org/project/labeled-enum/ )


--
Curtis

[Shai Berger]

Hi all,

A bit of an issue came about in the PR: If you define an enum class
which inherits `str`, there's an ambiguity when asking for its string
representation (that is, explicit conversion to `str`) -- between it
simply being a `str`, and Enum's `__str__()` which returns
`Type.NAME'.

The ambiguity is only where str-based enums are concerned.

I believe my comment linked below summarises the discussion well, but
others may (of course) disagree.

https://github.com/django/django/pull/11223#issuecomment-496113943

Anyway, input welcome,

Shai.

[Fernando Macedo]

Hi everyone, I come late to the party and only noticed that Django now has support for enums on the 3.0 release notes. As I'm not a core developer I never imagined this.

I've wrote a lib that does something similar in mid-2018: https://github.com/loggi/python-choicesenum

There's one main diff from the Python default behaviour: The default enum requires that you compare the enum against their value, something like `assert EnumClass.EnumItem.value == 3`, my custom implementation allow usage with `assert EnumClass.EnumItem == 3`, the expected behaviour IMHO.

Thanks for your hard work!

To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

[Shai Berger]

Hi Fernando,

On Mon, 16 Sep 2019 08:59:17 -0700 (PDT)
Fernando Macedo <fgma...@gmail.com> wrote:

> Hi everyone, I come late to the party and only noticed that Django
> now has support for enums on the 3.0 release notes. As I'm not a core
> developer I never imagined this.

Late is better than never... I should point out that contribution to
the project is open -- neither I, who initiated this work, nor Nick,
who pulled most of the rest of the way, are currently members of the
project's Technical Team ("core developers").

This, of course, is not intended as an admonition, but rather as an
invitation. Come join us, one and all!

>
> I've wrote a lib that does something similar in mid-2018:
> https://github.com/loggi/python-choicesenum
>
> There's one main diff from the Python default behaviour: The default
> enum requires that you compare the enum against their value,
> something like `assert EnumClass.EnumItem.value == 3`, my custom
> implementation allow usage with `assert EnumClass.EnumItem == 3`, the
> expected behaviour IMHO.
>

I believe Django's enum classes also have this property. I took a short
glance at your implementation, and I believe Django's has most of its
features; you're welcome to take a look at ours[1] and note if we
missed anything important. If we did, perhaps there's still time to fix
it before the Beta.

[1] https://github.com/django/django/blob/stable/3.0.x/django/db/models/enums.py

Thanks for coming forward with this,

Shai.