Technical Board statement on type hints for Django

[Carlton Gibson]

Hi all.

The question of using typing, or type hints, or type checking, in (or with) Django
has come up several times. Whether we would add inline annotations, or use stub
files, or what?

Most recently, this resulted in a draft DEP[0] to try and formalize the situation,
followed by a PR[1] adding the `__class_getitem__()` to `QuerySet` and `Manager`
that would ease the job of working on external stub files.

[0]: https://github.com/django/deps/pull/65
[1]: https://github.com/django/django/pull/12405

There was extended discussion on both the draft DEP and the PR, which I
recommend for anyone particularly interested in this topic.

Given the lack of consensus on the issue, I asked the Technical Board to review
the situation in full, and make a judgement on the way forward.

Having been charged to do so, I here post the Technical Board's response: 


    It is the view of the Django Technical Board that inline type annotations
    should not be added to Django at the current time.

    A brief idea of the considerations are these:

    * Typing in Python is still young and in flux. As such it is not
      sufficiently stable. 
    * There are competing technologies, which may settle with time, but at this
      moment Django is not in a position to favor any one.
    * Writing correct type hints is hard. And reviewing them to ensure
      correctness isn't any easier. Even ostensibly simple examples often have
      hidden complexity. Thus the barrier to contribution and maintenance is
      raised.
    * Due to the very dynamic nature of much of Django, example type hints very
      often do not pass an acceptable standard for readability. This might be
      manageable with extensive aliasing but that effort itself involves
      further overhead.
 
    The Technical Board acknowledges the type checking is gaining ground in
    Python and that many users are keen to employ it.

    The django-stubs project is notable for the impressive work that it has
    done adding external stub files for Django.

    It is the view of the Technical Board that non-invasive changes in order to
    assist the work of external projects such as django-stubs are acceptable.

    In particular, this means that the pull request to add the
    `__class_getitem__()` method to QuerySet and Manager should be accepted.

    A small number of further changes of a similar nature may be accepted, but
    the Technical Board wishes to emphasise that where possible typing should
    be done in external stub files, and that the barrier for further inline
    changes will be high.

    We're looking forward to seeing how typing evolves in Python and
    re-evaluating Django's position as things change.


On the basis of this, Mariusz and I (in our role as Mergers) shall review and
merge the PR #12405 in time for Django 3.1. (We will also close the draft DEP,
and any related tickets, over the coming period.)

Personally, I wish to thank everybody who has participated in the discussion to

bring us to this point.

Kind Regards,

Carlton

[Adam Johnson]

So we in the technical board were a bit opaque and had our discussion in private before Carlton posted our summary. Apologies for this. We'll repeat the discussion in the open so you can see our reasoning.

On 4 March Carlton prompted for our input. I replied:

My experience using types, so far:

• I'm the author of flake8-no-types, the flake8 plugin for banning type annotations 😂 I wrote it whilst reviewing a client's code base which had type annotations, mostly added by PyCharm autocomplete, but no MyPy running. Running MyPy produced >1k errors. Eventually I actually fixed the errors rather than stripping the annotations and installing flake8-no-types - there was thankfully a lot of repetition in the errors.
  
• I have a vague memory writing one open source PR using types but can't remember the project.
  
• I'm a little confused keeping up with the various ways of declaring types, 'best practices", and which type checkers are out there (last I heard there are 4!?).
  

I agree that the tools are young, and evolving relatively quickly - both MyPy and the standard library. I'm not sure there's quite enough stability yet to roll types out across the whole of Django.

There's the risk that we end up in a chicken/egg stalemate where if no major Python project commits to being typed, types don't get enough adoption to progress. But I think there are enough popular projects, like Starlette, that are using them.

At current it does seem to be a vocal niche of developers who want them.

Yes it will make PR's harder to review, and definitely adds a barrier to contribution. Even relatively Pythonic Django-ey constructs like "accept anything with an as_expression method, or castable to a string" require quite a lot of type system knowledge to write correctly. On top of this the best practices seem to still be changing.

One extra benefit would be a little more clarity around Django API's. For example, I think a past mistake was allowing database expressions' 'params' to be *either* a tuple or list, which means casting everywhere (a bug for years in django-mysql: https://github.com/adamchainz/django-mysql/pull/558/files ). That said this is relatively low impact - the documentation often doesn't mention types at all and Python developers are used to checking things in a REPL.

I think accepting the current PR is okay. [the one for __class_getitem__ ]

A next step could potentially be experimentally by providing type annotations in more limited scope projects. One that could work well is asgiref, as it could also provide "the" ASGI type signatures - which Starlette already started internally: https://github.com/encode/starlette/blob/master/starlette/types.py

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/47283a09-7899-4449-8792-e31b5c779c4e%40googlegroups.com.

--

Adam

[Markus Holtermann]

Thanks for pushing this public, Adam. In the discussion I brought up these points.

I've been using static typing in Python for about 1.5 years now. Every now and then it's neat, but often enough I get annoyed by it. Either because I simply don't know how to use the tools at hand correctly or because there's something that's just not expressible via type annotations.

The DEP is indeed very much draft. It covers so much, for a rather young setup, that at this point, I would not accept it. Everything from it's mention of `@overload` is too complex and I'm not sure any of that API is stable enough to rely on it in Django.

What I could see Django doing is having type annotations that don't go beyond the stdlib support: e.g. `def foo(bar: str) -> Optional[int]` or `def foo(instance: models.Model) -> int:`. But far more complex constructs will eventually cause breakage and displeasure during development of and with Django.

Often enough at work there's situations where no one can figure out how to properly type-annotate something. We then end up with `# type: ignore` or not typing a function at all. That's technical dept that we'll hardly ever lose. And because it doesn't show up in any CI, it won't cause any issue. But that means, we'll eventually have everything type-ignored or heaps of function not typed at all. I do not want to see that in Django. If something can't be typed or the typing is wrong: `# type: ignore` shouldn't ever be used. At this point, I'd instead drop the typing that causes it.

The DEP kind of singles out mypy. As Adam already mentioned, there's about four type checkers out there. I wouldn't want Django to pick one and have special support for that. Instead, everything that would be special to one of them should be developed externally. Especially since these tools iterate quickly and change quickly. If a plugin were to be bundled with Django, we'd need to support it for 3 years. What when that tool changes the entire API (again, young scenery, high velocity).

With regards to the PR Carlton asked about (https://github.com/django/django/pull/12405), if that helps people to build tooling around Django, let's do it.

Cheers,

Markus

On Fri, Apr 17, 2020, at 11:41 AM, Adam Johnson wrote:
> So we in the technical board were a bit opaque and had our discussion
> in private before Carlton posted our summary. Apologies for this. We'll
> repeat the discussion in the open so you can see our reasoning.
>
> On 4 March Carlton prompted for our input. I replied:
>
> > My experience using types, so far:

> > * I'm the author of flake8-no-types, the flake8 plugin for banning type annotations 😂 I wrote it whilst reviewing a client's code base which had type annotations, mostly added by PyCharm autocomplete, but no MyPy running. Running MyPy produced >1k errors. Eventually I actually fixed the errors rather than stripping the annotations and installing flake8-no-types - there was thankfully a lot of repetition in the errors.
> > * I have a vague memory writing one open source PR using types but can't remember the project.
> > * I'm a little confused keeping up with the various ways of declaring types, 'best practices", and which type checkers are out there (last I heard there are 4!?).

> > To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/47283a09-7899-4449-8792-e31b5c779c4e%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/47283a09-7899-4449-8792-e31b5c779c4e%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
>
> --
> Adam

>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-developers/CAMyDDM0-NYTrChBTzjHTak6HhH5Pbkrmo_UQ4yFqJ6i_WurcBA%40mail.gmail.com <https://groups.google.com/d/msgid/django-developers/CAMyDDM0-NYTrChBTzjHTak6HhH5Pbkrmo_UQ4yFqJ6i_WurcBA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[charettes]

Thanks for sharing your answer publicly Adam and Markus. Here are the points I brought up in the discussion

From my limited experience with type annotation on a large monolithic code base that was ported from Python 2 to 3 even gradual typing has been a lot of trouble for the few benefits it brought us.

This partially explains why I had reservations about the benefits of the proposal.

I also think that it's a bit too early to bundle type annotations in the core repository and I'd favour an approach where we make adjustments to the source to add entry points allowing third parties to implement and maintains such type annotations. That's the approach we've taken in the past when a new feature is suggested for inclusion in core; we add missing entry points for the feature to be implemented externally instead. This should allow different tools to compete organically in the ecosystem without having to choose the best candidate right now.

I'm -0 on allowing stdlib type annotations in the mean time since I feels like it would open a can of worms that we want to keep sealed for now. What if a function accepts/return a non-stdlib type (or an Union of a stdlib and non-stdlib), do we type annotate it? What if we have to change this function to a Union[non-stdlib-type, stdlib-type], should we completely drop the type annotation on that function?

Simon

> >  To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

> >  To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/47283a09-7899-4449-8792-e31b5c779c4e%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/47283a09-7899-4449-8792-e31b5c779c4e%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
>
> --
> Adam
>
>  --
>  You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
>  To unsubscribe from this group and stop receiving emails from it, send

> an email to django-d...@googlegroups.com.

[Aymeric Augustin]

Sorry for the late answer, I didn't have time to read this mailing list for the last month :-( Here's my position on the matter.

Adding type annotations only makes sense if a type checker checks them. Otherwise, they're unlikely to be correct and usable, so it would be counterproductive to add them. Unfortunately, I don't think we have a good enough type checker yet.

I'm having a rough experience maintaining type checking with mypy in websockets, a library that is very much smaller and simpler than Django. (Some problems are related to async, which isn't a concern for Django, but that doesn't explain everything.)

* Besides adding the type annotations, type checking required many other code changes. Importing types in modules that need them created circular dependencies. Fixing these import loops required moving imports to the bottom (ugh), splitting code in small modules (meh), or always importing modules rather than objects inside modules (perhaps a good practice, but changing this throughout a project creates a lot of noise). Also, import loops tend to reappear when implementing new features or refactoring.

* It's an uphill struggle to keep up with subtle changes of behavior in mypy — which is still 0.x software. I also encountered regressions that required pinning to an old version. Supporting more than one mypy version seems out of reach. I'm very pessimistic about supporting more than one type checker.

* I don't remember mypy catching interesting bugs. I'm keeping it only because I'm failing to deal with the sunk cost logical fallacy :-( Overall, adding type annotation to websockets has decreased my ability and willingness to maintain it. I believe it increased the barrier to entry for other contributors as well.

So, I think the ecosystem needs to stabilize a lot before Django can adopt static typing, even just as a best practice for new code. Comments like this one make me uncomfortable about living that close to the edge.

Also, given that static typing was bolted on Python very late in its history and that there's a strong tradition of relying on duck typing, I'm not yet convinced static typing will ever be widely accepted as a best practice.

tl;dr I'm recommending to wait and see.

For the record, I'm a huge fan of good type systems. My second programming language after Basic was Caml, which has fantastic type inference: you never declare types but everything is strongly statically typed at compile time. I loved it. In contrast, having to declare types manually to get passable type checking isn't good enough for me.

Finally, I have to say that I was impressed by the effort that Maksim (author of the DEP), along with others, put into making mypy work with the more dynamic parts of Django. For example, typing of QuerySet arguments is impressive: https://github.com/django/deps/pull/65/files#diff-afab5a339bc83e08032ffb9446fb5103R185. The documentation mentions a list of things that work, which suggests that edge cases not mentioned in the list may not work, so it isn't clear to me if it's possible to build a 100% correct solution with this approach. Still, it's impressive!

Best regards,

-- 

Aymeric.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/faeb43bf-5b00-47dc-81aa-56dd7b8a6fa6%40googlegroups.com.