Switching to PEP 621 project metadata

[Ofek Lev]

Hello! This is about using only `pyproject.toml` for packaging https://github.com/django/django/pull/15874

The main question I think as Nick asked is whether it's okay to remove support for `python setup.py bdist_rpm`.

As I mention in the OP:

1. Fedora does not use that for its packaging and even recommends against using their RPM unless developing a package specific to their ecosystem

2. "all direct invocations of setup.py are effectively deprecated"

3. Similar to 2, the bdist_rpm code is part of distutils which is officially deprecated

What does everyone think?

[James Bennett]

On Sun, Jul 24, 2022 at 10:06 AM Ofek Lev <ofekm...@gmail.com> wrote:

Hello! This is about using only `pyproject.toml` for packaging https://github.com/django/django/pull/15874

My initial reaction is "no", and that this request kind of rubs me the wrong way. In the pull request you say:

> As a result, the execution of setup.py files is now heavily discouraged.

But the blog post you quote is just saying to run "python -m build" instead of "python setup.py", not to entirely remove setup.py, or that using setup.py as a configuration file is deprecated. In fact, the highlighted tl;dr at the top of the post says exactly the *opposite* of that!

Setting that aside I also am, in general, strongly against Django being an early adopter of any new packaging tool. While it's certainly helpful for the tool, because our user base is large enough to uncover all sorts of edge cases, it's not great for our users, who then get to be beta testers against their will. I went through this myself with the code formatter `black`, which was an eager and early adopter of several things and as a result would break my CI pipelines every couple of months when one of those early-adopted things had unintended side effects. Doing that to Django's user base would be bad for our users, bad for Django ("why does Django keep breaking?"), and bad for Python ("look, if even these high-profile packages break all the time, Python packaging must really be as terrible as everyone claims it is!").

So for now, the non-setup.py packaging ecosystem is still rapidly developing and still has rough edges. Maybe in a few years when the situation is improved, Django can adopt new and solid standard methods of doing things. But for now, the solid standard still very much is setup.py and the setuptools build backend. 

About all I could support at the moment is switching the Makefile that release managers run to use `python -m build`: 

https://github.com/django/django/blob/main/extras/Makefile

[Ofek Lev]

> My initial reaction is "no", and that this request kind of rubs me the wrong way. In the pull request you say [...] But the blog post you quote is just saying to run "python -m build" instead of "python setup.py"

This issue is that the `python -m build`/PEP 517 way does not support the deprecated RPM logic. I tried to articulate that as best as I could; I'm sorry if it wasn't clear!

> strongly against Django being an early adopter of any new packaging tool [...] I went through this myself with the code formatter `black`, which was an eager and early adopter of several things and as a result would break my CI pipelines every couple of months when one of those early-adopted things had unintended side effects.

I'm not familiar with what happened with black but I think you might be referencing how the recommended style was in flux for a time. If so, I totally experienced that too at work and we ended up pinning + upgrading at fixed intervals! I think that situation is a bit different though because it was about the community iteratively coming to a consensus whereas in this case all the PEPs are already accepted and in widespread use.

[James Bennett]

On Sun, Jul 24, 2022 at 1:34 PM Ofek Lev <ofekm...@gmail.com> wrote:

> My initial reaction is "no", and that this request kind of rubs me the wrong way. In the pull request you say [...] But the blog post you quote is just saying to run "python -m build" instead of "python setup.py"

This issue is that the `python -m build`/PEP 517 way does not support the deprecated RPM logic. I tried to articulate that as best as I could; I'm sorry if it wasn't clear!

We can run `python -m build` pretty much as-is using setuptools as the build backend, though, and that suffices for Django's packaging needs.

I'm not familiar with what happened with black but I think you might be referencing how the recommended style was in flux for a time.

I am referring to the multiple occasions on which black's approach to packaging caused installation of the package, via pip, to fail in often mysterious ways.

This is something that's documented in the history of their issue tracker, and that the maintainers have since apologized for and adopted policies to try to prevent in the future.

[Carlton Gibson]

I think we need to be very cautious. Anecdotally, I've seen folks hitting issues trying to adopt the new methods, and I don't see any upside to Django being an earlier adopter here. 

We were slow adopting PEP 517 and still had regressions reported: 

https://github.com/django/django/pull/12013

https://github.com/django/django/pull/13994

I'd much rather let the bulk of smaller projects adopt, and iron out the details, and then update when it's a better beaten track. 

On the particulars, I'm sure Hatchling is great, but it's not at all clear (from here — to me) that it's mature enough to trust Django with. It seems the packaging options are emerging very quickly (which is great) but I'd like the current (evolutionarily) explosive period to settle some, so we can see the long-term survivors going forward before we jump. 

With setup.py, I think for a project like Django, there's a certain value in being second-to-last off the boat. 

Kind regards, 

Carlton 

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAL13Cg_F1JxY5ANKw53LvEYc8_JjJGb_%2BSs-dZ2YxWr_tRw1Vw%40mail.gmail.com.

[Mariusz Felisiak]

I agree with James and Carlton, -1 from me.

Best,
Mariusz

[Ofek Lev]

Okay then! Just 2 questions before I go:

> I'd much rather let the bulk of smaller projects adopt, and iron out the details, and then update when it's a better beaten track.

Can we please quantify such heuristics? I want something concrete to write in my OSS to-do list :)

> I am referring to the multiple occasions on which black's approach to packaging caused installation of the package, via pip, to fail in often mysterious ways.

> This is something that's documented in the history of their issue tracker, and that the maintainers have since apologized for and adopted policies to try to prevent in the future.

I can't find any such events; do you happen to have links? I'd like to improve Python's packaging experience, in this case by learning from the past.

[Carlton Gibson]

Hey Ofek. 

On Mon, 25 Jul 2022 at 17:04, Ofek Lev <ofekm...@gmail.com> wrote:

> I'd much rather let the bulk of smaller projects adopt, and iron out the details, and then update when it's a better beaten track.

Can we please quantify such heuristics? I want something concrete to write in my OSS to-do list :)

Well, I guess A decent number™ of projects in the Django eco-system, would do. (No I'm not saying how many :) 

But if you were to offer updating Jazzband projects, and that all went smooth, then that would directly feed into people here knowing more about the state of play. 

Also, I'd be happy to look at this for the Channels Trio (Channels/Daphne/Channels Redis. I'm working on updates for all of those now. 

I did have Flit in mind... so my first question would be why Hatch rather than that? — But if you were prepared to discuss that I'd be appreciative of the contribution. 

Kind Regards,

Carlton

[James Bennett]

On Mon, Jul 25, 2022 at 8:04 AM Ofek Lev <ofekm...@gmail.com> wrote:

Can we please quantify such heuristics? I want something concrete to write in my OSS to-do list :)

I don't think we can or should provide an exact number of other projects that need to adopt what you're doing before we will. It's more of a "we'll know it when we see it" thing.

I can't find any such events; do you happen to have links? I'd like to improve Python's packaging experience, in this case by learning from the past.

Their GitHub issues have a packaging tag which turns up a fair number of things, and searching their GitHub issues for terms like "wheel" or "PEP 517" or other packaging terms will generally find the rest.