[Django] #35980: PyPI will require uploaded source distribution filenames to comply with PEP 625 and contain the normalized project name "django"

[Django]

#35980: PyPI will require uploaded source distribution filenames to comply with PEP
625 and contain the normalized project name "django"
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Type:
| Cleanup/optimization
Status: new | Component: Core
| (Other)
Version: 5.1 | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
After the recent security release, I received some email notifications
from PyPI:

> This email is notifying you of an upcoming deprecation that we have
determined may affect you as a result of your recent upload to 'Django'.
>
> In the future, PyPI will require all newly uploaded source distribution
filenames to comply with PEP 625. Any source distributions already
uploaded will remain in place as-is and do not need to be updated.
>
> Specifically, your recent upload of 'Django-5.1.4.tar.gz' is
incompatible with PEP 625 because it does not contain the normalized
project name 'django'.
>
> In most cases, this can be resolved by upgrading the version of your
build tooling to a later version that supports PEP 625 and produces
compliant filenames.
>
> If you have questions, you can email ad...@pypi.org to communicate with
the PyPI ad...@pypi.org to communicate with the PyPI administrators.

I believe this relates to: https://github.com/pypi/warehouse/issues/12245
There's a chance that they might stop supporting non-normalized names by
"end of year":
https://github.com/pypi/warehouse/issues/12245#issuecomment-2272378958

I believe the name normalization was also discussed here:
https://github.com/django/django/pull/17806#issuecomment-2176465382 hence
cc-ing some people involved from that PR

I'm not 100% certain the path forward, it feels to me we might need to
make a change and backport to all supported versions and perhaps announce
this on the blog. I have no preference, but I need help to know what we
should do here.
--
Ticket URL: <https://code.djangoproject.com/ticket/35980>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35980: PyPI will require uploaded source distribution filenames to comply with PEP
625 and contain the normalized project name "django"

--------------------------------------+------------------------------------
Reporter: Sarah Boyce | Owner: (none)

Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: 5.1

Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

--------------------------------------+------------------------------------
Changes (by Claude Paroz):

* stage: Unreviewed => Accepted

Comment:

AFAIR, we refrained from using a recent setuptools with `requires =
["setuptools>=61.0.0,<69.3.0"]`, to avoid that change (Django to django).
Maybe you could try a setuptools version > 69.3 and see what's happening?
--
Ticket URL: <https://code.djangoproject.com/ticket/35980#comment:1>

[Django]

#35980: PyPI will require uploaded source distribution filenames to comply with PEP
625 and contain the normalized project name "django"
--------------------------------------+------------------------------------
Reporter: Sarah Boyce | Owner: (none)
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+------------------------------------

Changes (by Nick Pope):

* has_patch: 0 => 1

Comment:

Certainly using v69.3.0 as a minimum version for `setuptools` should make
us compliant with PEP 625.

But, honestly, I remain as baffled as last time I looked into this. I
found
[https://github.com/pypa/setuptools/issues/3593#issuecomment-2176739508
this comment] which basically says that the source distribution filename
should follow the same normalisation rules as the binary distribution
filename and that states that uppercase should be replaced with lowercase.
So why, oh why, do I get the following?

{{{
$ git clean -fdX
$ python -m build
...
Successfully built django-5.2.dev20241206115354.tar.gz and Django-5.2-py3
-none-any.whl
}}}

I even tried with the latest `setuptools`, at the time of writing v75.6.0.

While we could go to the PyPA and say, "What the heck?", it's probably
just easier to change `name = Django` to `name = django` in
`pyproject.toml` and move on:

{{{
$ git clean -fdX
$ python -m build
...
Successfully built django-5.2.dev20241206115354.tar.gz and django-5.2-py3
-none-any.whl
}}}

It also seems concerning that the version in the wheel filename is
incomplete...

Here is a draft [https://github.com/django/django/pull/18891 PR] for
consideration.

I'm not sure what we'll have to do regarding backports, but it might mean
a more significant backport of the `setup.cfg` to `pyproject.toml`
changes.
--
Ticket URL: <https://code.djangoproject.com/ticket/35980#comment:2>

[Django]

#35980: PyPI will require uploaded source distribution filenames to comply with PEP
625 and contain the normalized project name "django"
--------------------------------------+------------------------------------
Reporter: Sarah Boyce | Owner: (none)
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+------------------------------------

Comment (by Nick Pope):

Also looking at https://packaging.python.org/en/latest/specifications
/binary-distribution-format/#file-name-convention...

Oh the irony that "django" is shown as an example of the distribution
name... :')
--
Ticket URL: <https://code.djangoproject.com/ticket/35980#comment:3>

[Django]

#35980: PyPI will require uploaded source distribution filenames to comply with PEP
625 and contain the normalized project name "django"
--------------------------------------+------------------------------------
Reporter: Sarah Boyce | Owner: (none)
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+------------------------------------

Comment (by Baptiste Mispelon):

> AFAIR, we refrained from using a recent setuptools with requires =
["setuptools>=61.0.0,<69.3.0"], to avoid that change (Django to django).

Yes, that was done in this merge request:
https://github.com/django/django/pull/17806#issuecomment-2176465382

I've just tried to build with the latest setuptools (`python -m pip
install build twine && python -m build` from a clean checkout of the git
repo) and it creates two files:
* django-5.2.dev20241206115354.tar.gz
* Django-5.2-py3-none-any.whl

The inconsistent naming seems to be a bug in setuptools [1] (mentionned in
the comment on the patch I linked above).

My opinion is that we should be using the latest version of setuptools and
let it generate the filenames for the builds. Those filenames are internal
plumbing and are not normally exposed to users. They also don't change how
Django is installed when using tools like pip, since those tools will
normalize names internally as well.
One hitch with this plan is that the current release documentation (both
the public one and I assume the ad-hoc scripts used by the fellows) makes
assumptions about those filenames that would not hold true anymore.

[1] https://github.com/pypa/setuptools/issues/3777
--
Ticket URL: <https://code.djangoproject.com/ticket/35980#comment:4>

[Django]

#35980: PyPI will require uploaded source distribution filenames to comply with PEP
625 and contain the normalized project name "django"
--------------------------------------+------------------------------------
Reporter: Sarah Boyce | Owner: (none)
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+------------------------------------
Comment (by Baptiste Mispelon):

I took a look at the website's code to see what changes might be needed
if/when the build files' naming structure changes.

It's not too bad, but it's still quite substantial:
https://github.com/django/djangoproject.com/pull/1801.
With the changes proposed in this merge request there would be a slight
change in the release process, since the build files should now be
uploaded via the release creation form, and not via `scp` as currently
documented.
--
Ticket URL: <https://code.djangoproject.com/ticket/35980#comment:5>

[Django]

#35980: PyPI will require uploaded source distribution filenames to comply with PEP
625 and contain the normalized project name "django"
--------------------------------------+------------------------------------
Reporter: Sarah Boyce | Owner: (none)
Type: Cleanup/optimization | Status: new
Component: Core (Other) | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+------------------------------------
Comment (by Baptiste Mispelon):

Hi, me again 😁

I took a look at the Jenkins `confirm-release` job mentionned in the
release documentation and the only assumption it makes is that the
checksum file is named `Django-${VERSION}.checksum.txt`, then it fetches
the names of the build files (tarball and wheel as of now) from the
checksum file itself.

So we can either keep the capitalized name for the checksum, or change one
line in the Jenkins config.
--
Ticket URL: <https://code.djangoproject.com/ticket/35980#comment:6>

[Django]

#35980: PyPI will require uploaded source distribution filenames to comply with PEP
625 and contain the normalized project name "django"

-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: Natalia
Type: | Bidart
Cleanup/optimization | Status: assigned

Component: Core (Other) | Version: 5.1
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Natalia Bidart):

* cc: Baptiste Mispelon (added)
* owner: (none) => Natalia Bidart
* status: new => assigned

Comment:

Hello everyone, I'm assigning this ticket to myself to "own" the
transition process. I have read comments, reviewed PRs and pondered about
this. I have some thoughts:

* For the changes to djangoproject.com, I added a newer comment suggesting
to make the tarball and checksum a cached property (we could use something
not fancy but simple like "If the lowercase django-... file exists, return
that, otherwise return the uppercase one".
* Uploading the tarballs via the Django admin feels a step in the wrong
direction when thinking about automating the Django release process.
* For the changes to Django itself from Nick, those look good, we should
think what to do for 5.0 and 4.2. We may need to backport the
`pyproject.toml` changes to 5.0 and 4.2, I'm unclear about the
implications of this, but it certainly has advantages from the POV of
maintaining the project.
* It would certainly help the release process to use the same release
procedure for all versions.
--
Ticket URL: <https://code.djangoproject.com/ticket/35980#comment:7>