Re: [Django] #37159: Implement reproducible artifact builds

[Django]

#37159: Implement reproducible artifact builds
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Component: Packaging | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Charles Roelli):

Hi Jacob,

Thanks for bringing this up.

The current build backend setuptools has an
[https://github.com/pypa/setuptools/issues/2133 open issue for building
reproducible sdists], so we won't get reproducible sdists out-of-the-box.
As for the wheel, setting the environment variable `SOURCE_DATE_EPOCH`
keeps the file modification timestamp constant, so that may be sufficient
to get a reproducible wheel. For example, building the `6.0.6` tag twice
with `SOURCE_DATE_EPOCH=1` gives:

{{{
~/Code/django/6.0.6$ sha256sum ~1/build*/*
206304aa753040e562768b91669c2c79659d1b688332af94ce29a626aa26a85a
~/Run/django/6.0.6/build1/django-6.0.6-py3-none-any.whl
c915757dee35a461f569457ba73d567dc26934421971839886e3814196a821c0
~/Run/django/6.0.6/build1/django-6.0.6.tar.gz
206304aa753040e562768b91669c2c79659d1b688332af94ce29a626aa26a85a
~/Run/django/6.0.6/build2/django-6.0.6-py3-none-any.whl
ec46f79707689eb71eebe9aa9d4f2356e33d6e3234d1f3a344c76844b52c18eb
~/Run/django/6.0.6/build2/django-6.0.6.tar.gz
}}}

It may also be interesting to consider e.g. the build backend
[https://hatch.pypa.io/1.9/why/#build-backend hatchling] which has more of
a focus on reproducibility.

We could also work towards removing existing artifacts committed to the
repository like .mo files (#23321) and vendored CSS/JS, and building from
source where possible.
--
Ticket URL: <https://code.djangoproject.com/ticket/37159#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#37159: Implement reproducible artifact builds
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Component: Packaging | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Charles Roelli):

* has_patch: 0 => 1

Comment:

I've linked a PR to switch to `hatchling`. Moving away from `setuptools`
also allows us to remove `MANIFEST.in` (#36740).

This is the file listing diff from the unpacked setuptools sdist to the
unpacked hatchling sdist.

{{{
$ diff -u <(find setuptools/django-6.2.dev20260611202632 -printf '%P\n')
<(find hatchling/django-6.2.dev20260615084851/ -printf '%P\n')
@@ -6436,7 +6436,6 @@
tests/signing
tests/signing/tests.py
tests/signing/__init__.py
-tests/.coveragerc
tests/sites_framework
tests/sites_framework/tests.py
tests/sites_framework/migrations
@@ -10276,18 +10275,10 @@
docs/releases/3.0.2.txt
docs/lint.py
pyproject.toml
-MANIFEST.in
LICENSE
+.gitignore
AUTHORS
Gruntfile.js
INSTALL
LICENSE.python
-Django.egg-info
-Django.egg-info/entry_points.txt
-Django.egg-info/requires.txt
-Django.egg-info/PKG-INFO
-Django.egg-info/top_level.txt
-Django.egg-info/dependency_links.txt
-Django.egg-info/SOURCES.txt
-setup.cfg
CONTRIBUTING.rst
}}}

For the wheel, we have:

{{{
$ diff -u <(find setuptools/django-6.2.dist-info/ -printf '%P\n') <(find
hatchling/django-6.2.dev20260615084851.dist-info/ -printf '%P\n')
@@ -7,4 +7,3 @@
licenses/AUTHORS
licenses/LICENSE.python
METADATA
-top_level.txt
$ diff -u <(find setuptools/django/ -printf '%P\n') <(find
hatchling/django/ -printf '%P\n')
(empty)
}}}

The hashes for the hatchling artifact build which are hopefully
reproducible:

{{{
~/Code/django/hatchling-build$ sha256sum dist/*
275b282ec5efdb0d85926de3abf36d9f25cd4a1f7e010a8582b00f325123ae1c
dist/django-6.2.dev20260615084851-py3-none-any.whl
50e81862ee1a860fb17386201d739d4415e3ec2695cee38ea12a06cd3749fe5b
dist/django-6.2.dev20260615084851.tar.gz
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/37159#comment:6>

[Django]

#37159: Implement reproducible artifact builds
-------------------------------------+-------------------------------------

Reporter: Jacob Walls | Owner: Charles
Type: | Roelli

Cleanup/optimization | Status: assigned
Component: Packaging | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* owner: Jacob Walls => Charles Roelli

--
Ticket URL: <https://code.djangoproject.com/ticket/37159#comment:7>

[Django]

#37159: Implement reproducible artifact builds
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Charles
Type: | Roelli
Cleanup/optimization | Status: assigned
Component: Packaging | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Natalia Bidart):

* needs_better_patch: 0 => 1

Comment:

I support the goal of reproducible release artifacts, but I'm not
convinced that changing Django's build backend is the best way to achieve
it.

As mentioned in comment:5, the remaining reproducibility gap in
`setuptools` is a known issue rather than a fundamental limitation and is
expected to be addressed upstream. In the meantime, Django can already
benefit from reproducible wheels. There are also existing approaches, such
as
[https://github.com/ansible/ansible/blob/03d6209/packaging/release.py#L867-L899
Ansible's release tooling] and [https://github.com/wimglenn/setuptools-
reproducible setuptools-reproducible], that demonstrate potential paths
forward within the current ecosystem.

Hatchling is a reasonable backend, but the current issue does not seem to
justify a backend migration. Django's packaging needs are relatively
simple, reproducible wheels are already achievable today, and the
remaining setuptools gap is expected to be resolved.

My recommendation is to focus on improvements that provide the greatest
practical benefit while remaining aligned with the existing packaging
infrastructure. Therefore, I would favor adopting reproducible wheels and
allowing the `setuptools` ecosystem to address the remaining `sdist` gap
rather than introducing a new build backend to work around a temporary
limitation.
--
Ticket URL: <https://code.djangoproject.com/ticket/37159#comment:8>

[Django]

#37159: Implement reproducible artifact builds
-------------------------------------+-------------------------------------
Reporter: Jacob Walls | Owner: Charles
Type: | Roelli
Cleanup/optimization | Status: assigned
Component: Packaging | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Charles Roelli):

Thanks for the feedback. I agree that changing the build backend can't be
taken lightly, although given the advantages of hatchling over setuptools
it is worth discussing.

For the scope of this ticket, is it enough to adjust `docs/internals
/howto-release-django.txt` to mention running `python -m build` with
`SOURCE_DATE_EPOCH=1`?
--
Ticket URL: <https://code.djangoproject.com/ticket/37159#comment:9>