[Python-Dev] Migration plan for the distutils removal in Python 3.12
562 views
Skip to first unread message
Victor Stinner
May 10, 2022, 4:00:38 AM5/10/22
to Python Dev
Hi,
Two years ago, PEP 632 "Deprecate distutils module" was accepted: the
distutils package was deprecated in Python 3.10 and scheduled for
removal in Python 3.12. Questions.
* Is the Python ecosystem ready for the distutils removal? How many
projects are expected to be broken by this removal?
* Is setuptools a drop-in replacement of distutils for most cases? Are
there tools and documentation explaining how to replace distutils with
setuptools?
* Is there a tool to migrate from setup.py (distutils) to
pyproject.toml (setuptools)? The dephell project can convert a
setup.py script to pyproject.toml using poetry as the build system.
* Can we simply suggest installing setuptools to users who still use
"import distutils"? setuptools now provides the a "distutils" package.
* Should we keep distutils in Python stdlib a few more Python releases
if the removal causes too many practical issues?
A code search in top 5000 PyPI projects (at 2022-01-26) using the
regex '(import|from) distutils' matchs 5,567 lines in 1,229 projects.
Some statistics:
* 851 projects (1,372 lines) import distutils in setup.py
* 233 projects (700 lines) use distutils.version
* 92 projects (205 lines) use distutils.util
* 1,018 lines are type annotations (.pyi files)
I failed to group the remaining 2,272 lines.
Converting 851 projects from setup.py with distutils to pyproject.toml
is going to take a few months if not years. Python 3.12 will likely be
released in October 2023 (in year and a half).
Since setuptools 60 (December 2021), setuptools installs a
distutils-precedence.pth file to override the stdlib module with its
local distutils copy by default. The SETUPTOOLS_USE_DISTUTILS=stdlib
environment variable can be used to explicitly request the stdlib
flavor. If I understood correct, distutils is now maintained in
setuptools, rather than in the Python stdlib.
Python "make install" installs an up to date setuptools and so
indirectly its distutils-precedence.pth file which makes the distutils
local copy of setuptools available. But many Linux distributions
splits setuptools and pip from the "python" package.
Right now, Python still uses distutils internally for multiple use
cases. I propose to start with renaming the distutils package to
_distutils in the stdlib:
* https://github.com/python/cpython/issues/92584
* https://github.com/python/cpython/pull/92585
Right now, the change is blocked by pip which still imports distutils
(even if it prefers sysconfig in practice on Python 3.10 and newer). I
reported the issue to pip, it should be easy to fix.
Once Python will no longer depend on _distutils, it will be possible
to fully remove it.
By the way, Fedora 35 now longer provides automatically setuptools
when building a Python package. A dependency to setuptools must now be
explicit, since they are projects which don't use setuptools nor
distutils, but other build systems.
https://fedoraproject.org/wiki/Changes/Reduce_dependencies_on_python3-setuptools
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/XFZJ43VSIWCYNQ5ZYT64XF7PMH4YDUWC/
Code of Conduct: http://python.org/psf/codeofconduct/
Two years ago, PEP 632 "Deprecate distutils module" was accepted: the
distutils package was deprecated in Python 3.10 and scheduled for
removal in Python 3.12. Questions.
* Is the Python ecosystem ready for the distutils removal? How many
projects are expected to be broken by this removal?
* Is setuptools a drop-in replacement of distutils for most cases? Are
there tools and documentation explaining how to replace distutils with
setuptools?
* Is there a tool to migrate from setup.py (distutils) to
pyproject.toml (setuptools)? The dephell project can convert a
setup.py script to pyproject.toml using poetry as the build system.
* Can we simply suggest installing setuptools to users who still use
"import distutils"? setuptools now provides the a "distutils" package.
* Should we keep distutils in Python stdlib a few more Python releases
if the removal causes too many practical issues?
A code search in top 5000 PyPI projects (at 2022-01-26) using the
regex '(import|from) distutils' matchs 5,567 lines in 1,229 projects.
Some statistics:
* 851 projects (1,372 lines) import distutils in setup.py
* 233 projects (700 lines) use distutils.version
* 92 projects (205 lines) use distutils.util
* 1,018 lines are type annotations (.pyi files)
I failed to group the remaining 2,272 lines.
Converting 851 projects from setup.py with distutils to pyproject.toml
is going to take a few months if not years. Python 3.12 will likely be
released in October 2023 (in year and a half).
Since setuptools 60 (December 2021), setuptools installs a
distutils-precedence.pth file to override the stdlib module with its
local distutils copy by default. The SETUPTOOLS_USE_DISTUTILS=stdlib
environment variable can be used to explicitly request the stdlib
flavor. If I understood correct, distutils is now maintained in
setuptools, rather than in the Python stdlib.
Python "make install" installs an up to date setuptools and so
indirectly its distutils-precedence.pth file which makes the distutils
local copy of setuptools available. But many Linux distributions
splits setuptools and pip from the "python" package.
Right now, Python still uses distutils internally for multiple use
cases. I propose to start with renaming the distutils package to
_distutils in the stdlib:
* https://github.com/python/cpython/issues/92584
* https://github.com/python/cpython/pull/92585
Right now, the change is blocked by pip which still imports distutils
(even if it prefers sysconfig in practice on Python 3.10 and newer). I
reported the issue to pip, it should be easy to fix.
Once Python will no longer depend on _distutils, it will be possible
to fully remove it.
By the way, Fedora 35 now longer provides automatically setuptools
when building a Python package. A dependency to setuptools must now be
explicit, since they are projects which don't use setuptools nor
distutils, but other build systems.
https://fedoraproject.org/wiki/Changes/Reduce_dependencies_on_python3-setuptools
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/XFZJ43VSIWCYNQ5ZYT64XF7PMH4YDUWC/
Code of Conduct: http://python.org/psf/codeofconduct/
Christian Heimes
May 10, 2022, 5:35:20 AM5/10/22
to Victor Stinner, Python Dev
On 10/05/2022 09.53, Victor Stinner wrote:
> Hi,
>
> Two years ago, PEP 632 "Deprecate distutils module" was accepted: the
> distutils package was deprecated in Python 3.10 and scheduled for
> removal in Python 3.12. Questions.
>
> * Is the Python ecosystem ready for the distutils removal? How many
> projects are expected to be broken by this removal?
>
> * Is setuptools a drop-in replacement of distutils for most cases? Are
> there tools and documentation explaining how to replace distutils with
> setuptools?
>
> * Is there a tool to migrate from setup.py (distutils) to
> pyproject.toml (setuptools)? The dephell project can convert a
> setup.py script to pyproject.toml using poetry as the build system.
>
> * Can we simply suggest installing setuptools to users who still use
> "import distutils"? setuptools now provides the a "distutils" package.
>
> * Should we keep distutils in Python stdlib a few more Python releases
> if the removal causes too many practical issues?
Setuptools now provides _distutils_hack meta importer that injects a
> Hi,
>
> Two years ago, PEP 632 "Deprecate distutils module" was accepted: the
> distutils package was deprecated in Python 3.10 and scheduled for
> removal in Python 3.12. Questions.
>
> * Is the Python ecosystem ready for the distutils removal? How many
> projects are expected to be broken by this removal?
>
> * Is setuptools a drop-in replacement of distutils for most cases? Are
> there tools and documentation explaining how to replace distutils with
> setuptools?
>
> * Is there a tool to migrate from setup.py (distutils) to
> pyproject.toml (setuptools)? The dephell project can convert a
> setup.py script to pyproject.toml using poetry as the build system.
>
> * Can we simply suggest installing setuptools to users who still use
> "import distutils"? setuptools now provides the a "distutils" package.
>
> * Should we keep distutils in Python stdlib a few more Python releases
> if the removal causes too many practical issues?
facade distutils package. "from distutils import setup" will keep
working with modern setuptools.
> Right now, Python still uses distutils internally for multiple use
> cases. I propose to start with renaming the distutils package to
> _distutils in the stdlib:
>
> * https://github.com/python/cpython/issues/92584
> * https://github.com/python/cpython/pull/92585
Modules/Setup.stdlib. The remaining modules should be done in a couple
of weeks. I recommend that we do not rename distutils and instead remove
it entirely.
Christian
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/7R5DKP5NZI5MVYQ7H44LFIGDHHHO3FOM/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Michał Górny
May 10, 2022, 5:37:07 AM5/10/22
to Victor Stinner, Python Dev
On Tue, 2022-05-10 at 09:53 +0200, Victor Stinner wrote:
> Hi,
>
> Two years ago, PEP 632 "Deprecate distutils module" was accepted: the
> distutils package was deprecated in Python 3.10 and scheduled for
> removal in Python 3.12. Questions.
>
> * Is the Python ecosystem ready for the distutils removal? How many
> projects are expected to be broken by this removal?
In Gentoo, I have decided to handle migration from `setup.py install` to
> Hi,
>
> Two years ago, PEP 632 "Deprecate distutils module" was accepted: the
> distutils package was deprecated in Python 3.10 and scheduled for
> removal in Python 3.12. Questions.
>
> * Is the Python ecosystem ready for the distutils removal? How many
> projects are expected to be broken by this removal?
PEP517 backends and distutils replacement simultaneously. That is,
existing packages that still use the legacy codepath have =stdlib
distutils forced, and =local distutils are used when they are updated to
use PEP517 build. So far we're nearing updating 40% of Python packages
in Gentoo.
From my porting experience, I can't think of any major issue caused by
replacing stdlib distutils with the setuptools version. IIRC the only
problem I recall was that the "local" version installs .egg-info
as a directory (like setuptools does) while "stdlib" version installs it
as a file. This could mean trouble when the same version of the package
is rebuilt with the other distutils version but I don't know how that
impacts other package managers.
So far, much more trouble is caused by PEP517 backends and wheels,
though admittedly this is primarily a problem with setup.py allowing
very deep customization and people using it to do pretty much anything.
Therefore, from Gentoo's perspective, there should be no major problem
with removing distutils from stdlib in Python 3.12. Admittedly,
distributions are in the more convenient position here since we can
easily patch packages should we find any breakage, while users of pypi
are generally stuck with the published versions.
--
Best regards,
Michał Górny
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Christian Heimes
May 10, 2022, 5:44:36 AM5/10/22
to Victor Stinner, Python Dev
On 10/05/2022 09.53, Victor Stinner wrote:
> Hi,
>
> Two years ago, PEP 632 "Deprecate distutils module" was accepted: the
> distutils package was deprecated in Python 3.10 and scheduled for
> removal in Python 3.12. Questions.
>
> * Is the Python ecosystem ready for the distutils removal? How many
> projects are expected to be broken by this removal?
>
> * Is setuptools a drop-in replacement of distutils for most cases? Are
> there tools and documentation explaining how to replace distutils with
> setuptools?
>
> * Is there a tool to migrate from setup.py (distutils) to
> pyproject.toml (setuptools)? The dephell project can convert a
> setup.py script to pyproject.toml using poetry as the build system.
>
> * Can we simply suggest installing setuptools to users who still use
> "import distutils"? setuptools now provides the a "distutils" package.
>
> * Should we keep distutils in Python stdlib a few more Python releases
> if the removal causes too many practical issues?
>
> Two years ago, PEP 632 "Deprecate distutils module" was accepted: the
> distutils package was deprecated in Python 3.10 and scheduled for
> removal in Python 3.12. Questions.
>
> * Is the Python ecosystem ready for the distutils removal? How many
> projects are expected to be broken by this removal?
>
> * Is setuptools a drop-in replacement of distutils for most cases? Are
> there tools and documentation explaining how to replace distutils with
> setuptools?
>
> * Is there a tool to migrate from setup.py (distutils) to
> pyproject.toml (setuptools)? The dephell project can convert a
> setup.py script to pyproject.toml using poetry as the build system.
>
> * Can we simply suggest installing setuptools to users who still use
> "import distutils"? setuptools now provides the a "distutils" package.
>
> * Should we keep distutils in Python stdlib a few more Python releases
> if the removal causes too many practical issues?
Setuptools now provides _distutils_hack meta importer that injects a
facade distutils package. "from distutils import setup" will keep
working with modern setuptools.
facade distutils package. "from distutils import setup" will keep
working with modern setuptools.
> Right now, Python still uses distutils internally for multiple use
> cases. I propose to start with renaming the distutils package to
> _distutils in the stdlib:
>
> * https://github.com/python/cpython/issues/92584
> * https://github.com/python/cpython/pull/92585
> cases. I propose to start with renaming the distutils package to
> _distutils in the stdlib:
>
> * https://github.com/python/cpython/issues/92584
> * https://github.com/python/cpython/pull/92585
Erlend and I got most extension modules ported to autoconf and
Modules/Setup.stdlib. The remaining modules should be done in a couple
of weeks. I recommend that we do not rename distutils and instead remove
it entirely.
Christian
Modules/Setup.stdlib. The remaining modules should be done in a couple
of weeks. I recommend that we do not rename distutils and instead remove
it entirely.
Christian
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/FYRFMWETDJXU3SRAOTZJIQRN6QDLAMXT/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Victor Stinner
May 10, 2022, 7:25:49 AM5/10/22
to Christian Heimes, Python Dev
On Tue, May 10, 2022 at 11:28 AM Christian Heimes <chri...@python.org> wrote:
> > Right now, Python still uses distutils internally for multiple use
> > cases. I propose to start with renaming the distutils package to
> > _distutils in the stdlib:
> >
> > * https://github.com/python/cpython/issues/92584
> > * https://github.com/python/cpython/pull/92585
>
> Erlend and I got most extension modules ported to autoconf and
> Modules/Setup.stdlib. The remaining modules should be done in a couple
> of weeks. I recommend that we do not rename distutils and instead remove
> it entirely.
test_peg_generator and test_cppext build C extensions with distutils.
> > Right now, Python still uses distutils internally for multiple use
> > cases. I propose to start with renaming the distutils package to
> > _distutils in the stdlib:
> >
> > * https://github.com/python/cpython/issues/92584
> > * https://github.com/python/cpython/pull/92585
>
> Erlend and I got most extension modules ported to autoconf and
> Modules/Setup.stdlib. The remaining modules should be done in a couple
> of weeks. I recommend that we do not rename distutils and instead remove
> it entirely.
Until these tests are modified to use something else, we still need to
keep distutils. So I propose to rename it to _distutils to remove it
from the stdlib. Maybe these tests can use ensurepip to install
setuptools which provides distutils.
There is also the c-analyzer tool which uses distutils to run a C preprocessor.
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/TFBMHY4JV3QJL5I7VB65G7VHVUMJD4GX/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Christian Heimes
May 10, 2022, 11:40:10 AM5/10/22
to Victor Stinner, Python Dev
On 10/05/2022 13.18, Victor Stinner wrote:
> On Tue, May 10, 2022 at 11:28 AM Christian Heimes <chri...@python.org> wrote:
>>> Right now, Python still uses distutils internally for multiple use
>>> cases. I propose to start with renaming the distutils package to
>>> _distutils in the stdlib:
>>>
>>> * https://github.com/python/cpython/issues/92584
>>> * https://github.com/python/cpython/pull/92585
>>
>> Erlend and I got most extension modules ported to autoconf and
>> Modules/Setup.stdlib. The remaining modules should be done in a couple
>> of weeks. I recommend that we do not rename distutils and instead remove
>> it entirely.
>
> test_peg_generator and test_cppext build C extensions with distutils.
> Until these tests are modified to use something else, we still need to
> keep distutils. So I propose to rename it to _distutils to remove it
> from the stdlib. Maybe these tests can use ensurepip to install
> setuptools which provides distutils.
>
> There is also the c-analyzer tool which uses distutils to run a C preprocessor.
We can easily take care of test_cppext and add the build step to
> On Tue, May 10, 2022 at 11:28 AM Christian Heimes <chri...@python.org> wrote:
>>> Right now, Python still uses distutils internally for multiple use
>>> cases. I propose to start with renaming the distutils package to
>>> _distutils in the stdlib:
>>>
>>> * https://github.com/python/cpython/issues/92584
>>> * https://github.com/python/cpython/pull/92585
>>
>> Erlend and I got most extension modules ported to autoconf and
>> Modules/Setup.stdlib. The remaining modules should be done in a couple
>> of weeks. I recommend that we do not rename distutils and instead remove
>> it entirely.
>
> test_peg_generator and test_cppext build C extensions with distutils.
> Until these tests are modified to use something else, we still need to
> keep distutils. So I propose to rename it to _distutils to remove it
> from the stdlib. Maybe these tests can use ensurepip to install
> setuptools which provides distutils.
>
> There is also the c-analyzer tool which uses distutils to run a C preprocessor.
Makefile. How does test_peg_generator depend on distutils? I don't see
an import of distutils in the directory.
I would prefer to fix and remove all imports of distutils before we
resort to renaming the package. Please give us time until alpha 1. There
is no need to rush it *now*.
Christian
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/5FOZXCSYPFJGWVFI7S6QNUKKICNPU5AQ/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Jeremy Kloth
May 10, 2022, 11:58:48 AM5/10/22
to Christian Heimes, Python Dev
On Tue, May 10, 2022 at 9:34 AM Christian Heimes <chri...@python.org> wrote:
> We can easily take care of test_cppext and add the build step to
> Makefile. How does test_peg_generator depend on distutils? I don't see
> an import of distutils in the directory.
You need to look at the files in the Tools directory. That is where
> We can easily take care of test_cppext and add the build step to
> Makefile. How does test_peg_generator depend on distutils? I don't see
> an import of distutils in the directory.
c-analyzer and peg_generator live.
The peg_generator tests depend on Tools/peg_generator/pegen/build.py
which uses distutils to build
multiple extensions to test the grammar generator.
--
Jeremy Kloth
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/7WMU6N6XGHLSO2FFSTXEFZC6UDPMNOB3/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Victor Stinner
May 10, 2022, 12:05:45 PM5/10/22
to Christian Heimes, Python Dev
On Tue, May 10, 2022 at 5:33 PM Christian Heimes <chri...@python.org> wrote:
> We can easily take care of test_cppext and add the build step to
> Makefile.
What matters in test_cppext is that using the Python C API in C++ does
> We can easily take care of test_cppext and add the build step to
> Makefile.
not emit compiler warnings, so it uses -Werror. Since this test is
very new, I would prefer to not block the Python build just because of
that. Moreover, I'm not sure that I want to require a C++ compiler to
build CPython (written in C).
But yeah, something can be done in Makefile :-)
> How does test_peg_generator depend on distutils? I don't see
> an import of distutils in the directory.
extensions and then execute them.
> I would prefer to fix and remove all imports of distutils before we
> resort to renaming the package. Please give us time until alpha 1. There
> is no need to rush it *now*.
change as soon as possible (in the 3.12 dev cycle), to make sure that
we are aware of all issues, and have more time to fix them.
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/L6ECTYWEHYITDBFVMY4HTYCVFXUD6W3S/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Steve Dower
May 10, 2022, 12:29:02 PM5/10/22
to Christian Heimes, Victor Stinner, Python Dev
On 5/10/2022 4:33 PM, Christian Heimes wrote:
> On 10/05/2022 13.18, Victor Stinner wrote:
>> test_peg_generator and test_cppext build C extensions with distutils.
>> Until these tests are modified to use something else, we still need to
>> keep distutils. So I propose to rename it to _distutils to remove it
>> from the stdlib. Maybe these tests can use ensurepip to install
>> setuptools which provides distutils.
>>
>> There is also the c-analyzer tool which uses distutils to run a C
>> preprocessor.
>
> We can easily take care of test_cppext and add the build step to
> Makefile. How does test_peg_generator depend on distutils? I don't see
> an import of distutils in the directory.
>
> I would prefer to fix and remove all imports of distutils before we
> resort to renaming the package. Please give us time until alpha 1. There
> is no need to rush it *now*.
I agree. The internal tools that use it are all in our Tools directory,
> On 10/05/2022 13.18, Victor Stinner wrote:
>> test_peg_generator and test_cppext build C extensions with distutils.
>> Until these tests are modified to use something else, we still need to
>> keep distutils. So I propose to rename it to _distutils to remove it
>> from the stdlib. Maybe these tests can use ensurepip to install
>> setuptools which provides distutils.
>>
>> There is also the c-analyzer tool which uses distutils to run a C
>> preprocessor.
>
> We can easily take care of test_cppext and add the build step to
> Makefile. How does test_peg_generator depend on distutils? I don't see
> an import of distutils in the directory.
>
> I would prefer to fix and remove all imports of distutils before we
> resort to renaming the package. Please give us time until alpha 1. There
> is no need to rush it *now*.
so we can move distutils there and explicitly add an import path to
locate it. No need to keep it in the stdlib (Lib/) at all.
Migrating to Makefile builds is probably better long-term, but not as
important as moving distutils out from the stdlib so that setuptools can
rely on their copy being the "main" one.
Cheers,
Steve
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Guido van Rossum
May 10, 2022, 1:25:26 PM5/10/22
to Steve Dower, Christian Heimes, Victor Stinner, Python Dev
Shouldn't we wean our internal tools off this obsolete version of distutils too, rather than just move it around?
--Guido van Rossum (python.org/~guido)
Pronouns: he/him (why is my pronoun here?)
Brett Cannon
May 10, 2022, 2:47:39 PM5/10/22
to Victor Stinner, Python Dev
On Tue, May 10, 2022 at 12:59 AM Victor Stinner <vsti...@python.org> wrote:
Hi,
Two years ago, PEP 632 "Deprecate distutils module" was accepted: the
distutils package was deprecated in Python 3.10 and scheduled for
removal in Python 3.12. Questions.
* Is the Python ecosystem ready for the distutils removal? How many
projects are expected to be broken by this removal?
* Is setuptools a drop-in replacement of distutils for most cases? Are
there tools and documentation explaining how to replace distutils with
setuptools?
Change `import distutils` with `import setuptools`. 😉 But more thorough guidance can be found at https://setuptools.pypa.io/en/latest/deprecated/distutils-legacy.html?highlight=distutils .
* Is there a tool to migrate from setup.py (distutils) to
pyproject.toml (setuptools)? The dephell project can convert a
setup.py script to pyproject.toml using poetry as the build system.
You don't have to migrate a setuptools project to pyproject.toml to keep functioning. I would strongly encourage people at least specify a build-system table in pyproject.toml as it's easy to do, but pip doesn't require even this for setuptools projects:
```toml
[build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta"```
But the rest of the project's metadata can stay in setup.py or setup.cfg if they want. But there is experimental support to fully support PEP 621 and specifying metadata in pyproject.toml: https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html .
* Can we simply suggest installing setuptools to users who still use
"import distutils"? setuptools now provides the a "distutils" package.
If they are unwilling to update, yes.
* Should we keep distutils in Python stdlib a few more Python releases
if the removal causes too many practical issues?
I don't think so.
-Brett
Brett Cannon
May 10, 2022, 2:48:55 PM5/10/22
to Guido van Rossum, Christian Heimes, Victor Stinner, Python Dev
On Tue, May 10, 2022 at 10:23 AM Guido van Rossum <gu...@python.org> wrote:
Shouldn't we wean our internal tools off this obsolete version of distutils too, rather than just move it around?
I think so. We technically have a year to get this done, so if we can't do that then I'm afraid we have much bigger issues with the project's maintenance.
-Brett
Victor Stinner
May 10, 2022, 4:31:51 PM5/10/22
to Steve Dower, Christian Heimes, Python Dev
On Tue, May 10, 2022 at 6:16 PM Steve Dower <steve...@python.org> wrote:
> I agree. The internal tools that use it are all in our Tools directory,
> so we can move distutils there and explicitly add an import path to
> locate it. No need to keep it in the stdlib (Lib/) at all.
>
> Migrating to Makefile builds is probably better long-term, but not as
> important as moving distutils out from the stdlib so that setuptools can
> rely on their copy being the "main" one.
With my current PR, Lib/_distutils/ is not installed by "make
> I agree. The internal tools that use it are all in our Tools directory,
> so we can move distutils there and explicitly add an import path to
> locate it. No need to keep it in the stdlib (Lib/) at all.
>
> Migrating to Makefile builds is probably better long-term, but not as
> important as moving distutils out from the stdlib so that setuptools can
> rely on their copy being the "main" one.
install". Moving it to Tools/ would also work.
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/T43OZ4LWUT7QBENWPKIA3LFT52GWJV3Q/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Christopher Barker
May 10, 2022, 4:45:47 PM5/10/22
to Victor Stinner, Christian Heimes, Python Dev
Just a note:
The huge amount of work/progress that has gone into modernizing the building/packaging tools is great, but to be honest, C extensions have been somewhat neglected. e.g. I still need to use old fashioned setup.py to build some of my projects (C and Cython extensions)
This may not be relevant to this discussion, as I think setuptools does still support C extensions as well as it has for years -- All of my recent (last ten years) projects use it.
But it makes me nervous when I see suggestions to keep distutils in the stdlib: if the stdlib finds it still needs distutils, then that means that there may be others in the community that need it too.
So if you can't entirely drop distutils in the stdlib, I don't think we can count on the rest of the community being able to drop it as well.
Thanks,
-CHB
Christopher Barker, PhD (Chris)
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
Steve Dower
May 10, 2022, 4:51:39 PM5/10/22
to Victor Stinner, Christian Heimes, Python Dev
If we're not shipping it, we should just move it out of the way.
Otherwise, there are a few places in the various Windows packages that
will need it to be explicitly excluded. We should already have similar
exclusions for the test suite and a few other modules, but they're a bit
scattered around PC, PCbuild and Tools/msi. Easier to just remove it
from Lib.
Cheers,
Steve
On 5/10/2022 9:24 PM, Victor Stinner wrote:
> On Tue, May 10, 2022 at 6:16 PM Steve Dower <steve...@python.org> wrote:
>> I agree. The internal tools that use it are all in our Tools directory,
>> so we can move distutils there and explicitly add an import path to
>> locate it. No need to keep it in the stdlib (Lib/) at all.
>>
>> Migrating to Makefile builds is probably better long-term, but not as
>> important as moving distutils out from the stdlib so that setuptools can
>> rely on their copy being the "main" one.
>
> With my current PR, Lib/_distutils/ is not installed by "make
> install". Moving it to Tools/ would also work.
>
> Victor
Otherwise, there are a few places in the various Windows packages that
will need it to be explicitly excluded. We should already have similar
exclusions for the test suite and a few other modules, but they're a bit
scattered around PC, PCbuild and Tools/msi. Easier to just remove it
from Lib.
Cheers,
Steve
On 5/10/2022 9:24 PM, Victor Stinner wrote:
> On Tue, May 10, 2022 at 6:16 PM Steve Dower <steve...@python.org> wrote:
>> I agree. The internal tools that use it are all in our Tools directory,
>> so we can move distutils there and explicitly add an import path to
>> locate it. No need to keep it in the stdlib (Lib/) at all.
>>
>> Migrating to Makefile builds is probably better long-term, but not as
>> important as moving distutils out from the stdlib so that setuptools can
>> rely on their copy being the "main" one.
>
> With my current PR, Lib/_distutils/ is not installed by "make
> install". Moving it to Tools/ would also work.
>
> Victor
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/JG2BVL54C3RYXFFIX2TL7C53YPIWQHBQ/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Steve Dower
May 10, 2022, 4:56:53 PM5/10/22
to Christopher Barker, Victor Stinner, Christian Heimes, Python Dev
On 5/10/2022 9:38 PM, Christopher Barker wrote:
> So if you can't entirely drop distutils in the stdlib, I don't think we
> can count on the rest of the community being able to drop it as well.
The only reason distutils is used to build Python is for
> So if you can't entirely drop distutils in the stdlib, I don't think we
> can count on the rest of the community being able to drop it as well.
self-bootstrapping. setuptools would have to avoid any module that may
potentially be built as an extension module if it were to be used at
that stage of build, which is unfair (and this use is also likely to
become Makefile for Linux and macOS, just as it uses static build
scripts on Windows already).
The other uses are tests or tools. There's nothing wrong with
setuptools's functionality, merely the fact that it isn't part of the
CPython repository and we don't like relying on third-party repositories
for a CPython build/test.
Cheers,
Steve
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Victor Stinner
May 10, 2022, 5:06:13 PM5/10/22
to Guido van Rossum, Christian Heimes, Python Dev
On Tue, May 10, 2022 at 7:18 PM Guido van Rossum <gu...@python.org> wrote:
> Shouldn't we wean our internal tools off this obsolete version of distutils too, rather than just move it around?
Here is a first PR to build the test C++ extension with setuptools in
> Shouldn't we wean our internal tools off this obsolete version of distutils too, rather than just move it around?
test_cppext:
https://github.com/python/cpython/pull/92639
It implements what I proposed in the issue. It runs the test in a
venv. Python stdlib (ensurepip) includes a setuptools wheel package
and so is able to install it without adding any new external
dependency.
If this approach works, we can use the same for test_peg_generator.
Victor
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/S6UKSWFTJGPSTK2MZ4RIPVQJ7CKAJCQG/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Reply all
Reply to author
Forward
0 new messages