[cython/cython] [BUG] distutils used in cythonize (fails in 3.12) (Issue #5751)
Jon Crall
Describe the bug
The cythonize command imports distutils at the top of its script, which is no longer available in Python 3.12.
Code to reproduce the behaviour:
Simply running:
cythonize --help
With Python 3.12 results in
Traceback (most recent call last):
File "/home/joncrall/.pyenv/versions/3.12.0/envs/pyenv3.12.0/bin/cythonize", line 5, in <module>
from Cython.Build.Cythonize import main
File "/home/joncrall/.pyenv/versions/3.12.0/envs/pyenv3.12.0/lib/python3.12/site-packages/Cython/Build/Cythonize.py", line 6, in <module>
from distutils.core import setup
ModuleNotFoundError: No module named 'distutils'
Expected behaviour
distutils is deprecated, so I'm not sure what should be done.
OS
Linux
Python version
3.12
Cython version
3.0.3
Additional context
I'm working on adding 3.12 support to line-profiler in this MR: pyutils/line_profiler#246
I'm attempting to do some standalone debugging by simply calling the cythonize command:
# Standalone compile instructions for developers
# Assuming the cwd is the repo root.
cythonize --annotate --inplace \
./line_profiler/_line_profiler.pyx \
./line_profiler/timers.c \
./line_profiler/unset_trace.c
But obviously because the script can't even import itself it errors immediately.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
scoder
The setuptools docs suggest to change the imports.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
scoder
Note that there are still several places in the code base where we import from distutils. PR welcome to replace them.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Matus Valo
PR welcome to replace them.
There is no need to replace them. We need to ensure that setuptools is installed. setuptools provides distutils compatible API including backward compatible imports:
(cython312) matus@MacBook-Pro-2240:~/dev/cython$ pip freeze
-e git+https://github.com/matusvalo/cython.git@3e4f970e0cf712febd3b713094079b40d2252f6c#egg=Cython
setuptools==68.2.2
(cython312) matus@MacBook-Pro-2240:~/dev/cython$ python -V
Python 3.12.0
(cython312) matus@MacBook-Pro-2240:~/dev/cython$ cythonize --help
usage: cythonize [-h] [-X NAME=VALUE,...] [-E NAME=VALUE,...] [-s NAME=VALUE] [-2] [-3] [--3str] [-+] [-a] [--annotate-fullc] [-x PATTERN] [-b] [-i] [-j N] [-f] [-q] [--lenient] [-k] [--no-docstrings] [-M] [sources ...]
positional arguments:
sources
options:
-h, --help show this help message and exit
-X NAME=VALUE,..., --directive NAME=VALUE,...
set a compiler directive
-E NAME=VALUE,..., --compile-time-env NAME=VALUE,...
set a compile time environment variable
-s NAME=VALUE, --option NAME=VALUE
set a cythonize option
-2 use Python 2 syntax mode by default
-3 use Python 3 syntax mode by default
--3str use Python 3 syntax mode by default
-+, --cplus Compile as C++ rather than C
-a, --annotate Produce a colorized HTML version of the source.
--annotate-fullc Produce a colorized HTML version of the source which includes entire generated C/C++-code.
-x PATTERN, --exclude PATTERN
exclude certain file patterns from the compilation
-b, --build build extension modules using distutils
-i, --inplace build extension modules in place using distutils (implies -b)
-j N, --parallel N run builds in N parallel jobs (default: 6)
-f, --force force recompilation
-q, --quiet be less verbose during compilation
--lenient increase Python compatibility by ignoring some compile time errors
-k, --keep-going compile as much as possible, ignore compilation failures
--no-docstrings strip docstrings
-M, --depfile produce depfiles for the sources
Environment variables:
CYTHON_FORCE_REGEN: if set to 1, forces cythonize to regenerate the output files regardless
of modification times and changes.
Environment variables accepted by setuptools are supported to configure the C compiler and build:
https://setuptools.pypa.io/en/latest/userguide/ext_modules.html#compiler-and-linker-options
Hence, we need only to add setuptools as dependency in cython - see linked PR.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Harmen Stoppels
See my comment in #5753. "Just" adding a depedency on setuptools is not a solution.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Matus Valo
"Just" adding a depedency on setuptools is not a solution.
You mentioned that
Setuptools uses this distutils-precedence.pth "trick" to hijack sys.path. That only runs on startup if setuptools is installed as a system site-package.
It doesn't run if setuptools is located through PYTHONPATH.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Harmen Stoppels
The issue is with package managers like Nix, Spack, maybe Guix and others that have one prefix per package. They rely on PYTHONPATH in general
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Eisuke Kawashima
setuptools>=66.0 seems required.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.