Cython Failed to Build Against No-GIL Python 3.13

[Yuze Li]

Hi, I was testing on Python 3.13 with it's experimental `--disable-gil` feature that removed the GIL and tried to build Cython from source against it. But it shows error:

----------------

$ ../cpython/python -m pip install .
Defaulting to user installation because normal site-packages is not writeable
Processing /home/lyuze/workspace/cython
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: Cython
  Building wheel for Cython (pyproject.toml) ... error
  error: subprocess-exited-with-error
  × Building wheel for Cython (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [352 lines of output]
      Unable to find pgen, not compiling formal grammar.

[ ... ]

      creating build/temp.linux-x86_64-cpython-313/home/lyuze/workspace/cython/Cython
      creating build/temp.linux-x86_64-cpython-313/home/lyuze/workspace/cython/Cython/Compiler
      gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -I/home/lyuze/workspace/cpython/Include -I/home/lyuze/workspace/cpython -c /home/lyuze/workspace/cython/Cython/Compiler/Parsing.c -o build/temp.linux-x86_64-cpython-313/home/lyuze/workspace/cython/Cython/Compiler/Parsing.o
      /home/lyuze/workspace/cython/Cython/Compiler/Parsing.c:1148:1: warning: ‘Py_UNICODE’ is deprecated [-Wdeprecated-declarations]
       1148 | static CYTHON_INLINE size_t __Pyx_Py_UNICODE_strlen(const Py_UNICODE *u)
            | ^~~~~~
      /home/lyuze/workspace/cython/Cython/Compiler/Parsing.c: In function ‘__Pyx_Py_UNICODE_strlen’:
      /home/lyuze/workspace/cython/Cython/Compiler/Parsing.c:1150:5: warning: ‘Py_UNICODE’ is deprecated [-Wdeprecated-declarations]
       1150 |     const Py_UNICODE *u_end = u;
            |     ^~~~~
      /home/lyuze/workspace/cython/Cython/Compiler/Parsing.c: In function ‘__pyx_f_6Cython_8Compiler_7Parsing_p_dict_or_set_maker’:
      /home/lyuze/workspace/cython/Cython/Compiler/Parsing.c:31739:9: warning: ‘Py_OptimizeFlag’ is deprecated [-Wdeprecated-declarations]
      31739 |         if (unlikely(__pyx_assertions_enabled())) {
            |         ^~
      In file included from /home/lyuze/workspace/cpython/Include/Python.h:62,
                       from /home/lyuze/workspace/cython/Cython/Compiler/Parsing.c:31:
      /home/lyuze/workspace/cpython/Include/cpython/pydebug.h:13:37: note: declared here
         13 | Py_DEPRECATED(3.12) PyAPI_DATA(int) Py_OptimizeFlag;
            |                                     ^~~~~~~~~~~~~~~
      /home/lyuze/workspace/cython/Cython/Compiler/Parsing.c: In function ‘__Pyx_ParseOptionalKeywords’:
      /home/lyuze/workspace/cython/Cython/Compiler/Parsing.c:86928:25: error: ‘CYTHON_METH_FASTCALL’ undeclared (first use in this function); did you mean ‘METH_FASTCALL’?
      86928 |     int kwds_is_tuple = CYTHON_METH_FASTCALL && likely(PyTuple_Check(kwds));
            |                         ^~~~~~~~~~~~~~~~~~~~
            |                         METH_FASTCALL
      /home/lyuze/workspace/cython/Cython/Compiler/Parsing.c:86928:25: note: each undeclared identifier is reported only once for each function it appears in
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
 
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for Cython
Failed to build Cython
ERROR: Could not build wheels for Cython, which is required to install pyproject.toml-based projects

----------------

I have no problem building Cython against Python without the `--disable-gil` switch. It seems that Cython hasn't released the version for NO_GIL which are using different APIs with normal Python? 

[da-woods]

The original nogil fork based on Python 3.9 was reported working at one point (but was never something that we "officially" supported).

I think the main problem with the Python 3.13 version is that it's detecting the nogil fork and applying all the 3.9 adaptations to it. It would probably work better without that, if it just used the plain Python 3.13 adaptations.

Right now it's untested and unsupported. I think it's something where we'd take bug fixes but not bug reports.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/cf3dbd14-8f0a-4036-90e7-8c3769ed844cn%40googlegroups.com.

[Yuze Li]

Do you mean the Python 3.9 nogil version of this? I tried to build Cython against this 3.9 no_gil but also failed from `CYTHON_METH_FASTCALL`. Probably not the correct Cython branch I guess (I was using master). I need to test a successful Cython build against a no_gil Python for some preliminary proofing. Hope to get your follow up.

[da-woods]

I do mean that version. Allegedly it's used to build scikit learn.

I suspect the 0.29.x branch is functional and master isn't/never was, since CYTHON_METH_FASTCALL isn't in 0.29.x.

To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/af89ee5b-115a-46cb-beba-ec0de2317a9an%40googlegroups.com.

[Stefan Behnel]

Yuze Li schrieb am 29.11.23 um 21:10:

> Hi, I was testing on Python 3.13 with it's experimental `--disable-gil`
> feature that removed the GIL and tried to build Cython from source against
> it. But it shows error:
>
> ----------------

> /home/lyuze/workspace/cython/Cython/Compiler/Parsing.c:86928:25:
> error: ‘CYTHON_METH_FASTCALL’ undeclared (first use in this function); did
> you mean ‘METH_FASTCALL’?
> 86928 | int kwds_is_tuple = CYTHON_METH_FASTCALL &&
> likely(PyTuple_Check(kwds));
> | ^~~~~~~~~~~~~~~~~~~~
> | METH_FASTCALL
> /home/lyuze/workspace/cython/Cython/Compiler/Parsing.c:86928:25:
> note: each undeclared identifier is reported only once for each function it
> appears in

> ----------------

I pushed a proposed fix to the master branch:

https://github.com/cython/cython/commit/a4a0f1e1a2c848181e5474e093d96173cc566988

I couldn't validate it, so please give it a try if you can and report back.

BTW, is a nogil-CPython available for the Github setup-python action?

Stefan

[Yuze Li]

Hi Stepan,

I still couldn't install Cython or either the no_gil CPython 3.9, or 3.13 with your latest commit. Same `CYTHON_METH_FASTCALL undeclare` issue occurs. I simply did `<path-to-nogil-python> -m pip install .` under Cython directory. 

Regarding setup-python action, I'm not sure about that. I found the GitHub actions to build wheels for nogil CPython 3.9, but didn't find the ones for official CPython 3.13 nogil. 

Yuze

[Yuze Li]

I then tried `make` with the nogil Python, seems the macro declaration problem is gone. But has compile problem, most of which regard to:

```

/home/lyuze/workspace/cython/Cython/Compiler/Code.c: At top level:
/home/lyuze/workspace/cython/Cython/Compiler/Code.c:2337:80: error: unknown type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
 2337 | static CYTHON_INLINE PyObject *__Pyx_PyVectorcall_FastCallDict(PyObject *func, __pyx_vectorcallfunc vc, PyObject *const *args, size_t nargs, PyObject *kw);
      |                                                                                ^~~~~~~~~~~~~~~~~~~~
      |                                                                                vectorcallfunc

```

Yuze

On Monday, December 4, 2023 at 6:32:14 AM UTC-5 Stefan Behnel wrote:

[Stefan Behnel]

Yuze Li schrieb am 04.12.23 um 16:23:

> I then tried `make` with the nogil Python, seems the macro declaration
> problem is gone. But has compile problem, most of which regard to:
> ```
> /home/lyuze/workspace/cython/Cython/Compiler/Code.c: At top level:
> /home/lyuze/workspace/cython/Cython/Compiler/Code.c:2337:80: error: unknown
> type name ‘__pyx_vectorcallfunc’; did you mean ‘vectorcallfunc’?
> 2337 | static CYTHON_INLINE PyObject
> *__Pyx_PyVectorcall_FastCallDict(PyObject *func, __pyx_vectorcallfunc vc,
> PyObject *const *args, size_t nargs, PyObject *kw);
> |
> ^~~~~~~~~~~~~~~~~~~~
> |
> vectorcallfunc
> ```

Thanks. I pushed a few more fixes in

https://github.com/cython/cython/commit/e25e647954e06847a50393768b856863bb9d303d

https://github.com/cython/cython/commit/914cad5112c2042a26fc14489b4a768603f828bc

Seems worth trying again.

We should see that we get a nogil-CPython CI job up to get an idea of what
works and what doesn't.

Stefan

[Yuze Li]

Thanks, your latest commit works for a successful local `make`. And I can build a module using Cython now. But failed when doing `<path-to-python> -m pip install .` I guess that's somewhat not supported for building wheel? 

```

$ ~/workspace/cpython/python -m pip install .

[...]

File "/tmp/pip-build-env-04iufxmt/overlay/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
          ~~~~~~~~~~~~~~~~^^^^^
        File "/tmp/pip-build-env-04iufxmt/overlay/lib/python3.13/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
          ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
        File "/tmp/pip-build-env-04iufxmt/overlay/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
          ~~~~~~~~~~~^^
        File "/tmp/pip-build-env-04iufxmt/overlay/lib/python3.13/site-packages/wheel/bdist_wheel.py", line 405, in run
          impl_tag, abi_tag, plat_tag = self.get_tag()
                                        ~~~~~~~~~~~~^^
        File "/tmp/pip-build-env-04iufxmt/overlay/lib/python3.13/site-packages/wheel/bdist_wheel.py", line 355, in get_tag
          tag in supported_tags
      AssertionError: would build wheel with unsupported tag ('cp313', 'cp313t', 'linux_x86_64')

      [end of output]
 
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for Cython
Failed to build Cython
ERROR: Could not build wheels for Cython, which is required to install pyproject.toml-based projects

WARNING: There was an error checking the latest version of pip.

```

Another problem is, when I try to install NumPy from source with the Cython I just built, it shows:

```

$ rm -rf build; ~/workspace/cpython/python -m pip install .

      [...]

      numpy/random/_pcg64.cpython-313t-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c: In function ‘__Pyx_ParseOptionalKeywords’:
      numpy/random/_pcg64.cpython-313t-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:10774:25: error: ‘CYTHON_METH_FASTCALL’ undeclared (first use in this function); did you mean ‘METH_FASTCALL’?
      10774 |     int kwds_is_tuple = CYTHON_METH_FASTCALL && likely(PyTuple_Check(kwds));
            |                         ^~~~~~~~~~~~~~~~~~~~
            |                         METH_FASTCALL
      numpy/random/_pcg64.cpython-313t-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:10774:25: note: each undeclared identifier is reported only once for each function it appears in
      numpy/random/_pcg64.cpython-313t-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c: In function ‘__Pyx_CheckKeywordStrings’:
      numpy/random/_pcg64.cpython-313t-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:11463:9: error: ‘CYTHON_METH_FASTCALL’ undeclared (first use in this function); did you mean ‘METH_FASTCALL’?
      11463 |     if (CYTHON_METH_FASTCALL && likely(PyTuple_Check(kw))) {
            |         ^~~~~~~~~~~~~~~~~~~~
            |         METH_FASTCALL

      [...]

      [470/504] Compiling C object numpy/_core/lib_simd.dispatch.h_AVX2.a.p/meson-generated__simd.dispatch.c.o
      [471/504] Compiling C object numpy/_core/lib_simd.dispatch.h_FMA3__AVX2.a.p/meson-generated__simd.dispatch.c.o
      [472/504] Compiling C object numpy/_core/libloops_autovec.dispatch.h_AVX2.a.p/meson-generated_loops_autovec.dispatch.c.o
      [473/504] Compiling C object numpy/_core/lib_simd.dispatch.h_AVX512_SKX.a.p/meson-generated__simd.dispatch.c.o
      [474/504] Compiling C object numpy/_core/lib_simd.dispatch.h_AVX512F.a.p/meson-generated__simd.dispatch.c.o
      ninja: build stopped: subcommand failed.
      [end of output]

```

I'm confused about this issue since I already made sure NumPy recognizes the correct Cython I just built (~/.local/bin/cython): 

#!/home/lyuze/workspace/cpython/python

# -*- coding: utf-8 -*-

import re

import sys

sys.path.append('/home/lyuze/workspace/cython')

from Cython.Compiler.Main import setuptools_main

if __name__ == '__main__':

sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])

sys.exit(setuptools_main())

[Stefan Behnel]

Yuze Li schrieb am 05.12.23 um 16:17:

> File
> "/tmp/pip-build-env-04iufxmt/overlay/lib/python3.13/site-packages/wheel/bdist_wheel.py",
> line 355, in get_tag
> tag in supported_tags
> AssertionError: would build wheel with unsupported tag ('cp313',
> 'cp313t', 'linux_x86_64')
> [end of output]

Are you sure that you have the latest pip and wheel packages installed?

> Another problem is, when I try to install NumPy from source with the Cython
> I just built, it shows:
> ```
> $ rm -rf build; ~/workspace/cpython/python -m pip install .
> [...]
>
> numpy/random/_pcg64.cpython-313t-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:
> In function ‘__Pyx_ParseOptionalKeywords’:
>
> numpy/random/_pcg64.cpython-313t-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:10774:25:
> error: ‘CYTHON_METH_FASTCALL’ undeclared (first use in this function); did
> you mean ‘METH_FASTCALL’?
> 10774 | int kwds_is_tuple = CYTHON_METH_FASTCALL &&
> likely(PyTuple_Check(kwds));
> | ^~~~~~~~~~~~~~~~~~~~
> | METH_FASTCALL
>
> numpy/random/_pcg64.cpython-313t-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c

Could you delete that file (or all "*.pyx.c" files, probably) and try
again, to make sure it really got regenerated by Cython?

Stefan