Dealing with python version differences across cythonization/compilation/runtime

[Nils Bruin]

Hi everyone,

In sage there is some code that interacts with the internals of python's "atexit". Between Python 3.13 and 3.14 the internals of that have changed, probably in such a way that we can do away (eventually) with reaching into the internals. In the mean time, we need code that can deal with 3.13 and 3.14:

https://github.com/sagemath/sage/pull/41021

Ideally, we'd have some conditional compiles depending on the python version (via PY_HEX_VERSION) but we also need make decisions during runtime, currently via sys.version_info.

It seems like there could be a mismatch between the two. Are there any recommended methods to deal with possible version discrepancies between compilation and runtime?

Nils

[da-woods]

From Python 3.11 upwards you can use `Py_Version` (https://docs.python.org/3/c-api/apiabiversion.html#c.Py_Version) for fast C API access to the runtime version instead of `sys.version_info`.

In terms of `PY_VERSION_HEX` and the runtime version - the only question is "are you trying to use the stable ABI"?  (If you haven't deliberately chosen to use the Stable ABI then you aren't using it!)

* if not, then the first 4 bytes of PY_VERSION_HEX will always match the first two elements of sys.version_info so this should be fine. The compiled module will simply not work otherwise.
* if you are using the stable ABI then you really need to be doing runtime checks only. I don't think you are though given that you're looking at internal C arrays (according to your PR).

David

Nils --

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/cython-users/a45a67f2-a1fa-45b5-9e82-8c5f3da68598n%40googlegroups.com.

[Nils Bruin]

Thank you very much! That is already hugely helpful. Two questions:

 - do you mean "The first two bytes of PY_VERSION_HEX match ..."? From the documentation I understand that the two high bytes encode the major/minor version and the two low bytes encode micro/release_level/release_serial, so the first two entries of Py_Version (major/minor) should match the two high bytes of the 32 bit hex version.

- is it a feature of cython modules that they test compile-time major/minor against runtime major/minor? The python documentation does not explicitly mention that major/minor between compile time and runtime can be assumed to be the same. We'd be very happy if we can already be sure something else throws an error if runtime/compiletime major/minor don't agree.

(I see our "#if"s happen in a `cdef extern` block so that's just C compile time. No need to worry about cythonization time)

[da-woods]

Extensions built on different minor versions of Python are not compatible. So an extension built on Python 3.13 won't run on 3.14, and vice versa.

Extensions built on different micro versions are compatible (so 3.13.1 is compatible with 3.13.3, either way around).

This is mostly enforced by the name of the compiled extensions (e.g. "cpython-313-x86_64-linux-gnu" in the filename tells Python whether it's suitable) or the name of the wheel bundling the extension. So it's possible to trick Python by renaming things (in which case it either crash or fail to import for linking reasons). But normal users won't get the versions mixed up. And if they do, it's already broken - your changes aren't making it worse.

I don't think Cython adds any specific version checks because it would be very hard to write a "safe" check that actually works.

To view this discussion visit https://groups.google.com/d/msgid/cython-users/cfd328c2-8537-471a-a264-838c106780e1n%40googlegroups.com.

[Nils Bruin]

On Thursday, 23 October 2025 at 12:05:13 UTC-7 D Woods wrote:

And if they do, it's already broken - your changes aren't making it worse.

Thanks! that saves work. It makes one wonder why python is even bothering with a "Stable ABI" then. The between different minor versions, a new compile against the (much more stable) API would already be necessary!

 

[da-woods]

If you deliberately compile with the Stable ABI then you can get a single extension that's compatible across a wide range of Python versions. For example, we can build Cython so a single wheel works from Python 3.9 upwards.

There are lots of restrictions - your access into the internal atexit structures just wouldn't work for example.

But it's definitely worthwhile in some cases.

But it's not something that happens accidentally so don't worry about it in the PR you linked.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/cython-users/a91f7647-1f33-4ddd-806d-6d6119f57b59n%40googlegroups.com.

[衷琛鑫]

    // Dummy function for Python < 3.14 (never called)
     static PyObject* get_atexit_callbacks_list(PyObject *self) {
         PyErr_SetString(PyExc_RuntimeError, "Python < 3.14 has no atexit list");
         return NULL;
     }

Thank you for your answer. I would like to know if I runs this accidently,it will safely raise an error on Python level or go into segfault.

[Nils Bruin]

On Thursday, 23 October 2025 at 21:00:28 UTC-7 114278...@gmail.com wrote:

    // Dummy function for Python < 3.14 (never called)
     static PyObject* get_atexit_callbacks_list(PyObject *self) {
         PyErr_SetString(PyExc_RuntimeError, "Python < 3.14 has no atexit list");
         return NULL;
     }

Thank you for your answer. I would like to know if I runs this accidently,it will safely raise an error on Python level or go into segfault.

Perhaps a clarification: this is C declaration. Its cython header stub is:

atexit_callback_struct** get_atexit_callbacks_array(object module) except NULL

The need for this dummy function exposes a bit of a flaw in cython -- although that flaw might just be that it makes poking into python's internals harder: we end up with different code for <=3.13 and >=3.14, and the choice is made at compile-time (assuming linking the extension module to a different version is an error anyway). However, we end up with the branching happening at runtime anyway, since cython's compiler conditionals are deprecated (and cythonization time isn't compilation time). It's a bit of an edge case, but it is indicative of something that isn't quite supported by cython that one would sometimes like to do in C-extensions of Cpython: have cython source that compiles differently, depending on the version of python.

[Matthias Koeppe]

On Fri, Oct 24, 2025 at 10:09 AM Nils Bruin <bruin...@gmail.com> wrote:
> cythonization time isn't compilation time

They are the same time -- ever since the ancient practice of shipping
Cython-generated C files in the sdist fell out of favor.

--
Matthias Koeppe -- http://www.math.ucdavis.edu/~mkoeppe

[da-woods]

On 24/10/2025 23:32, 'Matthias Koeppe' via cython-users wrote:
> On Fri, Oct 24, 2025 at 10:09 AM Nils Bruin <bruin...@gmail.com> wrote:
>> cythonization time isn't compilation time
> They are the same time -- ever since the ancient practice of shipping
> Cython-generated C files in the sdist fell out of favor.

... and of course sagemath controls their own build process. So are in a
position to unilaterally make it the same time if it isn't already.

[Nils Bruin]

On Saturday, 25 October 2025 at 00:51:55 UTC-7 D Woods wrote:

... and of course sagemath controls their own build process. So are in a
position to unilaterally make it the same time if it isn't already. 

I think in practice, cython modules in sage indeed get translated to C and then immediately compiled. So assuming that cythonization time = compilation time for sage is probably fine. Is it nowadays mandated with cython that the two coincide? It didn't use to be and in principle it was supported (recommended even?) to distribute python extensions in the form of C files derived from cython. I think the rationale was that it saved cython from being a build-time dependency for the distributed module.

But more importantly, what would one gain from that assumption? From https://docs.cython.org/en/latest/src/userguide/migrating_to_cy30.html#deprecation-of-def-if  I understand that new uses of `IF/THEN/ELIF/ELSE` should be avoided and that the recommended way is now indeed to solve compile-time choices by verbatim C code and/or runtime choices.

The particular example we ran into here demands that some things are done compile-time (because of ABI incompatibilities), but then there is code that is really much more naturally written in cython. Translating that into verbatim C would be painful. So we end up with an unnatural-looking combination of compile times/runtime choices for something that can be resolved at compile-time. Maybe worth considering in https://github.com/cython/cython/issues/4310 (deprecation of conditional compilation in cython). I recall wrestling with cython conditional compilation at some point (also for version checking!) and I agree that the current (deprecated) design doesn't work very well  for that purpose. But since it comes up now again, perhaps there is a use-case in cython for having something close to `#if/then/else/endif`.

 

[Matthias Koeppe]

On Sat, Oct 25, 2025 at 3:17 PM Nils Bruin <bruin...@gmail.com> wrote:
> in principle it was supported (recommended even?) to distribute python extensions in the form of C files derived from cython.

That's correct, but the old recommendation was (finally) changed in
2024 (https://github.com/cython/cython/pull/6201) to match
contemporary practice.

> I think the rationale was that it saved cython from being a build-time dependency for the distributed module.

Also correct; but thanks to modern Python packaging practices (PEP
518), that's no longer a relevant concern.