Assertion failed: refcnt >= 0 when importing Cython module compiled with mingw32 for 32-bit Windows

[Michael Williamson]

Hello! I'm the maintainer for jq.py, and am trying (and failing!) to build wheels for cp314t for 32-bit Windows. Unfortunately, I don't have a Windows machine to test on, so I thought I'd post here before digging any deeper to see if anybody had any insight or thoughts.

I've reproduced the issue here: https://github.com/mwilliamson/cython-cp314t-mingw32

Essentially, defining a Cython module with a single variable in (x = 1) and compiling it with mingw32 causes an error when trying to import the module:

Assertion failed: refcnt >= 0, file C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\pythonx86-freethreaded.3.14.0\tools\include/refcount.h, line 151

This only seems to affect the free-threaded build i.e. cp314 works without issue.

Removing the compiler override (which presumably then defaults to msvc) resolves the issue, but isn't possible for jq.py itself (so far as I know -- would be happy to be corrected here!)

Defining and importing an empty module seems to work fine. From inspecting a diff of the generated C code between an empty module and a module with x = 1, it looks like the issue might be around immortal constants. If I compile with CYTHON_IMMORTAL_CONSTANTS set to 0, the error seems to disappear.

Since I don't have a Windows machine to hand, I've been testing this using cibuildwheel on GitHub Actions, which makes debugging trickier, so I thought seeing if I've done something obviously foolish first would be wise.

Any help would be appreciated!

Michael

[da-woods]

I don't think mingw is particularly supported as a compiler for Python extension modules so that doesn't help with understand anything.  The fact it works on MSVC suggests that it's a mingw configuration problem rather than a Cython problem specifically. My first guess would be that Py_ssize_t is defined wrongly.  If there are config errors with mingw then I'd be a little worried that this won't be the only bug you see.

We could potentially add a guard to disable CYTHON_IMMORTAL_CONSTANTS with 32 bit mingw. It'd be untested (because it's not a configuration we test) but if we know it's a problem then it's not a big deal to disable it.

clang-cl might be a better option if you can't use MSVC (depending on why you can't use MSVC...). It's a wrapper for clang, that goes to reasonable lengths to be binary compatible with MSVC. It's not currently supported by setuptools unfortunately though. But it might be worth exploring as the closest thing to a support alternative compiler as exists on Windows.

In the short-term though it sounds like you should disable CYTHON_IMMORTAL_CONSTANTS and not spend more time on it.

David

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/cython-users/24139783-aa95-4b18-a363-15b799204126n%40googlegroups.com.

[Michael Williamson]

Thanks for the information. Given it's not entirely clear what's going on and (as you say) this might be a symptom of other bugs, I'm tempted just to avoid building 32-bit Windows wheels entirely for the time being, and see if that actually causes any issues for anyone.

Thanks!

Michael

[da-woods]

You can use pypinfo to get download statistics.

If I do `pypinfo -l 200 "jq==1.10.0" file` then your most downloaded 32bit Windows wheels over the last month are:

| jq-1.10.0-cp310-cp310-win32.whl                                                                                     |              7 |
| jq-1.10.0-cp311-cp311-win32.whl                                                                                     |              6 |
| jq-1.10.0-cp313-cp313-win32.whl                                                                                     |              6 |
| jq-1.10.0-cp312-cp312-win32.whl                                                                                     |              6 |

with 6 or 7 downloads each. For comparison your top Window 64 download was:

| jq-1.10.0-cp38-cp38-win_amd64.whl                                                                                   |          2,311 |

(Not sure why, but your 64 bit Windows users don't seem to like updating Python).

And your most downloaded wheel overall is

| jq-1.10.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl                                                |        731,924 |

So you can probably safely drop 32 bit Windows.

To view this discussion visit https://groups.google.com/d/msgid/cython-users/8fb6c36b-a60b-4ef0-b017-e0f3b0c2f17bn%40googlegroups.com.

[Michael Williamson]

Thanks, both for digging out the information, and for making me aware of pypinfo.