Magic Cython compiling doesn't display errors

[Travis Scrimshaw]

It just does this:

tscrim@travis-apricot:~$ sage
Traceback (most recent call last):
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 345, in _rebuild
    subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=subprocess.DEVNULL, check=True)
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/ninja']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tscrim/sage/src/bin/sage-ipython", line 9, in <module>
    from sage.misc.banner import banner
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 311, in find_spec
    tree = self._rebuild()
           ^^^^^^^^^^^^^^^
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 347, in _rebuild
    raise ImportError(f're-building the {self._name} meson-python editable wheel package failed') from exc
ImportError: re-building the sagemath meson-python editable wheel package failed

Thus I cannot debug Cython errors unless I run "sage -b", which takes forever because the meson setup takes multiple minutes every time.

Two things we (specifically the people changing stuff in the build system) **really** need to do because this is unsustainable for Cython development in Sage:

1) Figure out what makes meson take so long to do its thing each time "sage -b" is run because this significantly slows many things down.

2a) Display the error message when cython builds fail so we can debug cython errors with the magic compiling. (I still *very* strongly believe we should give a message saying Cython code is recompiling verbosely on the files so it doesn't look like it is hanging. I also believe it is bad practice to make a non-jit-compiled language act like a jit-compiled language, but that is secondary.)

2b) Revert "sage -b" to actually just rebuilding the parts of Sage that should be rebuilt/recompiled; meson is not part of this AFAIK.

Travis

[Tobia...@gmx.de]

You can try the verbose mode by setting the env variable `MESONPY_EDITABLE_VERBOSE`, see https://mesonbuild.com/meson-python/how-to-guides/editable-installs.html#verbose-mode

Ad 1: it's the documentation build. This should be fixed by https://github.com/sagemath/sage/pull/41156 and https://github.com/sagemath/sage/pull/41162 but I currently don't have much time to work on this myself. So help eg in the form of PRs against the PR-branch is very welcome.

[Dima Pasechnik]

On February 4, 2026 8:57:21 AM CST, "'Tobia...@gmx.de' via sage-devel" <sage-...@googlegroups.com> wrote:

You can try the verbose mode by setting the env variable `MESONPY_EDITABLE_VERBOSE`, see https://mesonbuild.com/meson-python/how-to-guides/editable-installs.html#verbose-mode

Ad 1: it's the documentation build. This should be fixed by https://github.com/sagemath/sage/pull/41156 and https://github.com/sagemath/sage/pull/41162 but I currently don't have much time to work on this myself. So help eg in the form of PRs against the PR-branch is very welcome.

indeed, "export MESONPY_EDITABLE_VERBOSE=1" should make building more verbose

Also, after https://github.com/sagemath/sage/pull/41418 (should be in the latest betas)

./configure --disable-doc

will make rebuilding much quicker (the time was wasted otherwise in meson scanning documents in src/doc)

Merged last week https://github.com/sagemath/sage/pull/41174

tells "sage -b" to run menson directly

Regarding the build logs. Meson logs are in build/sage-distro/meson-logs/

and ninja logs are in  build/sage-distro/.ninja_log

 

On Wednesday, February 4, 2026 at 3:09:10 PM UTC+1 tcsc...@gmail.com wrote:

It just does this:

tscrim@travis-apricot:~$ sage
Traceback (most recent call last):
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 345, in _rebuild
    subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=subprocess.DEVNULL, check=True)
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/ninja']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tscrim/sage/src/bin/sage-ipython", line 9, in <module>
    from sage.misc.banner import banner
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 311, in find_spec
    tree = self._rebuild()
           ^^^^^^^^^^^^^^^
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 347, in _rebuild
    raise ImportError(f're-building the {self._name} meson-python editable wheel package failed') from exc
ImportError: re-building the sagemath meson-python editable wheel package failed

Thus I cannot debug Cython errors unless I run "sage -b", which takes forever because the meson setup takes multiple minutes every time.

Two things we (specifically the people changing stuff in the build system) **really** need to do because this is unsustainable for Cython development in Sage:

1) Figure out what makes meson take so long to do its thing each time "sage -b" is run because this significantly slows many things down.

2a) Display the error message when cython builds fail so we can debug cython errors with the magic compiling. (I still *very* strongly believe we should give a message saying Cython code is recompiling verbosely on the files so it doesn't look like it is hanging. I also believe it is bad practice to make a non-jit-compiled language act like a jit-compiled language, but that is secondary.)

this is  "export MESONPY_EDITABLE_VERBOSE=1"

2b) Revert "sage -b" to actually just rebuilding the parts of Sage that should be rebuilt/recompiled; meson is not part of this AFAIK.

after https://github.com/sagemath/sage/pull/41174

"sage -b" does

ninja -C $SAGE_ROOT/build/sage-distro 

(you can also run this explictly)

Tobias says it's not 100% fool-proof from meson point of view, but it's certainly fast and should very quickly rebuild 

what's needed.

Dima

Travis

[Travis Scrimshaw]

Okay, thanks. That helped a lot and made it so I could fix my compile-time issues.

That said, there is absolutely *no* way anybody except a sage build system expert could have found and known to do that. Also, I had no idea I should even look at meson/ninja logs for cython errors (much less where they would be located) either.

We need proper displaying of error messages when builds fail. The generic slop I posted is completely unhelpful (thankfully, I knew it was a Cython compile issue because of what I was changing, but imagine if I didn't; e.g., a bad branch merge). There is a balance to be had, but we need more general verbosity in starting Sage when its running (potentially time consuming) build steps.

Unfortunately I don't even have enough time to do the code I really need to do (even more so for should), nor do I have the expertise to help with the build system stuff.

That's good to hear at least.

Best,
Travis 

[Travis Scrimshaw]

Also, I just got bit by the hidden magic cython recompiling everything (and I don't know what I did to cause it either). Thankfully my computer is powerful and fast, it only took 10 minutes to do whatever it needed to do (which I don't know because I didn't enable the meson verbose on this computer).

Python files are okay since they are basically instantaneous, but Cython files are a different beast. IMO the proper thing to do would be to revert this feature (to what is effectively the status quo) and **first** have a proper discussion about whether we want it or not.

Travis

[Dima Pasechnik]

Sorry, what "feature" you want to revert? We're not going back to setuptools.

Besides, I doubt it was recompiling everything in sagelib, for this
you really need something
extraordinary like "make clean", or upgrading Cython, or I don't know.

But yes, if you don't have "--disable-doc" on, then indeed meson is
slow in processing the docs.
Someone needs to get their hands dirty with it and remove 15 years of
technical debt we accumulated
on building docs.

Dima

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/sage-devel/ebf355f3-7713-4e01-a773-0cfd6c63eedan%40googlegroups.com.

[Michael Orlitzky]

On 2026-02-05 17:09:04, Travis Scrimshaw wrote:
> Also, I just got bit by the hidden magic cython recompiling everything (and
> I don't know what I did to cause it either). Thankfully my computer is
> powerful and fast, it only took 10 minutes to do whatever it needed to do
> (which I don't know because I didn't enable the meson verbose on this
> computer).
>
> Python files are okay since they are basically instantaneous, but Cython
> files are a different beast. IMO the proper thing to do would be to revert
> this feature (to what is effectively the status quo) and **first** have a
> proper discussion about whether we want it or not.

Reverting only this aspect is not possible because it's (currently)
the intended behavior of meson-python with editable installs. There is
an open issue for it,

https://github.com/mesonbuild/meson-python/issues/820

and I fall on the side of "verbose by default" myself, so maybe it's
worthwhile making our voices heard there.

For an immediate solution I would recommend a non-editable install, but
there are some annoying issues with those as well. I took the time to
track down one of them, and opened an issue this morning:

https://github.com/mesonbuild/meson/issues/15531

In the process I found that by appending "-Dpython.bytecompile=-1" to
my "meson setup" command, I can get the best of both worlds: nothing
happens unless I ask it to rebuild, and when I do ask it to rebuild,
it is verbose and as fast as I'd expect.

[Nils Bruin]

On Thursday, 5 February 2026 at 17:09:05 UTC-8 tcsc...@gmail.com wrote:

Also, I just got bit by the hidden magic cython recompiling everything (and I don't know what I did to cause it either). Thankfully my computer is powerful and fast, it only took 10 minutes to do whatever it needed to do (which I don't know because I didn't enable the meson verbose on this computer).

Python files are okay since they are basically instantaneous, but Cython files are a different beast. IMO the proper thing to do would be to revert this feature (to what is effectively the status quo) and **first** have a proper discussion about whether we want it or not.

As far as I can see, a recompile (or at least a significantly longer start-up time) for sage is triggered whenever some packages have been upgraded (so a "dnf update" on a fedora system, for instance). I don't know which package updates trigger this. Is it a new python becoming available? It should really just stick to the python it's configured with and only change upon reconfiguration.

In a lot of cases it will not be necessary to do the recompile: even if a library got updated, it's probably still capable of resolving the same symbols as the previous version sage was compiled against.

It is a problem if people run "automatic" updates on their machines (which is somewhat doable nowadays): then it becomes totally unpredictable whether sage will recompile or not. Really, in that case part of the post-install cleanup should be "run/recompile sage". It looks like dnf can be configured to have "post-transaction-actions" but that's a big ask from users.

Perhaps we do need to recommend conda installs in stronger terms, to insulate the sage install more from system updates.

[Michael Orlitzky]

On 2026-02-06 10:04:16, Nils Bruin wrote:
> As far as I can see, a recompile (or at least a significantly longer
> start-up time) for sage is triggered whenever some packages have been
> upgraded (so a "dnf update" on a fedora system, for instance). I don't know
> which package updates trigger this. Is it a new python becoming available?
> It should really just stick to the python it's configured with and only
> change upon reconfiguration.

The ninja build rules are being generated with implicit dependencies
on the libraries being linked. For example,

build ... | /usr/lib/libglpk.so.40.3.1 /usr/lib/libgmp.so
LINK_ARGS = -Wl,--as-needed -Wl,--allow-shlib-undefined -shared
-fPIC -Wl,-O1 -Wl,--as-needed -O3 -pipe -mabi=lp64d -march=rv64gc
-Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing -flto
-Wl,--start-group -lglpk /usr/lib/libgmp.so -Wl,--end-group

So even though -lglpk is being used to link, the build system has
pre-resolved that and followed the symlink from libglpk.so to
libglpk.so.40.3.1. Unless I am mistaken, whenever libglpk.so.40.3.1
changes, this target will be rebuilt.

This bothers me because the library version (40.3.1) exists precisely
to indicate when the ABI/API has changed, and consumers need to
rebuild. On the other hand, we do these triggered rebuilds for an
entire linux distro, and the packages that get library versioning
correct are negligible. In the absence of hand-curated rebuild
information (what we do) rebuilding by default is the least bad
alternative.

> It is a problem if people run "automatic" updates on their machines (which
> is somewhat doable nowadays): then it becomes totally unpredictable whether
> sage will recompile or not. Really, in that case part of the post-install
> cleanup should be "run/recompile sage". It looks like dnf can be configured
> to have "post-transaction-actions" but that's a big ask from users.

For developers this is harder to avoid, but eventually end users will
be able to install a Sage distro package, and the distro can take care
of this. (The sage-on-gentoo overlay does automatic rebuilds, for
example.)

> Perhaps we do need to recommend conda installs in stronger terms, to
> insulate the sage install more from system updates.

A non-editable install would at least give you the option to run
"sage" and see if it crashes before wasting time on a rebuild.

[Tobia...@gmx.de]

I currently don't have the time to try it out, but I'm pretty sure that meson will not rebuild any cython file if you just update a dependency (that was found via pkg-config) since meson is not watching pc files (or the like) for changes. If the new version is incompatible, then I expect you get just a runtime error. Even in the case where a dependency is updated and a cython file is changed (so meson is run by meson-python with `--reconfigure`), the dependencies are normally cached and so the updated dependency will not be detected (there is an open issue to disable this caching: https://github.com/mesonbuild/meson/issues/6180).

In either case, I would recommend a full rebuild of sagemath after updating any dependencies.

[Nils Bruin]

On Sunday, 8 February 2026 at 11:17:21 UTC-8 Tobia...@gmx.de wrote:

[...] I would recommend a full rebuild of sagemath after updating any dependencies.

I think we need some different ways to install sage then and document them. Obviously, an ideal answer would be "just install the standard package for sage from your distribution" but then we need that sage is packaged for all/most distributions and that those packages are kept reasonably up-to-date. Neither is true at the moment. Even if that is going to be true in the future we still need an interim solution to bridge the time in between.

The kind of instructions I'm thinking about would be ones that are acceptable for a mildly competent sysadmin who is maintaining, say, a slew of VMs that people access for their general computing needs. They may well configure those to regularly run auto-updates  on. I'm not so sure they'd be willing to install custom post-update scripts.

It's probably somewhat acceptable for developers (like you and to some extent me)  to recompile sage after an update (note that you'll generally not know for sure that a *dependency* for sage has been updated because that's hidden in a list of 500 packages that are getting updated), but it's far outside the normal experience for outside software like magma, maple, matlab, google-chrome, etc: one installs them on a linux system and system updates don't end up touching the install (usually in "/opt" somewhere) and it keeps running fine.

Does a non-editable install from source fare better for this? (sure, an ABI-breaking upgrade would be a problem, but perhaps those don't happen so much?) Perhaps that needs to be documented a little more prominently then?

Conda seems to do a reasonable job (and even seems to provide binaries!). Can we have a guide written up for a sysadmin to install sage using conda in such a way that the users don't need to mess with conda themselves? I think that just involves a shim script that sets up the conda environment without relying on the user shell to do that. Perhaps some AppImage/flatpak/... builds? Those tend to have their own kinds of breakage but perhaps that can be avoided.

I understand the push to disentangle sage-the-library and sage-the-distribution, but sage-the-library still needs to be distributed and I suspect we'll need to do that ourselves at least until enough distributions agree to do it for us.