Magic Cython compiling doesn't display errors

[Travis Scrimshaw]

It just does this:

tscrim@travis-apricot:~$ sage
Traceback (most recent call last):
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 345, in _rebuild
    subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=subprocess.DEVNULL, check=True)
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/ninja']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tscrim/sage/src/bin/sage-ipython", line 9, in <module>
    from sage.misc.banner import banner
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 311, in find_spec
    tree = self._rebuild()
           ^^^^^^^^^^^^^^^
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 347, in _rebuild
    raise ImportError(f're-building the {self._name} meson-python editable wheel package failed') from exc
ImportError: re-building the sagemath meson-python editable wheel package failed

Thus I cannot debug Cython errors unless I run "sage -b", which takes forever because the meson setup takes multiple minutes every time.

Two things we (specifically the people changing stuff in the build system) **really** need to do because this is unsustainable for Cython development in Sage:

1) Figure out what makes meson take so long to do its thing each time "sage -b" is run because this significantly slows many things down.

2a) Display the error message when cython builds fail so we can debug cython errors with the magic compiling. (I still *very* strongly believe we should give a message saying Cython code is recompiling verbosely on the files so it doesn't look like it is hanging. I also believe it is bad practice to make a non-jit-compiled language act like a jit-compiled language, but that is secondary.)

2b) Revert "sage -b" to actually just rebuilding the parts of Sage that should be rebuilt/recompiled; meson is not part of this AFAIK.

Travis

[Tobia...@gmx.de]

You can try the verbose mode by setting the env variable `MESONPY_EDITABLE_VERBOSE`, see https://mesonbuild.com/meson-python/how-to-guides/editable-installs.html#verbose-mode

Ad 1: it's the documentation build. This should be fixed by https://github.com/sagemath/sage/pull/41156 and https://github.com/sagemath/sage/pull/41162 but I currently don't have much time to work on this myself. So help eg in the form of PRs against the PR-branch is very welcome.

[Dima Pasechnik]

On February 4, 2026 8:57:21 AM CST, "'Tobia...@gmx.de' via sage-devel" <sage-...@googlegroups.com> wrote:

You can try the verbose mode by setting the env variable `MESONPY_EDITABLE_VERBOSE`, see https://mesonbuild.com/meson-python/how-to-guides/editable-installs.html#verbose-mode

Ad 1: it's the documentation build. This should be fixed by https://github.com/sagemath/sage/pull/41156 and https://github.com/sagemath/sage/pull/41162 but I currently don't have much time to work on this myself. So help eg in the form of PRs against the PR-branch is very welcome.

indeed, "export MESONPY_EDITABLE_VERBOSE=1" should make building more verbose

Also, after https://github.com/sagemath/sage/pull/41418 (should be in the latest betas)

./configure --disable-doc

will make rebuilding much quicker (the time was wasted otherwise in meson scanning documents in src/doc)

Merged last week https://github.com/sagemath/sage/pull/41174

tells "sage -b" to run menson directly

Regarding the build logs. Meson logs are in build/sage-distro/meson-logs/

and ninja logs are in  build/sage-distro/.ninja_log

 

On Wednesday, February 4, 2026 at 3:09:10 PM UTC+1 tcsc...@gmail.com wrote:

It just does this:

tscrim@travis-apricot:~$ sage
Traceback (most recent call last):
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 345, in _rebuild
    subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=subprocess.DEVNULL, check=True)
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/ninja']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tscrim/sage/src/bin/sage-ipython", line 9, in <module>
    from sage.misc.banner import banner
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 311, in find_spec
    tree = self._rebuild()
           ^^^^^^^^^^^^^^^
  File "/home/tscrim/sage/local/var/lib/sage/venv-python3.12.5/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 347, in _rebuild
    raise ImportError(f're-building the {self._name} meson-python editable wheel package failed') from exc
ImportError: re-building the sagemath meson-python editable wheel package failed

Thus I cannot debug Cython errors unless I run "sage -b", which takes forever because the meson setup takes multiple minutes every time.

Two things we (specifically the people changing stuff in the build system) **really** need to do because this is unsustainable for Cython development in Sage:

1) Figure out what makes meson take so long to do its thing each time "sage -b" is run because this significantly slows many things down.

2a) Display the error message when cython builds fail so we can debug cython errors with the magic compiling. (I still *very* strongly believe we should give a message saying Cython code is recompiling verbosely on the files so it doesn't look like it is hanging. I also believe it is bad practice to make a non-jit-compiled language act like a jit-compiled language, but that is secondary.)

this is  "export MESONPY_EDITABLE_VERBOSE=1"

2b) Revert "sage -b" to actually just rebuilding the parts of Sage that should be rebuilt/recompiled; meson is not part of this AFAIK.

after https://github.com/sagemath/sage/pull/41174

"sage -b" does

ninja -C $SAGE_ROOT/build/sage-distro 

(you can also run this explictly)

Tobias says it's not 100% fool-proof from meson point of view, but it's certainly fast and should very quickly rebuild 

what's needed.

Dima

Travis

[Travis Scrimshaw]

Okay, thanks. That helped a lot and made it so I could fix my compile-time issues.

That said, there is absolutely *no* way anybody except a sage build system expert could have found and known to do that. Also, I had no idea I should even look at meson/ninja logs for cython errors (much less where they would be located) either.

We need proper displaying of error messages when builds fail. The generic slop I posted is completely unhelpful (thankfully, I knew it was a Cython compile issue because of what I was changing, but imagine if I didn't; e.g., a bad branch merge). There is a balance to be had, but we need more general verbosity in starting Sage when its running (potentially time consuming) build steps.

Unfortunately I don't even have enough time to do the code I really need to do (even more so for should), nor do I have the expertise to help with the build system stuff.

That's good to hear at least.

Best,
Travis 

[Travis Scrimshaw]

Also, I just got bit by the hidden magic cython recompiling everything (and I don't know what I did to cause it either). Thankfully my computer is powerful and fast, it only took 10 minutes to do whatever it needed to do (which I don't know because I didn't enable the meson verbose on this computer).

Python files are okay since they are basically instantaneous, but Cython files are a different beast. IMO the proper thing to do would be to revert this feature (to what is effectively the status quo) and **first** have a proper discussion about whether we want it or not.

Travis

[Dima Pasechnik]

Sorry, what "feature" you want to revert? We're not going back to setuptools.

Besides, I doubt it was recompiling everything in sagelib, for this
you really need something
extraordinary like "make clean", or upgrading Cython, or I don't know.

But yes, if you don't have "--disable-doc" on, then indeed meson is
slow in processing the docs.
Someone needs to get their hands dirty with it and remove 15 years of
technical debt we accumulated
on building docs.

Dima

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/sage-devel/ebf355f3-7713-4e01-a773-0cfd6c63eedan%40googlegroups.com.

[Michael Orlitzky]

On 2026-02-05 17:09:04, Travis Scrimshaw wrote:
> Also, I just got bit by the hidden magic cython recompiling everything (and
> I don't know what I did to cause it either). Thankfully my computer is
> powerful and fast, it only took 10 minutes to do whatever it needed to do
> (which I don't know because I didn't enable the meson verbose on this
> computer).
>
> Python files are okay since they are basically instantaneous, but Cython
> files are a different beast. IMO the proper thing to do would be to revert
> this feature (to what is effectively the status quo) and **first** have a
> proper discussion about whether we want it or not.

Reverting only this aspect is not possible because it's (currently)
the intended behavior of meson-python with editable installs. There is
an open issue for it,

https://github.com/mesonbuild/meson-python/issues/820

and I fall on the side of "verbose by default" myself, so maybe it's
worthwhile making our voices heard there.

For an immediate solution I would recommend a non-editable install, but
there are some annoying issues with those as well. I took the time to
track down one of them, and opened an issue this morning:

https://github.com/mesonbuild/meson/issues/15531

In the process I found that by appending "-Dpython.bytecompile=-1" to
my "meson setup" command, I can get the best of both worlds: nothing
happens unless I ask it to rebuild, and when I do ask it to rebuild,
it is verbose and as fast as I'd expect.

[Nils Bruin]

On Thursday, 5 February 2026 at 17:09:05 UTC-8 tcsc...@gmail.com wrote:

Also, I just got bit by the hidden magic cython recompiling everything (and I don't know what I did to cause it either). Thankfully my computer is powerful and fast, it only took 10 minutes to do whatever it needed to do (which I don't know because I didn't enable the meson verbose on this computer).

Python files are okay since they are basically instantaneous, but Cython files are a different beast. IMO the proper thing to do would be to revert this feature (to what is effectively the status quo) and **first** have a proper discussion about whether we want it or not.

As far as I can see, a recompile (or at least a significantly longer start-up time) for sage is triggered whenever some packages have been upgraded (so a "dnf update" on a fedora system, for instance). I don't know which package updates trigger this. Is it a new python becoming available? It should really just stick to the python it's configured with and only change upon reconfiguration.

In a lot of cases it will not be necessary to do the recompile: even if a library got updated, it's probably still capable of resolving the same symbols as the previous version sage was compiled against.

It is a problem if people run "automatic" updates on their machines (which is somewhat doable nowadays): then it becomes totally unpredictable whether sage will recompile or not. Really, in that case part of the post-install cleanup should be "run/recompile sage". It looks like dnf can be configured to have "post-transaction-actions" but that's a big ask from users.

Perhaps we do need to recommend conda installs in stronger terms, to insulate the sage install more from system updates.

[Michael Orlitzky]

On 2026-02-06 10:04:16, Nils Bruin wrote:
> As far as I can see, a recompile (or at least a significantly longer
> start-up time) for sage is triggered whenever some packages have been
> upgraded (so a "dnf update" on a fedora system, for instance). I don't know
> which package updates trigger this. Is it a new python becoming available?
> It should really just stick to the python it's configured with and only
> change upon reconfiguration.

The ninja build rules are being generated with implicit dependencies
on the libraries being linked. For example,

build ... | /usr/lib/libglpk.so.40.3.1 /usr/lib/libgmp.so
LINK_ARGS = -Wl,--as-needed -Wl,--allow-shlib-undefined -shared
-fPIC -Wl,-O1 -Wl,--as-needed -O3 -pipe -mabi=lp64d -march=rv64gc
-Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing -flto
-Wl,--start-group -lglpk /usr/lib/libgmp.so -Wl,--end-group

So even though -lglpk is being used to link, the build system has
pre-resolved that and followed the symlink from libglpk.so to
libglpk.so.40.3.1. Unless I am mistaken, whenever libglpk.so.40.3.1
changes, this target will be rebuilt.

This bothers me because the library version (40.3.1) exists precisely
to indicate when the ABI/API has changed, and consumers need to
rebuild. On the other hand, we do these triggered rebuilds for an
entire linux distro, and the packages that get library versioning
correct are negligible. In the absence of hand-curated rebuild
information (what we do) rebuilding by default is the least bad
alternative.

> It is a problem if people run "automatic" updates on their machines (which
> is somewhat doable nowadays): then it becomes totally unpredictable whether
> sage will recompile or not. Really, in that case part of the post-install
> cleanup should be "run/recompile sage". It looks like dnf can be configured
> to have "post-transaction-actions" but that's a big ask from users.

For developers this is harder to avoid, but eventually end users will
be able to install a Sage distro package, and the distro can take care
of this. (The sage-on-gentoo overlay does automatic rebuilds, for
example.)

> Perhaps we do need to recommend conda installs in stronger terms, to
> insulate the sage install more from system updates.

A non-editable install would at least give you the option to run
"sage" and see if it crashes before wasting time on a rebuild.

[Tobia...@gmx.de]

I currently don't have the time to try it out, but I'm pretty sure that meson will not rebuild any cython file if you just update a dependency (that was found via pkg-config) since meson is not watching pc files (or the like) for changes. If the new version is incompatible, then I expect you get just a runtime error. Even in the case where a dependency is updated and a cython file is changed (so meson is run by meson-python with `--reconfigure`), the dependencies are normally cached and so the updated dependency will not be detected (there is an open issue to disable this caching: https://github.com/mesonbuild/meson/issues/6180).

In either case, I would recommend a full rebuild of sagemath after updating any dependencies.

[Nils Bruin]

On Sunday, 8 February 2026 at 11:17:21 UTC-8 Tobia...@gmx.de wrote:

[...] I would recommend a full rebuild of sagemath after updating any dependencies.

I think we need some different ways to install sage then and document them. Obviously, an ideal answer would be "just install the standard package for sage from your distribution" but then we need that sage is packaged for all/most distributions and that those packages are kept reasonably up-to-date. Neither is true at the moment. Even if that is going to be true in the future we still need an interim solution to bridge the time in between.

The kind of instructions I'm thinking about would be ones that are acceptable for a mildly competent sysadmin who is maintaining, say, a slew of VMs that people access for their general computing needs. They may well configure those to regularly run auto-updates  on. I'm not so sure they'd be willing to install custom post-update scripts.

It's probably somewhat acceptable for developers (like you and to some extent me)  to recompile sage after an update (note that you'll generally not know for sure that a *dependency* for sage has been updated because that's hidden in a list of 500 packages that are getting updated), but it's far outside the normal experience for outside software like magma, maple, matlab, google-chrome, etc: one installs them on a linux system and system updates don't end up touching the install (usually in "/opt" somewhere) and it keeps running fine.

Does a non-editable install from source fare better for this? (sure, an ABI-breaking upgrade would be a problem, but perhaps those don't happen so much?) Perhaps that needs to be documented a little more prominently then?

Conda seems to do a reasonable job (and even seems to provide binaries!). Can we have a guide written up for a sysadmin to install sage using conda in such a way that the users don't need to mess with conda themselves? I think that just involves a shim script that sets up the conda environment without relying on the user shell to do that. Perhaps some AppImage/flatpak/... builds? Those tend to have their own kinds of breakage but perhaps that can be avoided.

I understand the push to disentangle sage-the-library and sage-the-distribution, but sage-the-library still needs to be distributed and I suspect we'll need to do that ourselves at least until enough distributions agree to do it for us.

[Tobia...@gmx.de]

I agree! The install instructions at https://doc.sagemath.org/html/en/installation/index.html also seem to be in the spirit that you recommend. What would you still change there? Note that sage-the-distro (or any other building from source) is only recommended for developers that want to work _on_ sage.

I hope in the not so far feature, we could just provide binary wheels (bundling all deps), so that a simple `pip install sagemath` works on most systems. It would also be nice to have up-to-date releases for ubuntu/debian/fedora/homebrew. So if anyone has connections to package maintainers there, please ask for their support (I'm happy to help in case of any meson build problems).

[Nils Bruin]

In my case, most of our computers run Fedora (for desktops) and Rocky/Redhat for servers. So on the guide I end up with "other distributions" and then "Conda". 

The problem there at the moment for me is that it punts to "conda" documentation for how to get conda set up -- and conda seems very insistent on messing with my ".bashrc". I don't want that! Particularly because if another user tries to run the sage install, they would not have those modifications to their shell environment. And also because I don't particularly want conda to mess with my startup scripts: I'm just setting up conda to run sage; not because I want to do anything else with it. In fact, since I'd probably just have a normal git-based install for development I would really want there to be *no* traces of the conda environment unless I specifically enter a subshell to have that environment. 

I'm not sure that "pip install" without shielding from system upgrades  will be reliable enough, but it's worth a try.

For conda it does seem the case that once the conda install is in place, just executing the ".../sage" script living in its subtree (without activating conda) does initialize enough of the environment to get sage running, so that might be an option if we can somehow find out if that is robust enough.

[Dima Pasechnik]

On February 9, 2026 4:05:44 PM CST, Nils Bruin <nbr...@sfu.ca> wrote:
>In my case, most of our computers run Fedora (for desktops) and
>Rocky/Redhat for servers. So on the guide I end up with "other
>distributions" and then "Conda".
>
>The problem there at the moment for me is that it punts to "conda"

I tried to make docs less conda-centric in
<https://github.com/sagemath/sage/pull/41527>

should be in the next beta

[Tobia...@gmx.de]

The problem there at the moment for me is that it punts to "conda" documentation for how to get conda set up -- and conda seems very insistent on messing with my ".bashrc". I don't want that! 

 You don't need this (and if I recall correctly the interactive installer asks if you want to init conda in the shell). The purpose of this option is to have the base conda env activated all the time (eg to make conda's python available on the shell), but it's perfectly fine to just have a `sage` env and activate that manually using `conda activate sage`. See https://github.com/conda-forge/miniforge?tab=readme-ov-file#automatic-activation-of-environments for more details.

[Nils Bruin]

I think the problem is in "conda init". That sets it up so that you can use "conda activate" in a shell and then can continue using that activated environment in the original shell. For that, the "conda" command must run as a function in the shell, not as an executable called from the shell (because one cannot scribble in the environment of the parent process -- at least not without horrible hackery).

I was not able to use miniforge without allowing it to do a "conda init". That's the part I don't like because it makes me expect that also other users who want to use the conda install of sage "the conda way" will have to have their ".bashrc" changed and I don't think that's a reasonable thing to ask just for people to run sage.

There is obviously a solution: have something like "conda-shell" which starts a *subshell* with the conda initialization inserted and/or an environment activated. We know which file conda wants "dot-loaded" into the shell, so that's the one to dot-load for the subshell initialization.

That's basically what "sage -sh" does. I do not know if that exists already. If it does then I think my complaint is that the sage documentation doesn't point to it.

It may also be that the "sage" script itself already sets up a strong enough environment that, once installed, conda-sage doesn't actually need conda activation in order to run properly (my limited tests point in that direction). In that case it would still be good if there were a "conda-shell" to install conda-sage with, so that one doesn't need to worry about "conda init" and sanitizing one's ".bashrc" afterwards.

[Tobia...@gmx.de]

On Tuesday, February 10, 2026 at 6:11:56 PM UTC+1 Nils Bruin wrote:

I think the problem is in "conda init". That sets it up so that you can use "conda activate" in a shell and then can continue using that activated environment in the original shell. For that, the "conda" command must run as a function in the shell, not as an executable called from the shell (because one cannot scribble in the environment of the parent process -- at least not without horrible hackery).

I was not able to use miniforge without allowing it to do a "conda init". That's the part I don't like because it makes me expect that also other users who want to use the conda install of sage "the conda way" will have to have their ".bashrc" changed and I don't think that's a reasonable thing to ask just for people to run sage.

That goes beyond my knowledge of the conda internals. I would suggest you just open an issue in the miniforge repo about the need for "conda init".

 

There is obviously a solution: have something like "conda-shell" which starts a *subshell* with the conda initialization inserted and/or an environment activated. We know which file conda wants "dot-loaded" into the shell, so that's the one to dot-load for the subshell initialization.

That's basically what "sage -sh" does. I do not know if that exists already. If it does then I think my complaint is that the sage documentation doesn't point to it.

It may also be that the "sage" script itself already sets up a strong enough environment that, once installed, conda-sage doesn't actually need conda activation in order to run properly (my limited tests point in that direction). In that case it would still be good if there were a "conda-shell" to install conda-sage with, so that one doesn't need to worry about "conda init" and sanitizing one's ".bashrc" afterwards.

Note that if you use conda then there is no need for sage-the-distro (and thus, in particular, for the sage shell env init). In particular, if you install sage via conda (conda create -n sage sage) then you only have a "sage" script/cli available if you first activate the conda env (conda activate sage). We could add to the documentation instructions on how to setup a custom subshell that automatically activates the sage conda env. I like that idea. Without trying, a bash script with ` #!/usr/bin/env conda run -n sage sage` should do the job.

[Nils Bruin]

Note that if you use conda then there is no need for sage-the-distro (and thus, in particular, for the sage shell env init). In particular, if you install sage via conda (conda create -n sage sage) then you only have a "sage" script/cli available if you first activate the conda env (conda activate sage). We could add to the documentation instructions on how to setup a custom subshell that automatically activates the sage conda env. I like that idea. Without trying, a bash script with ` #!/usr/bin/env conda run -n sage sage` should do the job.

That almost works! It starts sage but then it exists, so it looks like STDIN doesn't get properly connected (which means it's actively disconnected). What seems to work for now is to just run

 /opt/sage/miniforge3/envs/sage/bin/sage

(without letting conda do any environment setup!). That's just the standard sage startup script, so it sources sage-env. It could be more fragile than allowing conda to activate its environment properly, so it's probably preferable to involve "conda run" somehow. It would require a version of "conda run" that does connect STDIN properly, so really just something that doesn't mess with the standard streams.