Incremental build with "make" is slow

[Kwankyu Lee]

Hi,

Executing "make" with no changes takes 

...

Sage build/upgrade complete!
real 0m8.703s
user 0m4.959s
sys 0m2.084s

as of sage 10.7, on my machine. It takes less than a minute.

With sage 10.8.beta7 and  no changes, "make" 

[sagelib-10.8.beta7] [spkg-install]   + meson setup --reconfigure /Users/kwankyu/GitHub/sage-dev /Users/kwankyu/GitHub/sage-dev/build/sage-distro -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --default-library=static -DSAGE_LOCAL=/Users/kwankyu/GitHub/sage-dev/local --native-file=/Users/kwankyu/GitHub/sage-dev/build/sage-distro/meson-python-native-file.ini

...

[sagelib-10.8.beta7] [spkg-install] Successfully installed sagemath-10.8b7
[sagelib-10.8.beta7] [spkg-install] [InstallKernelSpec] Removing existing kernelspec in /Users/kwankyu/Library/Jupyter/kernels/sagemath
[sagelib-10.8.beta7] [spkg-install] [InstallKernelSpec] Installed kernelspec sagemath in /Users/kwankyu/Library/Jupyter/kernels/sagemath
[sagelib-10.8.beta7] Moving package files from temporary location /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.13/var/tmp/sage/build/sagelib-10.8.beta7/inst to /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.13
[sagelib-10.8.beta7] Deleting build directory /Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.13/var/tmp/sage/build/sagelib-10.8.beta7
[sagelib-10.8.beta7] Finished installing sagelib-10.8.beta7
[sagelib-10.8.beta7] real 2m3.801s user 2m0.759s sys 0m2.060s
Sage build/upgrade complete!
real 2m4.234s user 2m1.066s sys 0m2.134s

takes more than 2 minutes. Hence with trivial changes to sagelib, rebuilding always takes at least that amount of time. This is painful.

Is this best possible with meson build?

[Dima Pasechnik]

It's true that some rebuilds (which need full regeneration of ninja's build file)
are at the moment painfully slow, due to rather careless specification of dependencies between Sage docs (it's "everything depends on everything" (?) which is a rather poor way to put it).
Rebuilds which don't touch sage-distro are quite fast, in fact, and mostly automatic, just restarting Sage.

It's an urgent TODO item, yes.

[Tobia...@gmx.de]

As Dima said, no `make` is needed at all if you only change files in sagelib (doesn't matter if they are python or cython files). Just restart sage and your changes will automatically be reflected.

[Kwankyu Lee]

On Friday, October 17, 2025 at 1:14:20 PM UTC+9 Tobia...@gmx.de wrote:

... Just restart sage and your changes will automatically be reflected.

Wow, that is excellent! Thanks. Then we may get rid of "sage -b" and "sage -br".

[Travis Scrimshaw]

I am a very strong -1 on this. These are extremely useful for when you have to rebuild cython files but don't need to go through the entire "make" process.

Best,

Travis

[David Roe]

Tobias included Cython files in addition to Python files among those which would be automatically rebuilt when you restart Sage. Does that address your concern Travis?

David

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sage-devel/565333e1-395a-44f0-b582-93982013dfa7n%40googlegroups.com.

[Travis Scrimshaw]

That wasn't the case last time I checked (it was only Python files), so it's possible things have changed. However, right now I am trying to upgrade to the latest beta but my build keeps hanging (I was doing an incremental from beta0; I'm doing a make distclean now), so I can't test it.

Best,

Travis

[Dima Pasechnik]

On October 17, 2025 9:39:39 AM CDT, Travis Scrimshaw <tcsc...@gmail.com> wrote:
>I am a *very strong* -1 on this. These are extremely useful for when you

>have to rebuild cython files but don't need to go through the entire "make"
>process.

But you just don't need these any more - an equivalent of "sage -br" is happening in the background, if needed, when you start Sage
(built in the default, "editable" mode)

HTH
Dima

[Kwankyu Lee]

On Saturday, October 18, 2025 at 12:06:01 AM UTC+9 dim...@gmail.com wrote:

But you just don't need these any more - an equivalent of "sage -br" is happening in the background, if needed, when you start Sage
(built in the default, "editable" mode)

By the way, when was this introduced? Are they only Travis and I who were not aware of this change?

This seems to be another instance that should have been announced or advertised here, if not.

Release tours

https://github.com/sagemath/sage/wiki/Release-Tours

would be another place to advertise the change. But the venue is not maintained currently, mainly due to lack of attention...

[Travis Scrimshaw]

On October 17, 2025 9:39:39 AM CDT, Travis Scrimshaw <tcsc...@gmail.com> wrote:
>I am a *very strong* -1 on this. These are extremely useful for when you
>have to rebuild cython files but don't need to go through the entire "make"
>process.

But you just don't need these any more - an equivalent of "sage -br" is happening in the background, if needed, when you start Sage
(built in the default, "editable" mode)

I have verified that this does work, but it does so silently. It is also very dangerous because it can easily look that Sage is taking a long time to load because, say, you touched element.pxd trivially. While Python files are very fast, this is not generally true for Cython files, and I think it would be better to not have this by default and have more control for when these compile.

Best,
Travis

[Kwankyu Lee]

When you start sage first time after built, it takes much more time until you see the banner. After the first run, it takes less time to load.

This seems to be related with the change. 

[Dima Pasechnik]

On Fri, Oct 17, 2025 at 4:01 PM Travis Scrimshaw <tcsc...@gmail.com> wrote:

On October 17, 2025 9:39:39 AM CDT, Travis Scrimshaw <tcsc...@gmail.com> wrote:
>I am a *very strong* -1 on this. These are extremely useful for when you
>have to rebuild cython files but don't need to go through the entire "make"
>process.

But you just don't need these any more - an equivalent of "sage -br" is happening in the background, if needed, when you start Sage
(built in the default, "editable" mode)

I have verified that this does work, but it does so silently. It is also very dangerous because it can easily look that Sage is taking a long time to load because, say, you touched element.pxd trivially.

I don't see what's "dangerous" here. If you touched element.pxd then, by right, anything dependent on it must be rebuilt

one way or another. Anyhow,  you can (as documented, see src/doc/en/installation/source.rst - or the corresponding place in 

the html docs) do

 

export MESONPY_EDITABLE_VERBOSE=1 

Then when you start ./sage, you will see if any building is taking place. E.g.

$ touch src/sage/libs/gap/element.pyx

$ MESONPY_EDITABLE_VERBOSE=1 ./sage
meson-python: building sagemath: /usr/bin/ninja
[3/3] Linking target src/sage/libs/gap/element.cpython-313-x86_64-linux-gnu.so
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 10.8.beta7, Release Date: 2025-10-16              │
│ Using Python 3.13.5. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage:

--------------------

as it progresses towards giving you Sage prompt, you'll see [1/3], [2/3], until it's done [3/3]

(in this case, just 3 targets need a rebuild)

 

While Python files are very fast, this is not generally true for Cython files, and I think it would be better to not have this by default and have more control for when these compile.

In the above way you don't see a silent delay, you see what's going on.

HTH

Dima

 

Best,
Travis

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/sage-devel/751e1b84-f40a-4a39-bbde-5bf7c9ce818fn%40googlegroups.com.

[Nils Bruin]

On Friday, 17 October 2025 at 16:14:41 UTC-7 dim...@gmail.com wrote:

On Fri, Oct 17, 2025 at 4:01 PM Travis Scrimshaw <tcsc...@gmail.com> wrote:

I have verified that this does work, but it does so silently. It is also very dangerous because it can easily look that Sage is taking a long time to load because, say, you touched element.pxd trivially.

I don't see what's "dangerous" here. If you touched element.pxd then, by right, anything dependent on it must be rebuilt

one way or another. Anyhow,  you can (as documented, see src/doc/en/installation/source.rst - or the corresponding place in 

the html docs) do

 

export MESONPY_EDITABLE_VERBOSE=1 

Then when you start ./sage, you will see if any building is taking place.

This is certainly surprising compared to how sage used to work: it would start up normally, pick up changes from .py files and run with whatever the latest compile of pyx files was. I definitely agree that the new way sounds more consistent in most cases, but https://xkcd.com/1172/ comes to mind. You have to let the community know when you make changes like that! Even if it's a change for the better on the whole, people will be *unpleasantly* surprised when it happens without them asking for it. People could be running code where timestamps on pyx files get updated arbitrarily, so they'd prefer the old behaviour (because they knew it wasn't indicative of inconsistent state). 

I would definitely prefer to have the verbose on by default so that at least I get an indication that the system is doing *something*.

[Dima Pasechnik]

"sage -b" has been doing silly things for years. "make build" was the only reliable way to deal with updated code.

and, well, arbitrary timestamps on pyx files, or any source code, sounds like a high calibre  footgun to me.

Truth is that we are extremely short on human resources, saddled with ~400 packages in sage-distro without a reliable way to maintain their consistency, dozens of broken optional packages (e.g. look at cbc, cvxpy, their dependencies and consumers - these have been broken for at least a year), etc.
Proposals for reducing ~400 packages to something much more manageable have been defeated by people who mostly don't participate in maintenance of sage-distro.

[Oscar Benjamin]

Has anyone looked at the possibility of using spin for sagemath development?

https://github.com/scientific-python/spin

That is what python-flint uses after switching to meson-python and is
also what numpy, scipy and other libraries using meson-python use for
development.

With spin the idea is that you run e.g.

spin test
spin ipython
spin docs

and then it builds the library (incrementally) in ./build, "installs"
it to ./build-install and then runs whatever command with that
directory added to PYTHONPATH.

The meson-python editable setup that it sounds like sagemath now uses
is available as `spin install` but I don't use that because I prefer
running spin commands to trigger a rebuild explicitly.

[Michael Orlitzky]

On 2025-10-18 14:26:34, Oscar Benjamin wrote:
> The meson-python editable setup that it sounds like sagemath now uses
> is available as `spin install` but I don't use that because I prefer
> running spin commands to trigger a rebuild explicitly.

We switched to editable installs a long time ago (way before meson)
because people kept starting sage without running "sage -b", and then
were confused that their changes didn't take effect. If you're willing
to forego that ability, a non-editable sage build already works this
way.

The meson commands take a few days to get used to, but they're much
more flexible than the spin counterparts. There are thousands of
projects that use meson, and only a few that use spin. If you're going
to learn a new set of commands either way, why not learn the more
useful set?

A non-editable workflow[0] is basically...

$ meson setup --prefix="${HOME}/.local" build
$ meson install -C build
<do whatever>
$ meson install -C build

This does incremental builds, only when you ask for them, and then
installs the result to your user's PYTHONPATH so that no magic
commands are needed. To use it, you just run `python` or `ipython`.
Knowledge completely transferrable. You don't even need pip!


[0] Sage overrides one of the defaults, so you also need to pass
--python.install-env=prefix when configuring sage. But any other
meson-python project should work this way out-of-the-box. Also
note that "build" needs to be changed for sage due to the distro
files living there.

[Travis Scrimshaw]

Silent by default is still silent. I've changed element.pxd only to revert my change (and require no changes to cython otherwise), so I wanted to forgo basically rebuilding all of the cython files until I had to. If I just then saw that Sage was hanging while trying to start, I would think my build is completely broken somehow, we'd spend forever trying to debug it, only to realize it would be working as intended. This is the same for this meson setup build that is taking forever with no visible output for over 10 minutes (as per the other thread).

There is a strong case to be made for Python files automatically being reloaded. This is how Python works, both in the language design and interpreter design. Cython is completely different being a (not JIT) compiled language. Moreover with it being (default) silent despite the Cython compiler having output is really bad. Imagine if your favorite C compiler was silent by default.

Best,

Travis

[Michael Orlitzky]

On 2025-10-18 07:58:27, Travis Scrimshaw wrote:
>
> There is a strong case to be made for Python files automatically being
> reloaded. This is how Python works, both in the language design and
> interpreter design. Cython is completely different being a (not JIT)
> compiled language. Moreover with it being (default) silent despite the
> Cython compiler having output is really bad. Imagine if your favorite C
> compiler was silent by default.

Both gcc and clang are silent :)

(But I agree with you.)

[Travis Scrimshaw]

> There is a strong case to be made for Python files automatically being
> reloaded. This is how Python works, both in the language design and
> interpreter design. Cython is completely different being a (not JIT)
> compiled language. Moreover with it being (default) silent despite the
> Cython compiler having output is really bad. Imagine if your favorite C
> compiler was silent by default.

Both gcc and clang are silent :)

Okay, perhaps I should be more precise here (it's also been awhile since I used one, so my memory might be wrong). On any single command, it doesn't say anything (it would be difficult for it to say what it was doing), but when run from a makefile, I thought it did output what command was being run? (Okay, technically that isn't a C compiler either...but the act of compiling a C program.)

Best,

Travis

[Michael Orlitzky]

On 2025-10-18 08:43:13, Travis Scrimshaw wrote:
> Okay, perhaps I should be more precise here (it's also been awhile since I
> used one, so my memory might be wrong). On any single command, it doesn't
> say anything (it would be difficult for it to say what it was doing), but
> when run from a makefile, I thought it did output what command was being
> run? (Okay, technically that isn't a C compiler either...but the act of
> compiling a C program.)

"make" will print the commands it's running, but I was not seriously
trying to nitpick, your point was clear. When you run GCC, you expect
it to be compiling a C program; when you run Sage, you expect to see a
Sage banner.

My earlier message was not defending the behavior, only pointing out
that spin doesn't solve it. If you are willing to forego the editable
install, then meson alone works great. If not, the problem lies one
level below spin.

Regardless, while there's something to be said for going with the flow
of the python ecosystem, it sounds like we should at least make the
rebuilds verbose. (And this sounds easy.)

[Dima Pasechnik]

On October 18, 2025 9:58:27 AM CDT, Travis Scrimshaw <tcsc...@gmail.com> wrote:
>Silent by default is still silent. I've changed element.pxd only to revert
>my change (and require no changes to cython otherwise), so I wanted to
>forgo basically rebuilding all of the cython files until I had to.

no sane timestamp-based build system will let you do this, and for a good reason.
I've never heard about "ignore dependencies rebuild".

You could spend some time working on the Sage-distro build system, where any file change triggers a coffee-break, i.e. the whole bootstrap+configure+make run. :-)

>If I
>just then saw that Sage was hanging while trying to start, I would think my
>build is completely broken somehow, we'd spend forever trying to debug it,
>only to realize it would be working as intended. This is the same for this
>meson setup build that is taking forever with no visible output for over 10
>minutes (as per the other thread).

I explained how to make it visible, and this is documented. And yes, these long delays are fixable, it just needs more work.

It's not in any way unique. So we basically turned the whole sagelib into a JIT-compiled system, and similar JIT- compiled systems, e.g. anything Julia-based (for instance Oscar) are similar in spirit (and startup time could be like this too..)
Or if you use VSCode or a similar JIT system to write LaTeX, you'll also see delays in redisplaying pdf, too...

>
>There is a strong case to be made for Python files automatically being
>reloaded. This is how Python works, both in the language design and
>interpreter design. Cython is completely different being a (not JIT)
>compiled language. Moreover with it being (default) silent despite the
>Cython compiler having output is really bad. Imagine if your favorite C
>compiler was silent by default.

$ gcc helloworld.c
$

It IS silent, sorry.

Sure, we can make sagelib rebuilds noisier by default if there is sufficient demand.

Dima

[Travis Scrimshaw]

On Sunday, October 19, 2025 at 1:41:41 AM UTC+9 Michael Orlitzky wrote:

On 2025-10-18 08:43:13, Travis Scrimshaw wrote:
> Okay, perhaps I should be more precise here (it's also been awhile since I
> used one, so my memory might be wrong). On any single command, it doesn't
> say anything (it would be difficult for it to say what it was doing), but
> when run from a makefile, I thought it did output what command was being
> run? (Okay, technically that isn't a C compiler either...but the act of
> compiling a C program.)

"make" will print the commands it's running, but I was not seriously
trying to nitpick, your point was clear. When you run GCC, you expect
it to be compiling a C program; when you run Sage, you expect to see a
Sage banner.

My earlier message was not defending the behavior, only pointing out
that spin doesn't solve it. If you are willing to forego the editable
install, then meson alone works great. If not, the problem lies one
level below spin.

I know, but I think it is important to make sure the point can't be defeated by technicalities. :)

[Tobia...@gmx.de]

I completely understand that change is hard, and that the new workflow came as a surprise. I should have communicated this earlier and in more detail — that’s on me.

To clarify, the behavior under the new build system is standard for meson‑python editable installs: when the library is installed in editable mode, changes in Python or Cython files are picked up simply by restarting, without needing a full rebuild (https://mesonbuild.com/meson-python/how-to-guides/editable-installs.html). The same approach is used by NumPy, SciPy, and others. 

In my experience, it generally works quite nice and rebuilds for the usual changes to one or two Cython modules are often picked-up without any noticable delay. Personally, I don't mind that it's silently recompiling in the background, but can understand that personal preferences differ there. We can certainly set the toggle to 'loud', but that would need more testing (eg some doctests might fail because they directly check the output of sage).

I suggest to give it a bit of time and see how it feels in practice. Once people have actually used it for a while, we’ll have a clearer sense of what’s genuinely problematic versus what just takes getting used to, and we can revisit adjustments then.

[Tobia...@gmx.de]

On Saturday, October 18, 2025 at 9:26:52 PM UTC+8 oscar.j....@gmail.com wrote:

Has anyone looked at the possibility of using spin for sagemath development?

https://github.com/scientific-python/spin

 Thanks for suggestion! Pixi was another option that was mentioned previously in this context. My plan was to wait for uv to support running tasks, and then compare the options and make a proposal for what to 'offically' use in sage.

[Kwankyu Lee]

"sage -b" would be still useful if it works like "sage" (build changed python and cython files fast) but does not start a sage session. 

Currently "sage -b" works like "make" and is slow.

[Dima Pasechnik]

while (appropriately scripted)

$ echo quit | ./sage

does the job, it should be possible to avoid even starting Sage and
achieve the same effect.
I asked how exactly this can be done here:
https://github.com/mesonbuild/meson-python/issues/801

Dima

[Dima Pasechnik]

It seems it's just

ninja -C $SAGE_ROOT/build/sage-distro

which does the job 

(the path to that directory can be relative)

So we can make `sage -b` to do just this.

Dima

[Dima Pasechnik]

And, by the way, MESONPY_EDITABLE_VERBOSE=1 can be made default by passing

--config-settings=editable-verbose=true to pip

Cf.  https://mesonbuild.com/meson-python/how-to-guides/editable-installs.html#verbose-mode 

I won't mind this becoming default for the editable sagelib install.