How to build sagelib with multiple processes in parallel?

[Nils Bruin]

In the past I've build sagelib in parallel, with

export MAKE="make -j8"

make -j8 build

This seems to still work for the prerequisites, but for building sagelib everything slows to an agonizing one job at a time. Did something change here? I can't find anything in the documentation other than the export MAKE thing.

[dim...@gmail.com]

Are you on macOS?
It could be a macOS-specific recent change.

Dima

>
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/sage-devel/36b15c72-6aca-49a3-bf31-9edf31d4245bn%40googlegroups.com.

[Nils Bruin]

No, sorry. I should have included that this is on fedora (40 at the moment).

[Eric Gourgoulhon]

Hi,

I confirm the issue on Ubuntu 24.04: the build of sagelib for Sage 10.6.beta9 was performed on a single process despite the make -j8. This seems to be a new issue. I am currently building Sage 10.6.rc0 and will report on this.

Eric.

[Eric Gourgoulhon]

Le lundi 24 mars 2025 à 13:38:50 UTC+1, Eric Gourgoulhon a écrit :

I confirm the issue on Ubuntu 24.04: the build of sagelib for Sage 10.6.beta9 was performed on a single process despite the make -j8. This seems to be a new issue. I am currently building Sage 10.6.rc0 and will report on this.

On my computer (Ubuntu 24.04), the issue is gone with  Sage 10.6.rc0: after a make distclean, the command

MAKE="make -j16" make

resulted in a parallel build of all Sage, including sagelib.

Eric.

[Alexander Verner]

I can confirm that SageMath 10.6 RC 0 doesn't compile in parallel even if "-j8" is appended to the build command. Verified that with htop too.

понедельник, 24 марта 2025 г. в 15:41:03 UTC+2, Eric Gourgoulhon:

[Marc Culler]

Sage 10.6rc0 is building in parallel for me.

- Marc

[Marc Culler]

On Sunday, March 23, 2025 at 6:24:09 PM UTC-5 dima wrote:

Are you on macOS?
It could be a macOS-specific recent change.

I should have said that Sage-10.6rc0 is building in parallel on macOS for me.

- Marc

[John H Palmieri]

I'm a little puzzled about how this ever worked. By "this", I mean detecting the number of threads to use. The relevant script is src/bin/sage-env.

- if SAGE_NUM_THREADS and SAGE_NUM_THREADS_PARALLEL are both set, then those values are supposed to be detected, but the "case" statement doesn't use valid syntax, as far as I can tell: the current version only detects two digit numbers for those variables, it seems to me. We should make this change:

+shopt -s extglob
+
 # Handle parallel building/testing/...
 case "$SAGE_NUM_THREADS,$SAGE_NUM_THREADS_PARALLEL" in
-    [1-9][0-9]*,[1-9][0-9]*)
+    +([[:digit:]])*,+([[:digit:]])* )
         # Variables are set to positive values already,
         # sage-num-threads.py would just recompute them
         ;;
@@ -589,6 +595,8 @@ case "$SAGE_NUM_THREADS,$SAGE_NUM_THREADS_PARALLEL" in
         ;;
 esac
 
+shopt -u extglob
+
 # Multithreading in OpenBLAS does not seem to play well with Sage's attempts to
 # spawn new processes, see #26118. Apparently, OpenBLAS sets the thread
 # affinity and, e.g., parallel doctest jobs, remain on the same core.

- if either or both of those variables is not set, then the "-j N" argument to "make" is supposed to be used, detected by the script sage-num-threads.py. That thread is called in src/bin/sage-env like this:

        sage_num_threads_array=$(sage-num-threads.py 2>/dev/null || echo 1 2 1)

In particular, it discards all errors. If I remove "2>/dev/null", then I get this error:

  /Users/palmieri/Sage/TESTING/sage-10.6.rc0/src/bin/sage-env: line 586: sage-num-threads.py: command not found

So I think that maybe we should also make this change:

@@ -251,6 +251,9 @@ if [ -z "${SAGE_ORIG_PATH_SET}" ]; then
     SAGE_ORIG_PATH=$PATH && export SAGE_ORIG_PATH
     SAGE_ORIG_PATH_SET=True && export SAGE_ORIG_PATH_SET
 fi
+if [ -d "$SAGE_ROOT" ]; then
+    export PATH="$SAGE_ROOT/src/bin:$PATH"
+fi
 if [ -n "$SAGE_LOCAL" ]; then
     export PATH="$SAGE_LOCAL/bin:$PATH"
 fi
@@ -569,9 +572,11 @@ if [ "$TOUCH" = "" ]; then
     TOUCH="touch"  && export TOUCH
 fi
 

This should add src/bin to PATH, in such a way that it will be shadowed by local/bin and a few other directories, which might be the behavior we want, but maybe we only want to add this to the path in some circumstances?

All of this might depend on which shell you're using, so I'm not 100% sure of any of it. Can those of you who are having problems test (by removing "2>/dev/null" from line 583 in src/bin/sage-env) whether sage-num-threads.py is running correctly when you run "./sage --sh"? You could also add in some "echo" statements to see which branch of the "case" statement is running, if SAGE_NUM_THREADS and SAGE_NUM_THREADS_PARALLEL are both set.

--

John

[John H Palmieri]

I should also say that I don't have a lot of time in the next few weeks to work more on this, but I hope someone can investigate and produce a fix.

[Emmanuel Charpentier]

FWIW, upgrading 6.10.beta9 to 6.10.rc0 on Debian testins ran in parallel.

HTH,