Defining MAKE='make -j4' is recommended?

[Kwankyu Lee]

Hi,

According to our installation guide:

One useful setting for this variable when building Sage is MAKE='make -jNUM' to tell the make program to run NUM jobs in parallel when building. Note that some Sage packages may not support this variable.

and our main build scripts are sprinkled with $(MAKE) instead of invoking "make" directly. 

But as I understand the jobserver (https://www.gnu.org/software/make/manual/html_node/Job-Slots.html), only the top-level make command should get "-jNUM" flag and sub make commands should not get "-jNUM flag, for the jobserver to work across all make commands.

Hence we should revise the paragraph in the installation guide to not recommend  to define MAKE='make -jNUM', but to build sage by make -jNUM at the top-level.

Am I right?

Moreover I think SAGE_NUM_THREADS environment variable should be used if a sage package wants to run parallel jobs, instead of relying on MAKE='make -jNUM'.

Do you agree?

I am considering this issue since currently doc build does not do parallel jobs, and the above issue seems related.

Thank you for attention in advance.

Kwankyu

[Michael Orlitzky]

On 2024-11-13 01:12:25, Kwankyu Lee wrote:
> Hi,
>
> According to our installation guide:
>

> *One useful setting for this variable when building Sage is MAKE='make

> -jNUM' to tell the make program to run NUM jobs in parallel when building.

> Note that some Sage packages may not support this variable.*

>
> and our main build scripts are sprinkled with $(MAKE) instead of invoking
> "make" directly.
>
> But as I understand the jobserver
> (https://www.gnu.org/software/make/manual/html_node/Job-Slots.html), only
> the top-level make command should get "-jNUM" flag and sub make commands
> should not get "-jNUM flag, for the jobserver to work across all make
> commands.
>
> Hence we should revise the paragraph in the installation guide to not

> recommend to define *MAKE='make -jNUM', but to build sage by make -jNUM at
> the top-level.*
>
> *Am I right?*

Sub-invocations of $(MAKE) within a Makefile coordinate the number of
jobs by passing information around through the MAKEFLAGS variable. So
inside Sage's own Makefiles, either should work.

I think where the difference becomes important is in the
sage-the-distro packages, under build/pkgs. These are built with
scripts that run $MAKE, but not as part of another Makefile -- so I
doubt that the jobserver/number-of-jobs information gets relayed
unless you put it in the $MAKE variable yourself. (But if you are
thinking that this would ultimately start too many jobs, you are
probably right.)

> Moreover I think SAGE_NUM_THREADS environment variable should be used if a

> sage package wants to run parallel jobs, instead of relying on *MAKE='make
> -jNUM'.*

The doctest runner is capable of coordinating with the GNU Make
jobserver so that, for example, "make -j4 test" will start the correct
number of parallel test threads. This is akin to how "make -j4 test"
works in a typical autotools package; it will run up to four tests at
the same time. In short it is a way to pass sage-the-distro
information down to sage-the-library. Otherwise, users who are
familiar with "make -j" would also need to know about the sage-only
SAGE_NUM_THREADS variable, and set it redundantly.

[Kwankyu Lee]

Right. I think package build scripts should relay  the MAKEFLAGS variable 

so that the jobserver initiated by the top-level make run by a user controls

the number of jobs across the whole build process. 

Defining MAKE='make -jNUM' disables the jobserver, and hence seems to be a bad practice. 

 

> Moreover I think SAGE_NUM_THREADS environment variable should be used if a
> sage package wants to run parallel jobs, instead of relying on *MAKE='make
> -jNUM'.*

The doctest runner is capable of coordinating with the GNU Make
jobserver so that, for example, "make -j4 test" will start the correct
number of parallel test threads. This is akin to how "make -j4 test"
works in a typical autotools package; it will run up to four tests at
the same time. In short it is a way to pass sage-the-distro
information down to sage-the-library.

OK.

 

Otherwise, users who are
familiar with "make -j" would also need to know about the sage-only
SAGE_NUM_THREADS variable, and set it redundantly.

If defined, SAGE_NUM_THREADS may be used to overrule the jobserver.