Run multiple make in parallel with appropriate cores settings for each of them.

[hongy...@gmail.com]

I noticed the BuildPackages.sh [1] script is used by GAP to build the packages supported by it. Because there are so many packages, there is another option in the script to support parallel make. The following code snippet is the relevant logic corresponding to this option implemented in this script:

```
PARALLEL=no
[...]
while [[ "$#" -ge 1 ]]; do
option="$1" ; shift
case "$option" in
[...]
--parallel) PARALLEL=yes; ;;
[...]
esac
done
[...]

if [ "x$PARALLEL" = "xyes" ] && [ "x$STRICT" = "xyes" ]; then
error "The options --strict and --parallel cannot be used simultaneously"
fi

if [ "x$PARALLEL" = "xyes" ]; then
export MAKEFLAGS="${MAKEFLAGS:--j3}"
fi;

[...]

BUILD_PID=$!
if [ "x$PARALLEL" = "xyes" ]; then
# If more than 4 background jobs are running, wait for one to finish (if
# <wait -n> is available) or for all to finish (if only <wait> is available)
if [[ $(jobs -r -p | wc -l) -gt 4 ]]; then
wait -n 2>&1 >/dev/null || wait
fi
else
# wait for this package to finish building
if ! wait $BUILD_PID && [[ $STRICT = yes ]]
then
exit 1
fi
fi;
```
As you can see, by default, the parallel make is disabled, and when the `--parallel` option is used, 3 cores will be assigned to each make process.

I want to enhance this logic and tried the following simple and crude modification [2]:

Add the following line:

NCORE=$(sudo dmidecode -t 4 | grep 'Core Enabled:' | awk '{a+=$NF}END{print a}')

And change the following two lines respectively:

Change
export MAKEFLAGS="${MAKEFLAGS:--j3}"

to

export MAKEFLAGS="${MAKEFLAGS:--j$NCORE}"


Change

if [[ $(jobs -r -p | wc -l) -gt 4 ]]; then

to

if [[ $(jobs -r -p | wc -l) -gt $((NCORE +1)) ]]; then

But the above method has the drawbacks and possibly more feasible alternatives, as commented here [3]:

```
Note that by telling both make and jobs to use that many cores, you are in fact instructing the system to use ~NCORE^2 cores.

One could possibly rearrange things so that a single make job server is shared across all jobs, then it's be "only" 2*NCORE jobs.
```

But I don't know how to implement this or other more powerful and feasible solutions to solve the problems discussed here. Any hints will be highly appreciated.

[1] https://github.com/gap-system/gap/blob/master/bin/BuildPackages.sh
[2] https://github.com/gap-system/gap/pull/4879/commits/f0375b2be81900687e678e2c61a28f6643dffb97
[3] https://github.com/gap-system/gap/pull/4879#discussion_r857318524

Regards,
HZ

[Richard Harnden]

On 26/04/2022 03:50, hongy...@gmail.com wrote:
> I noticed the BuildPackages.sh [1] script is used by GAP to build the packages supported by it. Because there are so many packages, there is another option in the script to support parallel make. The following code snippet is the relevant logic corresponding to this option implemented in this script:
>
> ```
> PARALLEL=no
> [...]
> while [[ "$#" -ge 1 ]]; do
> option="$1" ; shift
> case "$option" in
> [...]
> --parallel) PARALLEL=yes; ;;
> [...]
> esac
> done
> [...]
>
> if [ "x$PARALLEL" = "xyes" ] && [ "x$STRICT" = "xyes" ]; then
> error "The options --strict and --parallel cannot be used simultaneously"
> fi
>
> if [ "x$PARALLEL" = "xyes" ]; then
> export MAKEFLAGS="${MAKEFLAGS:--j3}"
> fi;
>
> [...]

This [...] desends into a subdir and starts the build in the background.

>
> BUILD_PID=$!
> if [ "x$PARALLEL" = "xyes" ]; then
> # If more than 4 background jobs are running, wait for one to finish (if
> # <wait -n> is available) or for all to finish (if only <wait> is available)
> if [[ $(jobs -r -p | wc -l) -gt 4 ]]; then
> wait -n 2>&1 >/dev/null || wait
> fi
> else
> # wait for this package to finish building
> if ! wait $BUILD_PID && [[ $STRICT = yes ]]
> then
> exit 1
> fi
> fi;

So: either wait for each background job in finish if PARALLEL != yes, or
let 4 jobs run at once.

> ```
> As you can see, by default, the parallel make is disabled, and when the `--parallel` option is used, 3 cores will be assigned to each make process.
>
> I want to enhance this logic and tried the following simple and crude modification [2]:

You make everything way too complicated.

I would say that if you want to start 4 build jobs at a time, then you
should reduce -j3 ; not try to increace it. If you do anything at all.

I would just leave it alone. You are trying to add complexity for zero
benefit.

[Kees Nuyt]

On Tue, 26 Apr 2022 12:13:41 +0100, Richard Harnden
<richard...@gmail.com> wrote:

> You make everything way too complicated.
>
> I would say that if you want to start 4 build jobs at a time, then you
> should reduce -j3 ; not try to increace it. If you do anything at all.
>
> I would just leave it alone. You are trying to add complexity for zero
> benefit.

+1
--
Kees Nuyt