Error installing package scipy-1.12.0 in Develop

[Kevin Youren]

Thank you for looking at my scipy error and putting the debug information in the build.

I just last night installed the new LTS Ubuntu version Ubuntu 24.04.1 LTS (Noble Numbat)

My machine is a Lenovo tower I5 with 12 CPUs showing, and 8G memory and 2G of swap.

In previous builds, the scipy installl would cause my machine to have all cpus at 100% usage, the cooling fans were fully and easily heard, then the screen would  effectively "freeze". Then I would just reboot the machine, swap to my build directory, and retype "make", and the build would restart and continue to the end of the install.  Which I will not do again.

The present failure is about the same place where my previous builds froze:

[scipy-1.12.0] [spkg-install] [1026/1610] Compiling C++ object scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/bsr.cxx.o
[scipy-1.12.0] [spkg-install] ninja: build stopped: subcommand failed.

Please note I am in Melbourne, Australia, which is a 10 hour difference from UTC.

thank you, Kevin Youren

[Eric Gourgoulhon]

Hi,

From the log file:

[spkg-install] g++: fatal error: Killed signal terminated program cc1plus

This points towards a maximum memory reached. You may decrease the number of threads in the parallel build, e.g. using make -j4 instead of make -j8.

Best regards,

Eric.

[Kevin Youren]

Thanks for replying, Eric

I did precisely what you suggested, tried make -j4, but no luck.

I even opened and read my paper book "GNU Make" by Stallman.

So, I tried "make" by itself, and it did slow it down a bit.

However scipy simply took over all 12 cpus, at lightning speed.

The advantage of the Tower was you don't even have to look at the
System Monitor, the fans make so much noise trying to cool down the
cpus.

When I restarted the machine, and just typed in "make", they still used
all 12 cpus, but finished OK without over-heating, and the build
finished OK.

I am thinking about splitting the the makefile into 3 pieces, and see
what that achieves.

regs,

Kev

[Dima Pasechnik]

scipy itself is not built with configure/make, it's built with meson, which invokes ninja (a faster replacement for make, in particular

it parallelizes the tasks much better - but in your case it goes overboard with it).

You can see it in your log:

[spkg-install] Found ninja-1.11.1 at /usr/bin/ninja
[spkg-install] + /usr/bin/ninja

now, if you invoke "ninja -h" at the shell prompt, it will print, among other things,

  -j N     run N jobs in parallel (0 means infinity) [default=6 on this system]

(on your system the "default" is likely much bigger)

There is no direct way to specify a non-default "-j" value, however it appears to be possible to do this via meson,
which invokes ninja via "meson compile". https://mesonbuild.com/Commands.html does not make it clear whether

"JOBS" or "NINJA_ARGS" are shell environment variables, or just placeholders for the actual values,

but you can try something like

export NINJA_ARGS="-j4"

make

If this does not work (I can see "export NINJA_ARGS .." on the net at various places, so it seems to work for some people), one can create a shell script, called ninja, which merely invokes

ninja -j 4

and place it first in your PATH, so that it picked up first, and serves as a replacement for ninja command.

HTH

Dima

[Oscar Benjamin]

On Thu, 12 Sept 2024 at 09:31, Dima Pasechnik <dim...@gmail.com> wrote:
>
> scipy itself is not built with configure/make, it's built with meson,

...

> There is no direct way to specify a non-default "-j" value, however it appears to be possible to do this via meson,
> which invokes ninja via "meson compile".

How exactly does sage invoke the SciPy build?

It should be possible to pass -j through to the backend:

https://meson-python.readthedocs.io/en/latest/how-to-guides/meson-args.html#how-to-guides-meson-args

If Sage's make is already using parallelisation you probably want -j1
when building SciPy.

--
Oscar

[Dima Pasechnik]

Sage merely invokes pip to build scipy from source.

Indeed, apart from ninja paralellism there is meson parallelism,

which is controlled by JOBS env. variable (?).

Note that JOBS should be just a number.

[Oscar Benjamin]

On Thu, 12 Sept 2024 at 14:25, Dima Pasechnik <dim...@gmail.com> wrote:
>
> Sage merely invokes pip to build scipy from source.
>
> Indeed, apart from ninja paralellism there is meson parallelism,
> which is controlled by JOBS env. variable (?).
> Note that JOBS should be just a number.

I think meson just delegates the parallelism to ninja but the idea is
that if you are using meson then you communicate with meson rather
than with the ninja "backend".

My point though was more that maybe the Sage build should invoke the
SciPy build without parallelism like:

pip install --config-settings=setup-args="-j1" scipy

If someone is already running make -j6 or whatever to build Sage then
they likely don't need/want meson to bring in extra parallelism on
top.

--
Oscar

[Kevin Youren]

Thanks Oscar,

If I can work out how to do the pip install, I will try the -j1.

The simple thing is, make by itself is supposed to NOT have multiple instances running.

At the moment, I run make, it uses multiple cpus, but not all 12. When it gets to scipy , it uses all 12 cpus, then "overheats" and the machine stops.

I run make again, it restarts from the beginning of scipy, finishes scipy successfully, still using all 12 cpus, and then carries on to the end of the build.

It would be nice to have more control of the build process.

regards, Kevin

[Dima Pasechnik]

On 13 September 2024 14:20:01 BST, Kevin Youren <kyo...@gmail.com> wrote:
>Thanks Oscar,
>
>If I can work out how to do the pip install, I will try the -j1.

Did you try my suggestion, with exporting variables?
These export statements can be put into
build/pkgs/scipy/spkg-install.in

Oscar's hack can also be done by appropriate changes to the same file.

[Kevin Youren]

Dima,

thanks for the hint,

but

export NINJA_ARGS="-j4"
make

didn't work. It still 'froze' .

Restarting 

using

export NINJA_ARGS="-j4"
make

is working using the export and the make, but the scipy steps still had the 12 cpus at 100% together with the fan noise.

When scipy finished , the machine has gone  quiet and is currently in the documentation steps:

[sagemath_doc_html-none] [spkg-install] sage --docbuild --no-pdf-links reference/spkg inventory, etc.

regs, Kev

[Dima Pasechnik]

On Sat, Sep 14, 2024 at 7:00 AM Kevin Youren <kyo...@gmail.com> wrote:
>
> Dima,
>
> thanks for the hint,
>
> but
>
> export NINJA_ARGS="-j4"
> make
>
> didn't work. It still 'froze' .

Could you try -j1 rather than -j4 ?
And also

export JOBS=1

>
>
> Restarting
>
> using
> export NINJA_ARGS="-j4"
> make
>
> is working using the export and the make, but the scipy steps still had the 12 cpus at 100% together with the fan noise.

With the excessively many parallel job as the reason for this,
it's sort of expected.

Machine is running many jobs in parallel; one is crashing; the rest of
the jobs complete.
Restarting, you don't need to redo complete jobs, and so fewer jobs
are started in the 2nd run.
As far as I understand, scipy build gets started from the beginning in
the 2nd run.
Thus it's some other jobs that are overlapping with scipy in the 1st
run, and don't overlap
in the 2nd run.

> --
> You received this message because you are subscribed to the Google Groups "sage-support" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-support...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-support/b1af99d0-cac3-4f40-a80c-ee795275147an%40googlegroups.com.

[Kevin Youren]

Thanks for your suggestions, but the run still stops in the scipy steps.

# Sep 15 - no luck, stopped at 10:01
export NINJA_ARGS="-j1"
export JOBS=1

make

This time slightly earlier.

[scipy-1.12.0] [spkg-install] [1001/1610] Compiling C object scipy/io/matlab/_streams.cpython-312-x86_64-linux-gnu.so.p/meson-generated__streams.c.o
[scipy-1.12.0] [spkg-install] [1002/1610] Linking target scipy/io/matlab/_streams.cpython-312-x86_64-linux-gnu.so

During the week, I will do the install on a laptop, and see what happens.

regs, Kev

[Dima Pasechnik]

That's unfortunate, and it might be a bug in the meson/meson-python/ninja versions installed on your machine. (that's a curse of LTS versions, they often remain with old buggy versions).

You can check if using instead versions vendored by Sage would work.
Run

./configure --with-system-meson=no --with-system-ninja=no && make

[Kevin Youren]

Dima,

well done and thank you!

I started the rebuild very soon after your suggestion. The build is doing the html doc stage, well passed the scipy.

I think I did 5 or 6 builds that failed.

There was no freeze.

Thank you very much,

regards, Kevin

[Dima Pasechnik]

Curiously, this might be a regression in meson 1.3.2 (or perhaps your
OS mangles it somehow), as Sage carries 1.3.1.
Anyhow, the current meson version is 1.5.1, so both of these are old.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-support/b900ff1f-5c3e-4a43-b2dd-e0489d51b08en%40googlegroups.com.

[Kevin Youren]

Please note that the problem was caused by a lack of memory on my computer.

I modified the SWAP file to be about 10G, and the build has worked well since then. The build takes about 3 1/2 hours.

regs, Kev