sagemathlib only building single-threaded

[Nils Bruin]

For the trac-to-github transition I decided to move over my git upstream and origin. That all seems to have worked fine and I pulled the latest develop and ran "make configure" (was already happy) and then "make". This worked fine for the most part and with export MAKE="make -j8" it went quite quickly too. Until I got to "sagemathlib". It seemed to cythonize quite quickly but then spent about 30 minutes compiling and did so with a single job at a time.

I think previously sagemathlib built in parallel. What settings should I flip to convince the build system to build sagemathlib with multiple processes in parallel?

[Dima Pasechnik]

I don't think setting MAKE is a good idea.

Did you try simply

     make -j8 build

?

On Tue, 7 Feb 2023, 17:52 Nils Bruin, <nbr...@sfu.ca> wrote:

For the trac-to-github transition I decided to move over my git upstream and origin. That all seems to have worked fine and I pulled the latest develop and ran "make configure" (was already happy) and then "make". This worked fine for the most part and with export MAKE="make -j8" it went quite quickly too. Until I got to "sagemathlib". It seemed to cythonize quite quickly but then spent about 30 minutes compiling and did so with a single job at a time.

I think previously sagemathlib built in parallel. What settings should I flip to convince the build system to build sagemathlib with multiple processes in parallel?

--
You received this message because you are subscribed to the Google Groups "sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-support...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-support/22f0873f-aeaf-4a83-9905-0fca0992c780n%40googlegroups.com.

[Nils Bruin]

On Tuesday, 7 February 2023 at 09:56:06 UTC-8 dim...@gmail.com wrote:

I don't think setting MAKE is a good idea.

Did you try simply

     make -j8 build

 

Thanks! I've tried "touch src/sage/structure/*" -- that triggers quite a large part of sagemathlib to be rebuilt. It used only 4 processes during cythonization (when it was running just python processes), but it happily used in the range of 8 processes during compilation (and that's the much more expensive phase).

It reported less than 10 minutes real time for about 60 minutes user. With the previous single thread, it took about 30 minutes (real and user). I don't know if that's because this is on a 4-core machine with hyperthreading or because of other bottlenecks.

Documentation I was able to find is still recommending MAKE="...". So we should probably change that now that we have a more make-centric build system.

[Dima Pasechnik]

IIRC, some sanitising of MAKE was implemented, as with recursive make such a setting really might lead to over-parallelisation

--

You received this message because you are subscribed to the Google Groups "sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-support...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-support/7653bddd-bcc0-4724-a685-64f9b5e8ec12n%40googlegroups.com.

[Nils Bruin]

On Tuesday, 7 February 2023 at 09:56:06 UTC-8 Dima Pasechnik wrote:

> I don't think setting MAKE is a good idea.

>

> Did you try simply

>

 >   make -j8 build

This actually causes a bit of a problem when you want to install additional packages, like `sage -i normaliz pynormaliz`: I wouldn't know how to pass the "-j8" option to the implicitly called make in that situation.

[Dima Pasechnik]

     make -j8 normaliz pynormaliz

"sage -i"  is obsolete - at least in my mind :-)

--
You received this message because you are subscribed to the Google Groups "sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-support...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-support/b004c7b4-d4ca-4ced-817a-08a2b30a4335n%40googlegroups.com.

[Nils Bruin]

On Saturday, 11 February 2023 at 01:51:47 UTC-8 Dima Pasechnik wrote:

>

>    make -j8 normaliz pynormaliz

>

> "sage -i"  is obsolete - at least in my mind :-)

OK -- thanks! so what used to be "spkg"s are now makefile targets. That reduces the possibility of having "third party" spkgs -- I guess that means they should be packaged as "pip" packages instead.

Also for this, sage itself actually instructed me to install it with `sage -i`, so if "make" is now preferred, we should change docs on that. I think it even suggested something like "!sage -i normaliz" which is difficult to find a make analogue of.

[John H Palmieri]

I have been setting MAKE and MAKEFLAGS for a long time, and I have not seen any difference in the behavior of "make build". If I unset MAKE and MAKEFLAGS, then it takes longer to build sagelib, as it should.