[raxml-ng] Multiple builds of raxml-ng

[Matthieu Muffato]

Hello,

We need to compile raxml multiple times following the matrix: SSE3/AVX
vs pthreads/no-pthreads
RAxML used to ship one several makefiles (Makefile.AVX.gcc,
Makefile.AVX.PTHREADS.gcc, etc) so we could clone the repo, compile the
first one, backup the binary, remove the .o files, compile the next one, etc

How can I do that with RAxML-ng ? I have found the USE_PTHREADS flag so
I can do `cmake -DUSE_PTHREADS=OFF ..` to control pthreads but I haven't
found the option to control SSE3/AVX. Is RAxML-ng embedding code for
both architectures and choosing the right one at runtime ?

Secondly, the README advises to create a "build" directory but I've
noticed that there are also a number of files created
libs/pll-modules/libs/libpll. It looks like I can simply do "git clean
-d -f" in the latter and recreate the top-level build directory but I
wonder if you are aware of any other files I should clean up (I'd like
to avoid having to delete everything and close again)

Thank you,
Matthieu
--
Matthieu Muffato, Ph.D.
Ensembl Compara and TreeFam Project Leader
European Bioinformatics Institute (EMBL-EBI)
European Molecular Biology Laboratory
Wellcome Trust Genome Campus, Hinxton
Cambridge, CB10 1SD, United Kingdom
Room A3-145
Phone + 44 (0) 1223 49 4631
Fax + 44 (0) 1223 49 4468

[Alexey Kozlov]

Hi Matthieu,

> We need to compile raxml multiple times following the matrix: SSE3/AVX vs pthreads/no-pthreads
> RAxML used to ship one several makefiles (Makefile.AVX.gcc, Makefile.AVX.PTHREADS.gcc, etc) so we could clone the repo,
> compile the first one, backup the binary, remove the .o files, compile the next one, etc

In raxml-ng, I dropped this approach in favor of autodetection, since I noticed that many users were choosing a "wrong"
executable and thus wasting computing resources.

> How can I do that with RAxML-ng ? I have found the USE_PTHREADS flag so I can do `cmake -DUSE_PTHREADS=OFF ..` to
> control pthreads but I haven't found the option to control SSE3/AVX. Is RAxML-ng embedding code for both architectures
> and choosing the right one at runtime ?

Yes it does so for the likelihood computation kernels in libpll, so you don't need separate SSE3/AVX versions anymore.
However, by default raxml-ng will use compile-time autodetection for other parts of the code. So if you want to compile
a universal SSE3/AVX binary on a machine with AVX, please use:

`cmake -DENABLE_RAXML_SIMD=OFF -DENABLE_PLLMOD_SIMD=OFF ..`

Another thing to keep in mind with cross-compilation is that dynamic libraries might be incompatible across systems, so
you might need to compile a static binary:

`cmake -DSTATIC_BUILD=ON -DENABLE_RAXML_SIMD=OFF -DENABLE_PLLMOD_SIMD=OFF ..`

(I assume you are well aware of this, but just in case)

I will add this information to the README to avoid confusion in the future.

Also, unlike old raxml-pthreads, NG can run sequentially even if compiled with pthreads (--threads 1), so maybe you
don't really need a separate non-pthreads version after all.

> Secondly, the README advises to create a "build" directory but I've noticed that there are also a number of files
> created libs/pll-modules/libs/libpll. It looks like I can simply do "git clean -d -f" in the latter and recreate the
> top-level build directory but I wonder if you are aware of any other files I should clean up (I'd like to avoid having
> to delete everything and close again)

Yes, current build process is a bit clumsy, since raxml-ng includes "pll-modules" as a git submodule, which it turn
includes "libpll". And the latter two projects use autotools which doesn't obey to CMake rules for placing output files.
Anyways, you can make a full cleanup with either `git clean` or `make clean`, in *both* `libs/pll-modules/` and
`libs/pll-modules`/libs/libpll` directories. In practice, however, this should be rarely needed (e.g. if you switch to a
different compiler version or change ENABLE_PLLMOD_SIMD settings). And if you e.g. only change USE_PTHREADS or USE_MPI,
just normal `cmake .. && make` should suffice.

Hope this helps,
Alexey

[Matthieu Muffato]

Hi Alexey,

Thank you for your reply. See my questions inline

On 23/05/17 14:34, Alexey Kozlov wrote:
> Hi Matthieu,
>
>> We need to compile raxml multiple times following the matrix: SSE3/AVX
>> vs pthreads/no-pthreads
>> RAxML used to ship one several makefiles (Makefile.AVX.gcc,
>> Makefile.AVX.PTHREADS.gcc, etc) so we could clone the repo,
>> compile the first one, backup the binary, remove the .o files, compile
>> the next one, etc
>
> In raxml-ng, I dropped this approach in favor of autodetection, since I
> noticed that many users were choosing a "wrong" executable and thus
> wasting computing resources.
>
>> How can I do that with RAxML-ng ? I have found the USE_PTHREADS flag
>> so I can do `cmake -DUSE_PTHREADS=OFF ..` to
>> control pthreads but I haven't found the option to control SSE3/AVX.
>> Is RAxML-ng embedding code for both architectures
>> and choosing the right one at runtime ?
>
> Yes it does so for the likelihood computation kernels in libpll, so you
> don't need separate SSE3/AVX versions anymore. However, by default
> raxml-ng will use compile-time autodetection for other parts of the
> code. So if you want to compile a universal SSE3/AVX binary on a machine
> with AVX, please use:
>
> `cmake -DENABLE_RAXML_SIMD=OFF -DENABLE_PLLMOD_SIMD=OFF ..`

"OFF" means disabling the compile-time auto-detection of SSE3 vs AVX but
these are still enabled in the resulting binary, right ? (the option
name looks counter-intuitive).

The CPU of the machine on which we compute raxml only has AVX version 1.
Would it still work if I want to compile for AVX2 or AVX512 (if and when
they are available in libpll) ? I imagine it would be fine because the
makefile is just switching gcc parameters, and parameters is able to
optimize the code for these even if the current CPU doesn't have them.

> Another thing to keep in mind with cross-compilation is that dynamic
> libraries might be incompatible across systems, so you might need to
> compile a static binary:
>
> `cmake -DSTATIC_BUILD=ON -DENABLE_RAXML_SIMD=OFF
> -DENABLE_PLLMOD_SIMD=OFF ..`
>
> (I assume you are well aware of this, but just in case)

The build is part of a linuxbrew environment that is shared across all
machines in the cluster. I've just run ldd on the raxml-ng binary and it
only links to libraries of our linuxbrew install, nothing in
raxml-ng/libs or raxml-ng/src so it should be fine

> I will add this information to the README to avoid confusion in the future.
>
> Also, unlike old raxml-pthreads, NG can run sequentially even if
> compiled with pthreads (--threads 1), so maybe you don't really need a
> separate non-pthreads version after all.

Interesting. I can remember the exact details, but I think the pthreads
version was spawning at least two threads and we wanted in some
occasions to guarantee there was a single thread running (otherwise our
job scheduler may think we're using too many resources and kill the
job). Yes, it sounds like we wouldn't need a non-pthreads version any
more. Together with the fact we don't need an SSE3 version *and* an AVX
version, it's going to be a lot easier to deploy :) !

>> Secondly, the README advises to create a "build" directory but I've
>> noticed that there are also a number of files
>> created libs/pll-modules/libs/libpll. It looks like I can simply do
>> "git clean -d -f" in the latter and recreate the
>> top-level build directory but I wonder if you are aware of any other
>> files I should clean up (I'd like to avoid having
>> to delete everything and close again)
>
> Yes, current build process is a bit clumsy, since raxml-ng includes
> "pll-modules" as a git submodule, which it turn includes "libpll". And
> the latter two projects use autotools which doesn't obey to CMake rules
> for placing output files.
> Anyways, you can make a full cleanup with either `git clean` or `make
> clean`, in *both* `libs/pll-modules/` and
> `libs/pll-modules`/libs/libpll` directories. In practice, however, this
> should be rarely needed (e.g. if you switch to a different compiler
> version or change ENABLE_PLLMOD_SIMD settings). And if you e.g. only
> change USE_PTHREADS or USE_MPI, just normal `cmake .. && make` should
> suffice.

Cool. I'll give it a try

Thank you !
Matthieu

> Hope this helps,
> Alexey

[Alexey Kozlov]

Hi Matthieu,

>> `cmake -DENABLE_RAXML_SIMD=OFF -DENABLE_PLLMOD_SIMD=OFF ..`
>
> "OFF" means disabling the compile-time auto-detection of SSE3 vs AVX but these are still enabled in the resulting
> binary, right ? (the option name looks counter-intuitive).
>
> The CPU of the machine on which we compute raxml only has AVX version 1. Would it still work if I want to compile for
> AVX2 or AVX512 (if and when they are available in libpll) ? I imagine it would be fine because the makefile is just
> switching gcc parameters, and parameters is able to optimize the code for these even if the current CPU doesn't have them.

yes, AVX2 (and later AVX512) libpll kernels will be supported as long as your gcc version supports them. The above
switches disable SSE/AVX instructions in non-libpll code *only*, and since those parts are not performance-critical we
don't have multiple versions with runtime auto-detection here.

> The build is part of a linuxbrew environment that is shared across all machines in the cluster. I've just run ldd on the
> raxml-ng binary and it only links to libraries of our linuxbrew install, nothing in raxml-ng/libs or raxml-ng/src so it
> should be fine

ok, that's fine then.

>> I will add this information to the README to avoid confusion in the future.
>>
>> Also, unlike old raxml-pthreads, NG can run sequentially even if
>> compiled with pthreads (--threads 1), so maybe you don't really need a
>> separate non-pthreads version after all.
>
> Interesting. I can remember the exact details, but I think the pthreads version was spawning at least two threads and we
> wanted in some occasions to guarantee there was a single thread running (otherwise our job scheduler may think we're
> using too many resources and kill the job). Yes, it sounds like we wouldn't need a non-pthreads version any more.
> Together with the fact we don't need an SSE3 version *and* an AVX version, it's going to be a lot easier to deploy :) !

I hope so as well :)

Best,
Alexey