A proper way to call MPI wrappers
Sergey Kosukhin
Elizabeth A. Fischer
--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+unsubscribe@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at https://groups.google.com/group/spack.
For more options, visit https://groups.google.com/d/optout.
Denis Davydov
There is already documentation on how to pass MPI compiler wrappers
On Thursday, November 17, 2016 at 2:49:39 PM UTC+1, Elizabeth A. Fischer wrote:
On Thu, Nov 17, 2016 at 8:44 AM, Sergey Kosukhin <skos...@gmail.com> wrote:
Hello all,I would like to ask what is the proper way to tell a configure script to use compiler wrappers (mpicc, etc.) that come along with MPI libraries. As far as I know, packages that require MPI just pass self.spec['mpi'].mpicc to their configure scripts, but doesn't this mean that, in this case, we lose the features that Spack's compiler wrapper gives us (-I, -L, and -Wl,rpath flags that are automatically passed to a compiler)? If yes, are there ideas on how to make spack's compiler wrapper to call the corresponding MPI compiler wrapper instead of calling the original compiler?My second question is about preparing dependent environments. The documentation says that there is a method for this:def setup_dependent_environment(self, module, spec, dep_spec)But there is an example on how to use it for the python package, which shows a different set of arguments:def setup_dependent_environment(self, spack_env, run_env, extension_spec)I got more confused when I opened the package for openmpi which has the two following methods:def setup_dependent_environment(self, spack_env, run_env, extension_spec)def setup_dependent_package(self, module, dep_spec)Could someone please explain all this to me?Thank you.
--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
Sergey Kosukhin
Denis Davydov
On Friday, November 18, 2016 at 10:16:07 AM UTC+1, Sergey Kosukhin wrote:
Denis, thank you for the link. Unfortunately, I don't understand how this patching you are talking about works for openmpi, for example. If I'm not mistaken, it detaches itself from spack's wrappers so openmpi, when installed with spack, could be used by users outside of spack (mpi wrapper start calling original compiler instead of calling spack's wrapper). But I don's see how the mentioned spack's wrapper's features are supported, when a dependent package uses self.spec['mpi'].mpicc which calls openmpi wrapper directly.
def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
spack_env.set('MPICC', join_path(self.prefix.bin, 'mpicc'))
spack_env.set('MPICXX', join_path(self.prefix.bin, 'mpic++'))
spack_env.set('MPIF77', join_path(self.prefix.bin, 'mpif77'))
spack_env.set('MPIF90', join_path(self.prefix.bin, 'mpif90'))
spack_env.set('OMPI_CC', spack_cc)
spack_env.set('OMPI_CXX', spack_cxx)
spack_env.set('OMPI_FC', spack_fc)
spack_env.set('OMPI_F77', spack_f77)
which makes Openmpi use Spack's wrappers when building packages.
Erik Schnetter
--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+unsubscribe@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at https://groups.google.com/group/spack.
For more options, visit https://groups.google.com/d/optout.
Elizabeth A. Fischer
From: Erik Schnetter <schn...@gmail.com>
Date: Wed, Nov 23, 2016 at 9:44 AM
Subject: Re: [spack] A proper way to call MPI wrappers
To: elizabet...@columbia.edu
Would you be willing to submit a PR that does this? Remember that different MPIs add different flags... but I agree, Spack should be able to handle it well.
Sergey Kosukhin
Thank you. I think you're right.
As far as I understand, it works like this (on the example of OpenMPI):
1. The last step of the install() method of the OpenMPI package is to modify OpenMPI's configuration files, so if the OpenMPI wrappers are called not from Spack, they will call the original compilers.
2. When OpenMPI wrappers are called from Spack, they work with the Spack's wrappers because values of the environment variables OMPI_CC, OMPI_CXX, etc. override contents of the OpenMPI's configuration files. And the variables are set by the method setup_dependent_environment().
P.S. I looks like we should update the Packaging Guide, because the information about the setup_dependent_environment() method is inconsistent with the API Docs.
Sergey Kosukhin
I would like to thank Erik for bringing this up. I just can't agree more.
Where do all those wrappers come from? Why do they exist for this particular class of libraries? I hate those blackboxes.
The only thing if it is possible to make sure that developers of MPI libraries do not (and will not) put any additional logic into the wrappers.
On Wednesday, November 23, 2016 at 3:51:34 PM UTC+1, Elizabeth A. Fischer wrote:
Any thoughts on this from the wider community?
---------- Forwarded message ----------
From: Erik Schnetter <schn...@gmail.com>
Date: Wed, Nov 23, 2016 at 9:44 AM
Subject: Re: [spack] A proper way to call MPI wrappers
To: elizabet...@columbia.edu
I _had_ to open my big mouth.Regarding the different MPI flags -- there should be ways (a different one for each implementation, of course) to extract the flags. I might steal this from cmake. There are two important questions, though:(1) Does this have a reasonable chance of being accepted in the end? Many people feel quite conservative about building MPI with mpicc, because "that's the way it should be done".(2) For building with MPI outside of Spack, the current mechanism should of course be retained. Also, other packages will expect to find an mpicc so that they can extract the MPI flags by themselves.
-erik
On Tue, Nov 22, 2016 at 9:56 PM, Elizabeth A. Fischer <elizabet...@columbia.edu> wrote:
Would you be willing to submit a PR that does this? Remember that different MPIs add different flags... but I agree, Spack should be able to handle it well.
On Tue, Nov 22, 2016 at 9:39 PM, Erik Schnetter <schn...@gmail.com> wrote:
There's an argument to be made to treat MPI just like any other package, and ignore the compiler wrappers. Several other packages also provide compiler wrappers for "convenience", and we ignore them. All that mpicc and friends to is adding -I, -L, and -l flags. Spack is good at that; we don't need to use mpicc.-erik
On Sat, Nov 19, 2016 at 4:51 AM, Denis Davydov <davy...@gmail.com> wrote:
Hi Sergey,
On Friday, November 18, 2016 at 10:16:07 AM UTC+1, Sergey Kosukhin wrote:Denis, thank you for the link. Unfortunately, I don't understand how this patching you are talking about works for openmpi, for example. If I'm not mistaken, it detaches itself from spack's wrappers so openmpi, when installed with spack, could be used by users outside of spack (mpi wrapper start calling original compiler instead of calling spack's wrapper). But I don's see how the mentioned spack's wrapper's features are supported, when a dependent package uses self.spec['mpi'].mpicc which calls openmpi wrapper directly.I don't understand it in full either, but i think this is done indef setup_dependent_environment(self, spack_env, run_env, dependent_spec):
spack_env.set('MPICC', join_path(self.prefix.bin, 'mpicc'))
spack_env.set('MPICXX', join_path(self.prefix.bin, 'mpic++'))
spack_env.set('MPIF77', join_path(self.prefix.bin, 'mpif77'))
spack_env.set('MPIF90', join_path(self.prefix.bin, 'mpif90'))
spack_env.set('OMPI_CC', spack_cc)
spack_env.set('OMPI_CXX', spack_cxx)
spack_env.set('OMPI_FC', spack_fc)
spack_env.set('OMPI_F77', spack_f77)
which makes Openmpi use Spack's wrappers when building packages.
--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at https://groups.google.com/group/spack.
For more options, visit https://groups.google.com/d/optout.
--
--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at https://groups.google.com/group/spack.
For more options, visit https://groups.google.com/d/optout.
Gamblin, Todd
Hi all,
I wanted to chime in on this to clarify some of the stuff that's done to handle MPI wrappers in Spack packages. The short story is that if you want to use MPI wrappers to compile your whole build, do this:
env[‘CC’] = spec[‘mpi’].mpicc
env[‘CXX’] = spec[‘mpi’].mpicxx
env[‘F77’] = spec[‘mpi’].mpif77
env[‘FC’] = spec[‘mpi’].mpifc
That’s all. This and the reasons why it works should probably be documented in the packaging guide. Consider this a first draft. And, as always, fire away with feedback.
I. Philosophy
We don’t try to force any particular build method on packagers. In my experience, whether you want to use the wrappers depends on the way the package is written, on common practice, and on “what works”. There are three-ish types of builds:
1. Some build systems work well without the wrappers and can treat MPI as an external library, where the person doing the build has to supply includes/libs/whatever. I think this is fairly uncommon.
2. Others really want the wrappers and assume you’re using an MPI “compiler” – i.e. they have no mechanism to add MPI includes/libraries/etc.
3. CMake’s FindMPI wants the wrappers, but it just uses them to extract –I/-L/-D arguments and then treats MPI like a library.
Note that some CMake builds fall into case #2 because they either don’t know about or don’t like CMake’s FindMPI support – they just assume an MPI compiler. Also, some autotools builds fall into case #3 (e.g. here is an autotools version of the CMake FindMPI stuff – I’ve seen a few projects use that since I wrote it).
So, given all that, we just leave the use of the wrappers up to the packager. If someone knows how to build a package one of these three ways, we don’t want to leave them out.
II. Packaging Conventions
By default, in a Spack package’s install() method, CC, CXX, F77, and FC point to Spack’s wrappers around the compiler you’ve chosen. See here for details. Spack’s wrappers are not the MPI compiler wrappers, though they do automatically add –I, –L, and –Wl,-rpath args for dependencies in a similar way. The MPI wrappers are a bit different in that they also add -l arguments for the MPI libs, and they used to add some special -D arguments, though I don’t know whether that’s common anymore.
For case 1, you generally don’t need to do more than patch your Makefile or add configure args as you normally would.
For case 3, you don’t need to do much of anything, as Spack puts the MPI compiler wrappers in the PATH, and the build will find them and interrogate them.
For case 2, things are a bit more complicated, as you’ll need to tell the build to use the MPI compiler wrappers instead of Spack’s compiler wrappers. All it takes some lines like this:
env[‘CC’] = spec[‘mpi’].mpicc
env[‘CXX’] = spec[‘mpi’].mpicxx
env[‘F77’] = spec[‘mpi’].mpif77
env[‘FC’] = spec[‘mpi’].mpifc
Or, if you pass CC, CXX, etc. directly to your build with, e.g., `--with-cc=<path>`, you’ll want to substitute `spec[‘mpi’].mpicc` in there instead, e.g.:
configure(‘—prefix=%s’ % prefix,
‘—with-cc=%s’ % spec[‘mpi’].mpicc)
Now, you may think that doing this will lose the includes, library paths, and RPATHs that Spack’s compiler wrapper get you, but we’ve actually set things up so that the MPI compiler wrappers use Spack’s compiler wrappers when run from within Spack. So using the MPI wrappers should really be as simple as the code above.
III. More Details
Ok, so how does all this work?
III.A. spec[‘mpi’]
If your package has a virtual dependency like `mpi`, then referring to spec[‘mpi’] within install() will get you the concrete `mpi` implementation in your dependency DAG. That is a spec object just like the one passed to install, only the MPI implementations all set some additional properties on it to help you out. E.g., in mvapich2, you’ll find this:
def setup_dependent_package(self, module, dep_spec):
self.spec.mpicc = join_path(self.prefix.bin, 'mpicc')
# … etc …
That code allows the mvapich2 package to associate an `mpicc` property with the `mvapich2` node in the DAG, so that dependents can access it. `openmpi` and `mpich` do similar things. So, no matter what MPI you’re using, spec[‘mpi’].mpicc gets you the location of the MPI compilers. This allows us to have a fairly simple polymorphic interface for information about virtual dependencies like MPI. We actually do more extensive things in the BLAS/LAPACK/SCALAPACK implementations, and I wouldn’t be opposed to adding more information like this for MPI implementations. Erik suggested this below --- I think it’s a good idea as long as we don’t force people to use it and we continue to allow builds that need the wrappers to use them.
III.B. Wrapping wrappers
Spack likes to use its own compiler wrappers to make it easy to add RPATHs to builds, and to try hard to ensure that your builds use the right dependencies. This doesn’t play nicely by default with MPI, so we have to do a couple tricks.
1. If we build MPI with Spack’s wrappers, mpicc and friends will be installed with hard-coded paths to Spack’s wrappers, and using them from outside of Spack will fail because they only work within Spack. To fix this, we patch mpicc and friends to use the regular compilers. Look at the filter_compilers method in mpich, openmpi, or mvapich2 for details.
2. We still want to use the Spack compiler wrappers when Spack is calling mpicc. Luckily, wrappers in all mainstream MPI implementations provide environment variables that allow us to dynamically set the compiler to be used by mpicc, mpicxx, etc. Denis pasted some code from this below – Spack’s build environment sets MPICC, MPICXX, etc. for mpich derivatives and OMPI_CC, OMPI_CXX, etc. for OpenMPI. This makes the MPI compiler wrappers use the Spack compiler wrappers so that your dependencies still get proper RPATHs even if you use the MPI wrappers.
III.C. Cray stuff
One notable exception to all this is the Cray environment, which uses ITS OWN wrappers, which function like MPI wrappers. On Cray systems, the CC, cc, and ftn wrappers ARE the MPI compiler wrappers, and it’s assumed that you’ll just be using them all the time. So on Cray we don’t bother with mpicc, mpicxx, etc, and we just set spec[‘mpi’].mpicc to Spack’s wrappers, which wrap the Cray wrappers, which wrap the regular compilers and include MPI flags. For packagers, that means the thing from (II) will work, even on a Cray:
env[‘CC’] = spec[‘mpi’].mpicc
as spec[‘mpi’].mpicc is just spack_cc on a Cray.
Ok, hopefully that was enough detail for most people. Let me know if you want to know more.
-Todd
Visit this group at
http://secure-web.cisco.com/1c6dHQFQZ905xqcu9LJfBmh0YKykfwTx-JUlBzYbk3PA4eIjVud3g_p2I4PydCTvH1GfGlrRGmDSdU1zfhziF_KMggD_17bp7bYybQdCCXZq3CU_boXQh08aWm_oLb_R4Zz9xi7dsFN3Xn5QHnsA21XcuVcgsEwRjzibpj-vQ6GCAjUX23aXfhEJh672MFjmcHsGUC20vLlHQIYQtN0kkeKw51W6dh8AIVIDd-N_yXu9iPxkIhnWHnVaj5qJ6B4QJNfIkQeXP3Z8KvkRWsWay_9E5uEZPgM29bRuLMlcQktkCfkWjfc_qRV4Jy2rM873gM0qdoW0KfIOLHiGPHkVINnSgHHlIeQIb-YT95pDz8nv8heSIlIDPdLXue9Dz_r0cLvAmoMF2GSJUVSLw1Cr6ah7UQ3a4ZwrDJq-tyDfmXdI/l37%3Ahttps%3A%2F%2Fgroups.google.com%2Fgroup%2Fspacke.
For more options, visit
http://secure-web.cisco.com/1YmO468B62am-h9jzj5eIV6441nIbEggn5Rj7yiWafd2xKKbP8zvxS8YLbU2saTNNZ8qesWieA2Y_BjqNJkz5UINMFCUpGQSEUJa4XLrC5bUHdi2d906H0Cd-L2R3zMjBQffbJxvW3WjSkVGTdD7znc1KQljID3txX7kVpptd_6FW82XyHiNLv5eOUCc1sSXzJvXv0k61WNbRHfRUNxgqEoJ7VBpUEYfcuOSGGCLvSlZv0fA_a6FgwDXq3JuWloIPrAAnHifHrP_dtnJuHc3nE_BAS6mSR9aSibPNNZfFKFHHtTgELUoSDRT81tkNCN8jfyhfwUUYbiHzP5kGbuNoQ02y-mg7G0qMGz-fn5ZddwYIAO4QPlvbVzGrmiIz4qYrKqubCXkFgbS3JdxmMrkue4C56alIO2qigTXOMVi2J9o/l34%3Ahttps%3A%2F%2Fgroups.google.com%2Fd%2Foptoute.