Extra option to configure from within spack

[Ingve Simonsen]

Hi,

We are trying to install openmpi via spack with support for slurm on a cluster running debian 10:

spack spec ope...@4.0.5 %g...@10.2.0 +cxx+pmi schedulers=slurm

When I try to install this package via spack, I end up with a longer error message from spack (partly given below). From the log file (see below), it seems that the problem is related to the library libpmi or libpmi2 that are not found by the configure script.

The slurm version that we use is provided by the OS and defined in the packages.yaml file by the entry

  slurm:
    externals:
    - spec: sl...@18.08.5-2
      prefix: /usr
    buildable: false

The related SLURM PMI2 files are provided by the debian package "libpmi2-0-dev" that contains the following files

=> apt-file list libpmi2-0-dev
libpmi2-0-dev: /usr/include/slurm/pmi2.h  
libpmi2-0-dev: /usr/lib/x86_64-linux-gnu/libpmi2.a
libpmi2-0-dev: /usr/lib/x86_64-linux-gnu/libpmi2.la
libpmi2-0-dev: /usr/lib/x86_64-linux-gnu/libpmi2.so
libpmi2-0-dev: /usr/share/doc/libpmi2-0-dev/changelog.Debian.gz
libpmi2-0-dev: /usr/share/doc/libpmi2-0-dev/copyright

The configuration script run during the spack installation of openmpi finds the headerfile /usr/include/slurm/pmi2.h but not the libraries under /usr/lib/x86_64-linux-gnu/

I have checked that if I manually provide the EXTRA option '--with-pmi-libdir=/usr/lib/x86_64-linux-gnu' to the configure script for openmpi [in addition to '--with-pmi=/usr'], it finds libpmi2, and I can compile the package.

So finally my question: Is there, from within spack, is a way to provide this extra option to configure (or another way to proceed) to install this package. I want to make sure there is no simple way within spack to sort this out before starting to making links inside the OS file hierarchy or to compiling openmpi manually.

Thank you in advance for your help and suggestions.

Ingve



List of the error :

==> Installing openmpi
==> No binary for openmpi found: installing from source
==> Using cached archive: /share/spack/tux/v1.0/var/spack/cache/_source-cache/archive/c5/c58f3863b61d944231077f344fe6b4b8fbb83f3d1bc93ab74640bf3e5acac009.tar.bz2
==> openmpi: Executing phase: 'autoreconf'
==> openmpi: Executing phase: 'configure'
==> Error: ProcessError: Command exited with status 1:
    '/tmp/spack/spack-stage/spack-stage-openmpi-4.0.5-xfdk2qnhecv7pjzfv4zueo5eakaudocy/spack-src/configure' '--prefix=/share/spack/tux/v1.0/opt/spack/linux-debian10-sandybridge/gcc-10.2.0/openmpi-4.0.5-xfdk2qnhecv7pjzfv4zueo5eakaudocy' '--enable-shared' '--disable-silent-rules' '--disable-builtin-atomics' '--with-pmi=/usr' '--enable-static' '--with-zlib=/share/spack/tux/v1.0/opt/spack/linux-debian10-sandybridge/gcc-10.2.0/zlib-1.2.11-iot5la6sevsboyajeshgsv6i2nc7zgg5' '--enable-mpi1-compatibility' '--without-xpmem' '--without-ofi' '--without-ucx' '--without-psm' '--without-knem' '--without-verbs' '--without-hcoll' '--without-psm2' '--without-mxm' '--without-fca' '--without-cma' '--with-slurm' '--without-lsf' '--without-sge' '--without-tm' '--without-alps' '--without-loadleveler' '--disable-memchecker' '--with-hwloc=/share/spack/tux/v1.0/opt/spack/linux-debian10-sandybridge/gcc-10.2.0/hwloc-2.2.0-ih3bkme6h5m5yxp5y2lubrlpypyfbyj6' '--disable-java' '--disable-mpi-java' '--without-cuda' '--enable-wrapper-rpath' '--disable-wrapper-runpath' '--enable-mpi-cxx' '--disable-cxx-exceptions' '--with-wrapper-ldflags=-Wl,-rpath,/share/spack/tux/v1.0/opt/spack/linux-debian10-sandybridge/gcc-8.3.0/gcc-10.2.0-dk2b74ada3mkbmqyw6inbzavao3qf2wz/lib64 -Wl,-rpath,/share/spack/tux/v1.0/opt/spack/linux-debian10-sandybridge/gcc-8.3.0/gcc-10.2.0-dk2b74ada3mkbmqyw6inbzavao3qf2wz/lib/gcc/x86_64-pc-linux-gnu/10.2.0 -Wl,-rpath,/share/spack/tux/v1.0/opt/spack/linux-debian10-sandybridge/gcc-8.3.0/gcc-10.2.0-dk2b74ada3mkbmqyw6inbzavao3qf2wz/lib64'

1 error found in build log:
     171    configure: WARNING:     /usr/slurm
     172    configure: WARNING: Specified path: /usr
     173    configure: WARNING: OR neither libpmi, libpmi2, or libpmix were found under:
     174    configure: WARNING:     /lib
     175    configure: WARNING:     /lib64
     176    configure: WARNING: Specified path:
  >> 177    configure: error: Aborting

See build log for details:
  /tmp/spack/spack-stage/spack-stage-openmpi-4.0.5-xfdk2qnhecv7pjzfv4zueo5eakaudocy/spack-build-out.txt

------------------------

The last part of the log file   /tmp/spack/spack-stage/spack-stage-openmpi-4.0.5-xfdk2qnhecv7pjzfv4zueo5eakaudocy/spack-build-out.txt


checking for netdb.h... yes
checking netinet/in.h usability... yes
checking netinet/in.h presence... yes
checking for netinet/in.h... yes
checking netinet/tcp.h usability... yes
checking netinet/tcp.h presence... yes
checking for netinet/tcp.h... yes
checking for struct sockaddr_in... yes
checking if --with-cuda is set... not set (--with-cuda=no)
checking whether CU_POINTER_ATTRIBUTE_SYNC_MEMOPS is declared... no
checking whether cuPointerGetAttributes is declared... no
checking if have cuda support... no
checking if user requested PMI support... yes
checking for pmi.h in /usr... not found
checking for pmi.h in /usr/include... not found
checking for pmi.h in /usr/include/slurm... not found
checking for libpmi in /usr/lib... checking for libpmi in /usr/lib64... not found
checking for pmi2.h in /usr... not found
checking for pmi2.h in /usr/include... not found
checking for pmi2.h in /usr/include/slurm... found
checking pmi2.h usability... yes
checking pmi2.h presence... yes
checking for pmi2.h... yes
checking for libpmi2 in /usr/lib... checking for libpmi2 in /usr/lib64... not found
checking for pmix.h in /usr... not found
checking for pmix.h in /usr/include... not found
checking can PMI support be built... no
configure: WARNING: PMI support requested (via --with-pmi) but neither pmi.h,
configure: WARNING: pmi2.h or pmix.h were found under locations:
configure: WARNING:     /usr
configure: WARNING:     /usr/slurm
configure: WARNING: Specified path: /usr
configure: WARNING: OR neither libpmi, libpmi2, or libpmix were found under:
configure: WARNING:     /lib
configure: WARNING:     /lib64
configure: WARNING: Specified path:
configure: error: Aborting

--

----------------------------------------------------------------------------------------------

Ingve Simonsen                       http://web.phys.ntnu.no/~ingves         

Physics Department, NTNU        

NO-7491 Trondheim               Tel (Fax)  : +47 73 59 34 17 (..7710)

NORWAY                          

   

[Gamblin, Todd]

Hi Ingve:

Honestly as a short-term workaround, I would just do:

spack edit openmpi

And add this option (or a variant for it) yourself, e.g. somewhere in the config_args method:

https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/openmpi/package.py#L476

What should probably happen after that is for us to add an argument to the SLURM package so that it knows to look for special libdirs for external SLURM instances.  Then the openmpi package can just ask slurs where its libraries are and add the arg you want.

If you want to take a stab at that, feel free, but the shorter path in this particular case is to just hack in your argument.

-Todd

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spack/CAGE8CSXh6Z5pmONU2XZ0r_bPur%2B-YRVJPPj1otM8iRCao5ttZA%40mail.gmail.com.

[Ingve Simonsen]

Hi Todd,

Thanks for the help. Your suggestioned workaround by adding the '--with-pmi-libdir' option directly to the openmpi package configuration file indeed worked as intended.

Once I get to know spack better, I hope to be able to provide a more permanant solution to this issue along the lines that you suggested. 

Ingve