When installing the CLI version of MEME 5, the 'sudo ./configure ...' command improperly sets the path to MPI

[Joel Wallenius]

The final output from the configure command gives the code below. In contrast, the path to the MPI library is actually

/usr/lib64/mpi/gcc/openmpi3/lib64, meaning the numbers 6 and 4 are missing from the path given by the configure command.

How do I fix this?

The consequent error is that shared library files are missing, causing MEME to crash and tests to fail etc...

-----------------------------------------------------------------------------

  Install path:               /home/joel/Programs/MEME

  Install UID:               
  Version:                    5.0.1
  C compiler:                 gcc
  C compiler flags:           -std=gnu89 -Wall -Wno-unused -DUNIX -D__USE_FIXED_PROTOTYPES__   -O3
  Linker:                     /usr/x86_64-suse-linux/bin/ld -m elf_x86_64
  Special Libs:               -lz -lm
  MPICC                       /usr/lib64/mpi/gcc/openmpi3/bin/mpicc
  MPIRUN                      /usr/lib64/mpi/gcc/openmpi3/bin/mpirun
  MPI_CMD                     /usr/lib64/mpi/gcc/openmpi3/bin/mpirun -np
  MPIINC                      -DMPI -I/usr/lib64/mpi/gcc/openmpi3/include
  MPILIBDIR                   /usr/lib64/mpi/gcc/openmpi3/lib
  MPIFLAGS                    -DPARALLEL
  MEME URL:                   http://meme-suite.org
  PREVIOUS VERSION URL:      
  SERVER CONTACT:            
  DEVELOPER CONTACT:          meme-...@uw.edu
  GO-SERVER URL:              http://amigo.geneontology.org/cgi-bin/amigo/term_details?term=GO_TERM_ID
  PERL:                       /usr/bin/perl
  PYTHON:                     /usr/bin/python
  PYTHON3:                    /usr/bin/python3
  CONVERT:                    /usr/bin/convert
  GHOSTSCRIPT:                /usr/bin/gs
  BUILD LIBXML2:              yes
  LIBXML2 compiler flags:     -I${top_srcdir}/src/libxml2/include
  LIBXML2 libs:               ${top_builddir}/src/libxml2/libxml2.la
  BUILD LIBXSLT:              yes
  LIBXSLT compiler flags:     -I${top_srcdir}/src/
  LIBXSLT libs:               ${top_builddir}/src/libxslt/libxslt.la
  MEME DB:                    ${datarootdir}/${PACKAGE_NAME}-5.0.1/db
  MEME LOGS:                  ${localstatedir}/${PACKAGE_NAME}-5.0.1/LOGS
  MEME TEMP FILES:           
  OPAL URL:                   no
  OPAL DEPLOY DIRECTORY:     

  Run the following commands to compile, test and install meme:
        make
        make test
        make install

[Joel Wallenius]

I managed to change the path to the correct one, I edited the few files that mentioned the path to the lib64 folder and changed it from lib to lib64.

However this only seems to affect the configure output, the rest of the installation still fails because it can't find the library files...

Generating meme motifs...
meme -oc results/motif.meme.crp0 -nostatus -p 5 -dna -nmotifs 3 common/crp0.s
/home/joel/Programs/meme-5.0.1/src/parallel/meme: error while loading shared libraries: libmpi.so.40: cannot open shared object file: No such file or directory

The same error repeats more than 30 times during the make test part.

[CharlesEGrant]

You can specify the location of the MPI executables by using the '--with-mpidir' option for configure. This is described in the installation guide. However, if you install the supporting MPI libraries somewhere other than the standard library directories (/usr/lib or /usr/lib64 you have to add that directory to the LD_LIBRARY_PATH environment variable. This is not a MEME Suite issue, but a generic UNIX library support issue. You may want to refer to the MPI build instructions, for example OpenMPI FAQ.

[Joel Wallenius]

--with-mpidir is the only mpi option, correct? I'm using it already of course, otherwise the mpi executables are not found. However the library path is still incorrect, so I had to change it manually. That didn't help though, the error message remains.

I have also added the directories to the LD_LIBRARY_PATH variable, but that changes nothing.

Are you sure it's a UNIX problem? Couldn't it be a bug in the MEME configure code? I'm not using any non-standard OS. I have the latest release of OpenSUSE, which in contrast to older releases, has Open MPI 3 available (without having to install it from a tarball, I mean).

I don't know what to do :[

[CharlesEGrant]

There are two possible different configuration issues here: compile time and run time. The MEME Suite configuration step only cares about the compile time configuration. If you got the MEME Suite to compile, then the compile time configuration was OK. It looks like the problem you face now is the run-time configuration, and the MEME Suite configure step doesn't have anything to do with that. However, I would note that we don't recommend that users modify the Makefile used for the MEME Suite build. The Makefile is built using Automake and is quite complex and easy to break in subtle ways. The MEME Suite installation should always be able to be done using options to the 'configure' or 'make' commands. Otherwise it's a bug and it's up to us to fix it.

I don't know what to do :[

The first step is to demonstrate the your installation of OpenMPI is working correctly independent of installing the MEME Suite.  Some OpenMPI distributions include test programs that can be run to check that the installation is working. Failing that, the source code for simple MPI test programs are widely available on the web. See for example this one: MPI Hello World, and this one, Open-MPI Examples. See if you can get the simplest possible MPI program compiled and working on your system. Also be aware that if you have multiple copies of the Open-MPI  libraries installed, that can be a major source of headaches.

[Joel Wallenius]

OK I figured it out now. I'm not sure exactly what the problem was, but I know these things:

Installing openmpi3 from the software installation manager in openSUSE, called YaST2, would result in the above error where it can't find the library files.

Installing openmpi3 from tarball instead, to a home folder, then installing fortran and c++ compilers, then installing MEME without sudo makes everything work.

However, I'm curious as to why the -np option is restricted to 2? I have 4 cpunits on my machine, surely I should be able to do -np 4 ??? If I try -np 3 or higher, it says there are not enough cores.

I've been googling and there are ways to add command line options that solves this, but in the case of MEME, the mpi execution is wrapped. Can I change it anyway, somehow?

Big thanks!!

[CharlesEGrant]

The MEME code doesn't do anything to limit the number of process invoked by MPI.  MEME simply passes the number of processes specified by the MEME  '-p' option onto  the MPI launcher program (mpirun). Just to make sure we hadn't made some gross error that broke the MPI code for MEME, I double checked the current version of MEME on local MacOS and Linux machines. In both cases I was able to start MEME with 15 processes despite only having four cores on each machine. 

This is an OpenMPI configuration issue which is beyond the scope of what we can help with.  Your best bet is to debug your OpenMPI installation by compiling and trying to run one of the simple example MPI programs I linked to in the answer above.

[Joel Wallenius]

I wasn't saying that MEME somehow restricted the execution of the mpi implementation, only that MEME runs mpi implicitly, somewhere in MEME's own binaries or scripts, and if I would like to add options to the mpi execution, like "--map-by socket:pe=2", which would allow 2 threads per core, I wouldn't know how to do that simply because MEME is wrapping it. As for why I can't specify more cores than are physically present, I don't know. Some quick googling suggests it's a new thing for OpenMPI3.

Anyway I think my problem would be solved if I knew how to add options to the mpi execution that MEME performs when being run. Is this possible?


[CharlesEGrant]

Anyway I think my problem would be solved if I knew how to add options to the mpi execution that MEME performs when being run. Is this possible

This is the description of the '-p' parameter for MEME:

"-p np

Use faster, parallel version of MEME with np processors. The parameter np may be a number or it may be a quoted string starting with a number and followed by arguments to the particular MPI run command for your installation (e.g., mpirun). "

So for example you mentioned, you'd use, say,

meme -dna -nmotifs 15 -p '4 --map-by socket:pe=2' seq.fa

Note the quotes around '4 --map-by socket:pe=2'.

[Joel Wallenius]

Ah.

I believe I have everything I need, then. :-]

Thank you very much Charles. I am sorry for not knowing these things already to instead impose on your time!