[maker-devel] Help with MPI installation and configuration

[Tim Fallon]

Hi there,

I’ve been troubleshooting the maker 3.0.0 beta plus MPI installation and configuration with our institute’s IT department.  Despite a lot of tries and recompilations from scratch, we haven’t been able to get it to work (non-MPI maker works fine, albeit a bit slow).  I’ve attached our most recent crash log when trying to run maker with MPI.  Any tips?  I’ve never used MPI before maker so not that familiar with troubleshooting it.

See below for my most recent contact with IT on it.  I’ve also attached the most recent log file.  We are trying to execute it through the LSF cluster submission tool, but I don’t think that is related to the problem.  Any ideas?

"

I think one option is to try a build on a host without the full suite of cluster software.  It may actually help, but if nothing else we can say that our problem exists with a (relatively) vanilla ubuntu linux host.

Peter Macfarlane / WIBR IT


Description : Hi Peter,

Hoping to hear back from you on this.  It does seem like we’ve exhausted the reasonable troubleshooting for this maker MPI execution, but maybe worth another shot before I ask the developers?

All the best,

-Tim

On Apr 7, 2017, at 9:11 AM, Tim Fallon <tfa...@mit.edu> wrote:

Hi Peter,

I’ve tested with the following command:

bsub -o bsub_maker_with_mpi_log.txt  "/nfs/apps/test/openmpi/bin/mpirun -n 8 /nfs/apps/test/maker/bin/maker”

Unfortunately it still seems to be giving errors and not running, with things like:

"“mca_base_component_repository_open: unable to open mca_patcher_overwrite: /nfs/apps/test/openmpi/lib/openmpi/mca_patcher_overwrite.so: undefined symbol: mca_patcher_base_patch_t_class (ignored)””

It also doesn’t work if I try to run it directly on Tak4.

I’ve attached the relevant error log file.  Any more solutions?

All the best,

-Tim

On Apr 5, 2017, at 3:50 PM, IT Systems Group <unix...@wi.mit.edu> wrote:

I tried rebuilding maker 3 with it's own also-built-from-source openmpi instance.  So with that:

- /nfs/apps/test/maker is now maker 3.00
- /nfs/apps/test/openmpi was used to make the aforementioned maker

I think that all you'll need to do differently here is invoke the version of mpirun in /nfs/apps/test/openmpi/bin

"

All the best,

-Tim

Timothy R. Fallon

PhD candidate

Laboratory of Jing-Ke Weng

Department of Biology
MIT

tfa...@mit.edu

[Chandler]

Please try with MPICH instead of openmpi... we were having similar
errors with openmpi, but when we switched to mpich it all started
working. Make sure only the mpich executables are in your PATH, and
that maker is built specifying the mpich package.

Chandler / Systems Administrator
Arizona Genomics Institute
www.genome.arizona.edu



_______________________________________________
maker-devel mailing list
maker...@box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

[Carson Holt]

I agree that just using MPICH3 is the easiest solution. But to use OpenMPI you will need to export the environmental variables add the command line flags to mpiexec explained in the ../maker/INSTALL file.

#this must be doen exery time you run it

export LD_PRELOAD=/location/of/openmpi/lib/libmpi.so

# -mca btl ^openib myust be on the command line

mpiexec -mca btl ^openib -n 20 maker

#alternatively add --mca btl vader,tcp,self --mca btl_tcp_if_include ib0 (use ib0 or whatever adapter interface exists on your cluster)

mpiexec --mca btl vader,tcp,self --mca btl_tcp_if_include ib0 -n 20 maker

—Carson

<bsub_maker_with_mpi_log.txt>

[Tim Fallon]

Hi Carson,

Thanks for the feedback, I’ll look into the maker INSTALL file to ensure things are setup correctly.

Should I be looking into MPICH3 instead?  I’m trying to run maker on a single node (~20 CPUs), rather than distribute it across our cluster.  I assume that using MPICH3 changes the mpiexec / mpirun executable, but not the commands otherwise?  I ask as I may have to configure this myself one day, so any tips appreciated.  In the meantime your comments seem straightforward, will give it a shot.

All the best,

-Tim

[Carson Holt]

MPICH3 works without any special configuration. OpenMPI is built to support infiniband which internally can drive other problems related to how registered memory is used when infiniband libraries are loaded. So you have have to set more parameters.

Both will use shared memory for communications when running on a single machine, but you will still need to set parameters on OpenMPI.

—Carson