Receive a segfault output file when trying to use pbdMPI

[Ben Fotovich]

Hello!

I'm running into an issue, and based on my internet searches (including searching RBigDataProgramming), I'm not finding a solution.

I am able to compile R and pbdMPI, but when I go to submit a job to a job queue, the following error messages are sent to the output files:

 *** caught segfault ***

address (nil), cause 'memory not mapped'

Traceback:

 1: .Call("spmd_initialize", PACKAGE = "pbdMPI")

 2: fun(libname, pkgname)

 3: doTryCatch(return(expr), name, parentenv, handler)

 4: tryCatchOne(expr, names, parentenv, handlers[[1L]])

 5: tryCatchList(expr, classes, parentenv, handlers)

 6: tryCatch(fun(libname, pkgname), error = identity)

 7: runHook(".onLoad", env, package.lib, package)

 8: loadNamespace(package, lib.loc)

 9: doTryCatch(return(expr), name, parentenv, handler)

10: tryCatchOne(expr, names, parentenv, handlers[[1L]])

11: tryCatchList(expr, classes, parentenv, handlers)

12: tryCatch(expr, error = function(e) {    call <- conditionCall(e)    if (!is.null(call)) {        if (identical(call[[1L]], quote(doTryCatch)))             call <- sys.call(-4L)        dcall <- deparse(call)[1L]        prefix <- paste("Error in", dcall, ": ")        LONG <- 75L        msg <- conditionMessage(e)        sm <- strsplit(msg, "\n")[[1L]]        w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w")        if (is.na(w))             w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L],                 type = "b")        if (w > LONG)             prefix <- paste0(prefix, "\n  ")    }    else prefix <- "Error : "    msg <- paste0(prefix, conditionMessage(e), "\n")    .Internal(seterrmessage(msg[1L]))    if (!silent && identical(getOption("show.error.messages"),         TRUE)) {        cat(msg, file = stderr())        .Internal(printDeferredWarnings())    }    invisible(structure(msg, class = "try-error", condition = e))})

13: try({    attr(package, "LibPath") <- which.lib.loc    ns <- loadNamespace(package, lib.loc)    env <- attachNamespace(ns, pos = pos, deps)})

14: library(pkg, character.only = TRUE, logical.return = TRUE, lib.loc = lib.loc,     quietly = quietly)

15: .getRequiredPackages2(pkgInfo, quietly = quietly)

16: library(pbdDMAT, quiet = TRUE)

An irrecoverable exception occurred. R is aborting now ...

Here is the configure command for OpenMPI 2.0.1 (which pbdMPI is compiled against) (additionally, the same install of OpenMPI 2.0.1 is being successfully used with other MPI aware applications):

./configure --prefix=/usr/local/openmpi/2.0.1 --enable-mpi-thread-multiple --enable-mpi-fortran=yes --enable-mpi-cxx --with-sge --with-tm --disable-dlopen

Here is the Rmpi 0.6-6 configure string (just in case, I know it isn't your project):

R CMD INSTALL --configure-args="--with-Rmpi-include=/usr/local/openmpi/2.0.1/include --with-Rmpi-libpath=/usr/local/openmpi/2.0.1/lib --with-Rmpi-type=OPENMPI" Rmpi_0.6-6.tar.gz

I looked at the pbdMPI-guide.pdf, and made sure to set "--no-test-load", to get around the issue receiving an error when pbdMPI is compiled, in the package install command:

R CMD INSTALL pbdMPI_0.3-2.tar.gz --configure-args=" --with-mpi-type=OPENMPI --with-mpi-include=/usr/local/openmpi/2.0.1/include --with-mpi-libpath=/usr/local/openmpi/2.0.1/lib --with-mpi=/usr/local/openmpi/2.0.1" --no-test-load

I also exported the environment variables right below this, to compile pbdMPI, to see if that would assist (which it didn't seem to):

export MPI_ROOT=/usr/local/openmpi/2.0.1

export MPI_LIBS=" -L/usr/local/openmpi/2.0.1/lib"

export MPI_LIBNAME=/usr/local/openmpi/2.0.1/lib/libmpi.so

The above environment variables are not set for R, as it doesn't seem to make a difference.

Here is the output of "env":

MODULE_VERSION_STACK=3.2.6

MANPATH=/usr/local/proj-4.9.2/share/man:/usr/local/gdal-2.1.1/share/gdal:/usr/local/python/2.7.3/share/man:/usr/local/libpng-1.4.13/share/man:/usr/local/freetype-2.5.3/share/man:/usr/local/JAGS-4.2.0/share/man:/usr/local/curl-7.50.3/share/man:/usr/local/pcre-8.39/share/man:/usr/local/xz-5.2.2/share/man:/usr/local/bzip2-1.0.6/man:/usr/local/zlib-1.2.8/share/man:/usr/local/openmpi/2.0.1/share/man:/usr/local/gcc-4.9.2/man:/usr/local/gmp-5.0.5/share/info:/usr/local/mpfr-3.1.1/share/doc/mpfr:/usr/local/mpc-1.0.1/share/info:/usr/local/Modules/3.2.6/share/man:/usr/local/Modules/3.2.6/man

HOSTNAME=du-hpc

PYTHON_LIBRARIES=/usr/local/python/2.7.3/lib

MPICC=/usr/local/openmpi/2.0.1/bin/mpicc

TERM=xterm

SHELL=/bin/bash

HISTSIZE=1000

SSH_CLIENT=130.253.15.56 58038 22

SGE_CELL=default

SGE_ARCH=lx-amd64

QTDIR=/usr/lib64/qt-3.3

QTINC=/usr/lib64/qt-3.3/include

F90FLAGS=-I/usr/local/gcc-4.9.2/include -L/usr/local/gcc-4.9.2/lib64

SSH_TTY=/dev/pts/10

USER=sgetest

LD_LIBRARY_PATH=/usr/local/R/3.3.1-SGE/lib64/R/lib:/usr/local/binutils-2.26/lib:/usr/local/proj-4.9.2/lib:/usr/local/gdal-2.1.1/lib:/usr/local/geos-3.5.0/lib:/usr/local/python/2.7.3/lib:/usr/local/libpng-1.4.13/lib:/usr/local/freetype-2.5.3/lib:/usr/local/boost/1.61/lib:/usr/local/JAGS-4.2.0/lib:/usr/local/curl-7.50.3/lib:/usr/local/pcre-8.39/lib:/usr/local/xz-5.2.2/lib:/usr/local/bzip2-1.0.6/lib:/usr/local/zlib-1.2.8/lib:/usr/local/lapack-3.5.0/lib:/usr/local/blas-3.5.0/lib:/usr/local/openmpi/2.0.1/lib:/usr/local/openmpi/2.0.1/lib/openmpi:/usr/local/openmpi/2.0.1/lib/pkgconfig:/usr/local/gcc-4.9.2/lib64:/usr/local/gmp-5.0.5/lib:/usr/local/mpfr-3.1.1/lib:/usr/local/mpc-1.0.1/lib

LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:

CXXFLAGS=-I/usr/local/gcc-4.9.2/include -L/usr/local/gcc-4.9.2/lib64

CPP=/usr/local/openmpi/2.0.1/bin/

MODULE_VERSION=3.2.6

MAIL=/var/spool/mail/sgetest

PATH=/usr/local/R/3.3.1-SGE/lib64/R/bin:/usr/local/binutils-2.26/bin:/usr/local/proj-4.9.2/bin:/usr/local/gdal-2.1.1/bin:/usr/local/geos-3.5.0/bin:/usr/local/swig-3.0.10/bin:/usr/local/python/2.7.3/bin:/usr/local/libpng-1.4.13/bin:/usr/local/freetype-2.5.3/bin:/usr/local/JAGS-4.2.0/bin:/usr/local/curl-7.50.3/bin:/usr/local/pcre-8.39/bin:/usr/local/xz-5.2.2/bin:/usr/local/bzip2-1.0.6/bin:/usr/local/openmpi/2.0.1/bin:/usr/local/gcc-4.9.2/bin:/usr/local/Modules/3.2.6/bin:/usr/local/sge/8.1.9/bin:/usr/local/sge/8.1.9/bin/lx-amd64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/sgetest/bin

LD_RUN_PATH=/usr/local/gcc-4.9.2/lib64

F90=/usr/local/gcc-4.9.2/bin/gfortran

PWD=/home/sgetest/r

_LMFILES_=/usr/local/Modules/modulefiles/null:/usr/local/Modules/modulefiles/modules:/usr/local/Modules/modulefiles/tools/mpc-1.0.1:/usr/local/Modules/modulefiles/tools/mpfr-3.1.1:/usr/local/Modules/modulefiles/tools/gmp-5.0.5:/usr/local/Modules/modulefiles/compilers64/gcc-4.9.2:/usr/local/Modules/modulefiles/mpi2/openmpi-2.0.1-gcc.64.ib:/usr/local/Modules/modulefiles/tools/blas-3.5.0:/usr/local/Modules/modulefiles/tools/lapack-3.5.0:/usr/local/Modules/modulefiles/tools/zlib-1.2.8:/usr/local/Modules/modulefiles/tools/bzip2-1.0.6:/usr/local/Modules/modulefiles/tools/xz-5.2.2:/usr/local/Modules/modulefiles/tools/pcre-8.39:/usr/local/Modules/modulefiles/tools/curl-7.50.3:/usr/local/Modules/modulefiles/tools/jags-4.2.0:/usr/local/Modules/modulefiles/tools/boost-1.61:/usr/local/Modules/modulefiles/tools/freetype-2.5.3:/usr/local/Modules/modulefiles/tools/libpng-1.4.13:/usr/local/Modules/modulefiles/tools/python-2.7.3:/usr/local/Modules/modulefiles/tools/swig-3.0.10:/usr/local/Modules/modulefiles/tools/geos-3.5.0:/usr/local/Modules/modulefiles/tools/gdal-2.1.1:/usr/local/Modules/modulefiles/tools/proj-4.9.2:/usr/local/Modules/modulefiles/tools/binutils-2.26:/usr/local/Modules/modulefiles/apps/R-3.3.1-SGE

SGE_EXECD_PORT=6445

LANG=en_US.UTF-8

MODULEPATH=/usr/local/Modules/versions:/usr/local/Modules/$MODULE_VERSION/modulefiles:/usr/local/Modules/modulefiles:

SGE_QMASTER_PORT=6444

LOADEDMODULES=null:modules:tools/mpc-1.0.1:tools/mpfr-3.1.1:tools/gmp-5.0.5:compilers64/gcc-4.9.2:mpi2/openmpi-2.0.1-gcc.64.ib:tools/blas-3.5.0:tools/lapack-3.5.0:tools/zlib-1.2.8:tools/bzip2-1.0.6:tools/xz-5.2.2:tools/pcre-8.39:tools/curl-7.50.3:tools/jags-4.2.0:tools/boost-1.61:tools/freetype-2.5.3:tools/libpng-1.4.13:tools/python-2.7.3:tools/swig-3.0.10:tools/geos-3.5.0:tools/gdal-2.1.1:tools/proj-4.9.2:tools/binutils-2.26:apps/R-3.3.1-SGE

SGE_ROOT=/usr/local/sge/8.1.9

F77=/usr/local/gcc-4.9.2/bin/gfortran

F77FLAGS=-I/usr/local/gcc/4.9.2-include -L/usr/local/gcc-4.9.2/lib64 -fPIC

CXX=/usr/local/gcc-4.9.2/bin/c++

SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass

HISTCONTROL=ignoredups

SHLVL=1

HOME=/home/sgetest

GCCHOME=/usr/local/gcc-4.9.2

CFLAGS=-I/usr/local/gcc-4.9.2/include -L/usr/local/gcc-4.9.2/lib64 -O2 -fPIC

MPIHOME=/usr/local/openmpi/2.0.1

FC=/usr/local/gcc-4.9.2/bin/gfortran

MATHEMATICA_HOME=/usr/local/mathematica/10.4.1

PYTHONPATH=/usr/local/python/2.7.3/lib/python2.7:/usr/local/python/2.7.3/lib/python2.7/site-packages

LOGNAME=sgetest

BOOST_ROOT=/usr/local/boost/1.61

QTLIB=/usr/lib64/qt-3.3/lib

CVS_RSH=ssh

SSH_CONNECTION=130.253.15.56 58038 130.253.21.117 22

MODULESHOME=/usr/local/Modules/3.2.6

LESSOPEN=||/usr/bin/lesspipe.sh %s

CC=/usr/local/gcc-4.9.2/bin/gcc

SGE_CLUSTER_NAME=p6444

INCLUDE=/usr/local/binutils-2.26/include:/usr/local/proj-4.9.2/include:/usr/local/gdal-2.1.1/include:/usr/local/geos-3.5.0/include:/usr/local/python/2.7.3/include/python2.7:/usr/local/libpng-1.4.13/include:/usr/local/freetype-2.5.3/include/freetype2:/usr/local/boost/1.61/include/boost:/usr/local/JAGS-4.2.0/include:/usr/local/curl-7.50.3/include:/usr/local/pcre-8.39/include:/usr/local/xz-5.2.2/include:/usr/local/bzip2-1.0.6/include:/usr/local/zlib-1.2.8/include:/usr/local/openmpi/2.0.1/include:/usr/local/gmp-5.0.5/include:/usr/local/mpfr-3.1.1/include:/usr/local/mpc-1.0.1/include

G_BROKEN_FILENAMES=1

FFLAGS=-I/usr/local/gcc-4.9.2/include -L/usr/local/gcc-4.9.2/lib64 -fPIC

_=/bin/env

Thanks for any assistance or guidance you are able to offer! And I apologize if the fix is easy and I missed it.

Ben

[Benson Muite]

Hi,

Found version of MPI to be somewhat picky. Not sure if it will help, but in collaboration with others setup an install script that might work with some modifications for you. It can be found at:

https://github.com/Digitaalhumanitaaria/biohpc

Benson

--
Programming with Big Data in R
Simplifying Scalability
http://r-pbd.org/
---
You received this message because you are subscribed to the Google Groups "RBigDataProgramming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rbigdataprogram...@googlegroups.com.
To post to this group, send email to rbigdatap...@googlegroups.com.
Visit this group at https://groups.google.com/group/rbigdataprogramming.
To view this discussion on the web visit https://groups.google.com/d/msgid/rbigdataprogramming/28f27be6-292f-4678-a53f-51b7b2bc5bae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Wei-Chen Chen]

I have
- set "CFLAGS=-fPIC" when configuring openmpi-2.0.1.
- set openmpi binary in PATH and openmpi library in LD_LIBRARY_PATH as well.
- install pbdMPI-0.3-3.tar.gz
then allgather in 4 ranks worked as usual and fine.
Please check if your openmpi installation correct.

[Ben Fotovich]

Hey Benson and Wei-Chen,

Thanks for the additional information! I was able to get pbdMPI to work, by looking at the scripts Benson linked.

Here is ultimately what I ended up doing to get pbdMPI working:

I compiled OpenMPI 1.10.4 with this configure string:

./configure --prefix=/usr/local/openmpi/1.10.4 --enable-mpi-thread-multiple --enable-mpi-fortran=yes --enable-mpi-cxx --with-sge --with-tm --disable-dlopen

I compiled R, with this configure string:

./configure --prefix=/usr/local/R/3.3.1-SGE --with-recommended-packages --enable-static --enable-shared LDFLAGS="-L/usr/local/xz-5.2.2/lib" CPPFLAGS="-I/usr/local/zlib-1.2.8/include -I/usr/local/bzip2-1.0.6/include -I/usr/local/xz-5.2.2/include -I/usr/local/pcre-8.39/include -I/usr/local/curl-7.50.3/include -L/usr/local/bzip2-1.0.6/lib -L/usr/local/pcre-8.39/lib -L/usr/local/xz-5.2.2/lib -L/usr/local/curl-7.50.3/lib" --with-x=no

I compiled Rmpi with this string:

R CMD INSTALL --configure-vars="CPPFLAGS=-I/usr/local/openmpi/1.10.4/include LDFLAGS=' -L/usr/local/openmpi/1.10.4/lib'" --configure-args="--with-Rmpi-include=/usr/local/openmpi/1.10.4/include --with-Rmpi-libpath=/usr/local/openmpi/1.10.4/lib --with-Rmpi-type=OPENMPI" Rmpi_0.6-6.tar.gz

I compiled pbdMPI with this string:

R CMD INSTALL pbdMPI_0.3-2.tar.gz --configure-args=" --with-mpi-type=OPENMPI --with-mpi-include=/usr/local/openmpi/1.10.4/include --with-mpi-libpath=/usr/local/openmpi/1.10.4/lib --with-mpi=/usr/local/openmpi/1.10.4" --no-test-load

Thanks again for your assistance!

Ben

[Benson Muite]

Hi Ben,

Thanks for the feedback.

Benson

--

Programming with Big Data in R
Simplifying Scalability
http://r-pbd.org/
---
You received this message because you are subscribed to the Google Groups "RBigDataProgramming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rbigdataprogram...@googlegroups.com.
To post to this group, send email to rbigdatap...@googlegroups.com.
Visit this group at https://groups.google.com/group/rbigdataprogramming.

To view this discussion on the web visit https://groups.google.com/d/msgid/rbigdataprogramming/c2cc1ef1-4ef7-4839-a74f-be916404a069%40googlegroups.com.