build issue

[Nichols A. Romero]

I am getting the following compile time error on an admittedly old-ish Linux machine.
/soft/apps/mpich-1.2.7p1-1/bin/mpicxx -g -O2 -DUSE_STDARG -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_UNISTD_H=1 -DHAVE_STDARG_H=1 -DUSE_STDARG=1 -DMALLOC_RET_VOID=1 -DHAVE_MPI_CPP -fexceptions -I/soft/apps/mpich-1.2.7p1-1/include/mpi2c++ -I/soft/apps/mpich-1.2.7p1-1/include -I/home/naromero/elemental-0.79-p1-src/build-gnu-mkl/include -o CMakeFiles/elemental.dir/src/core/imports/mpi.cpp.obj -c /home/naromero/elemental-0.79-p1-src/src/core/imports/mpi.cpp
/home/naromero/elemental-0.79-p1-src/src/core/imports/mpi.cpp: In function ‘int elem::mpi::QueryThread()’:
/home/naromero/elemental-0.79-p1-src/src/core/imports/mpi.cpp:96: error: ‘MPI_Query_thread’ was not declared in this scope
make[2]: *** [CMakeFiles/elemental.dir/src/core/imports/mpi.cpp.obj] Error 1

Is the version of MPICH just too old?

See the tool chain below:
set(CMAKE_SYSTEM_NAME Eureka-static)

set(MKLROOT "/soft/apps/intel-cc-11.0.074/mkl")

# The serial compilers
set(CMAKE_C_COMPILER mpicc)
set(CMAKE_CXX_COMPILER mpicxx)
# set(CMAKE_Fortran_COMPILER mpif77)

# The MPI wrappers for the C and C++ compilers
set(MPI_C_COMPILER mpicc)
set(MPI_CXX_COMPILER mpicxx)

set(CXX_FLAGS_PUREDEBUG "-g")
set(CXX_FLAGS_PURERELEASE "-g -O2")
set(CXX_FLAGS_HYBRIDDEBUG "-g")
set(CXX_FLAGS_HYBRIDRELEASE "-g -O2")

# set(CMAKE_THREAD_LIBS_INIT "-fopenmp")
# set(OpenMP_CXX_FLAGS "-fopenmp")

##############################################################

# adjust the default behaviour of the FIND_XXX() commands:
# search headers and libraries in the target environment, search
# programs in the host environment
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

##############################################################

set(MATH_LIBS "-Wl,--start-group ${MKLROOT}/lib/em64t/libmkl_intel_lp64.a ${MKLROOT}/lib/em64t/libmkl_sequential.a ${MKLROOT}/lib/em64t/lib\
mkl_core.a -Wl,--end-group -lm")

--
Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

[Jack Poulson]

Hi Nick,

As far as I can tell, MPI_Query_thread was added in the MPI-2.1 standard, which was released in June of 2008:
http://www.mpi-forum.org/docs/mpi21-report.pdf

Unfortunately, your version of MPICH, 1.2.7, was released in November of 2005:
http://www.mcs.anl.gov/research/projects/mpi/mpich1-old/

With that said, I would still consider this a bug in Elemental, as many other MPI features released in the MPI2 standard or later are checked for by the CMake build system and avoided with preprocessor directives if not found. Such a check should be added for MPI_Query_thread (and any other related MPI routines). I just opened an issue for this here:
http://code.google.com/p/elemental/issues/detail?id=4

Jack

--
You received this message because you are subscribed to the Google Groups "elemental-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elemental-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Nichols A. Romero]

Jack,

Thanks for the quick reply. I am on an x86 box and they have newer version of MPICH so I would use that for now. But 1.2.7 is the default version of MPICH on BG/P and compiling your own MPICH on BG/P is quite involved.

When you get around to a fix, please ping me.

---

[Jeff Hammond]

No, MPICH2-1.2 (note the first 2) not MPICH-1.2.x is the basis for MPI
on BGP and it supports the MPI-2.1, which I know includes
MPI_Query_thread because I use it in ARMCI-BGP.

There is no reason to implement any fix. MPICH 1.x is 5+ years old
and is as obsolete as a Sears catalog.

Jeff

On Tue, Jun 11, 2013 at 1:28 PM, Nichols A. Romero

Jeff Hammond
Argonne Leadership Computing Facility
University of Chicago Computation Institute
jham...@alcf.anl.gov / (630) 252-5381
http://www.linkedin.com/in/jeffhammond
https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond
ALCF docs: http://www.alcf.anl.gov/user-guides

[Nichols A. Romero]

Then the correct behaviour of Elemental build system is to refuse to build and give an error, "Please use > MPI-2.1 compliant version of MPI."

[Jeff Hammond]

I'll add a preprocessor macro so that the build itself will abort with
a useful warning but I have no desire whatsoever to try to make CMake
do the right thing because I think it's a steaming pile of donkey
feces :-)

Jeff

On Tue, Jun 11, 2013 at 1:45 PM, Nichols A. Romero

[Nichols A. Romero]

You can call the macro:

-DSEARS_CATALOG_OF_MPI

[Jeff Hammond]

Actually, I have a much nicer solution that doesn't require any CMake
BS. I'll just return MPI_THREAD_SINGLE from Query_thread when it is
not supported.

Jeff

On Tue, Jun 11, 2013 at 1:50 PM, Nichols A. Romero

[Jeff Hammond]

This was attached to
https://code.google.com/p/elemental/issues/detail?id=4 earlier today
but I'm not sure if anyone saw it.

Jeff

[Jack Poulson]

Hi Jeff,

I am on the road right now and had worked-on/committed a patch before I noticed yours. I simply modified the CMakeLists.txt to ensure the necessary MPI2.1 support and then simplified src/core/imports/mpi.cpp:
https://github.com/poulson/Elemental/commit/63c8f5e1b3de3dfa9326d855428754c68ca108be

My apologies for the confusion!

Jack

[Jeff Hammond]

Maybe I misunderstand the meaning of FATAL_ERROR in CMake, in which
case the following comments may be misguided...

My patch renders "message(FATAL_ERROR "Could not find
MPI_Query_thread")" unnecessary since it will enable Elemental to work
just fine with MPI-1. There's really no reason to let CMake stand in
the way of a clean workaround.

Also, I don't know why you added "message(FATAL_ERROR "Could not find
MPI_Reduce_scatter_block")" since this will block Elemental from
building on BGP and is similarly unnecessary since you already
implemented the REDUCE_SCATTER_BLOCK = ALLREDUCE+MEMCPY ||
REDUCE+SCATTER in the MPI wrappers.

Assuming FATAL_ERROR means what it should mean, I will write the
workaround for MPI_IN_PLACE such that "message(FATAL_ERROR "Support
for MPI_IN_PLACE is required")" is not necessary either.

Best,

Jeff

[Jeff Hammond]

Regarding, MPI_IN_PLACE, I see that you had the fix but removed it.
Why? I know a few different cases where one would want to force
out-of-place for performance reasons even if it was functional. It
doesn't make sense why you want to remove that code.

Jeff

[Jack Poulson]

Hi Jeff,

My thinking was to simplify the library in exchange for requiring a slightly less decrepit version of MPI. As far as I can tell, the current modifications are effectively requiring support for MPI-2.2 (despite me writing MPI-2.1 in the commit logs), which seems to have been released in 2009.
http://www.mpi-forum.org/docs/mpi-2.2/mpi22-report.pdf

Is Blue Gene/P still using MPI-2.1?

My plan is also to gradually phase out the AVOID_COMPLEX_MPI preprocessor guards. Notice that the option is now disabled by default.

Jack

[Jeff Hammond]

> My thinking was to simplify the library in exchange for requiring a slightly
> less decrepit version of MPI. As far as I can tell, the current

I think this will lead to a very minor benefit for developers and a
major pain for users. I'm very happy to maintain and extend
Elemental's MPI wrappers to support just about any borked MPI because
it's my day job to ensure that no user's project is ever blocked by
MPI.

> modifications are effectively requiring support for MPI-2.2 (despite me
> writing MPI-2.1 in the commit logs), which seems to have been released in 2009.
> http://www.mpi-forum.org/docs/mpi-2.2/mpi22-report.pdf

OpenMPI, which is widely used, is still formally only at MPI 2.1, as
demonstrated in
http://www.open-mpi.org/hg/hgwebdir.cgi/ompi-svn-mirror/file/tip/ompi/include/mpi.h.in.
You might get away with testing functions rather than MPI_VERSION and
MPI_SUBVERSION, but you have no expectation of full support for MPI
2.2 from OpenMPI even today.

> Is Blue Gene/P still using MPI-2.1?

Yes, and it always will be. BGP at ALCF will be turned off in 2014
but I know other sites that may run theirs until 2015.

> My plan is also to gradually phase out the AVOID_COMPLEX_MPI preprocessor
> guards. Notice that the option is now disabled by default.

That's fine, but recognize that sometimes a user wants to avoid
functional but non-performant implementations as well.

Best,

Jeff

[Jack Poulson]

Hi Jeff,

In that case, I will revert my changeset and apply your patch, after the following issue is resolved: in your patch you are assuming that the constant MPI_THREAD_SINGLE is defined even if MPI_Init_thread/MPI_Query_thread are not provided. Is this actually the case?

Jack

[Jed Brown]

Jeff Hammond <jham...@alcf.anl.gov> writes:

> No, MPICH2-1.2 (note the first 2) not MPICH-1.2.x is the basis for MPI
> on BGP and it supports the MPI-2.1, which I know includes
> MPI_Query_thread because I use it in ARMCI-BGP.

It is MPICH2-1.1, not that this changes the rest of the discussion here.

jedb...@login3.surveyor:~> /bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/default/bin/mpich2version
MPICH2 Version: 1.1
MPICH2 Release date: Unknown, built on Sat May 8 07:09:57 CDT 2010
MPICH2 Device: dcmfd:BGP
MPICH2 configure: --enable-cache --prefix=/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/default --mandir=/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/default/man --htmldir=/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/default/www --host=powerpc-bgp-linux --target=powerpc-bgp-linux --build=powerpc64-linux-gnu --enable-dependencies --enable-debuginfo --enable-g=none --enable-sharedlibs=gcc --with-device=dcmfd:BGP --enable-romio=yes --enable-f77=yes --enable-f90=yes --with-file-system=bgl+bglockless --enable-timer-type=device --enable-threads=multiple --with-fwrapname=fmpich.cnk --with-cross=/bghome/bgbuild/V1R4M2_200_2010-100508P/ppc/bgp/comm/lib/dev/mpich2/src/mpid/dcmfd/cross8 --with-mpe=no --enable-wrappers --disable-fast --enable-timing=no --disable-collchk --disable-graphics --disable-rlog --disable-sample --with-assert-level=2 --with-aint-size=8
MPICH2 CC: /bgsys/drivers/V1R4M2_200_2010-100508P/ppc/gnu-linux/bin/powerpc-bgp-linux-gcc -I/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/sys/include -g -Wall -O3 -mcpu=450fp2 -mstrict-align -fno-strict-aliasing
MPICH2 CXX: /bgsys/drivers/V1R4M2_200_2010-100508P/ppc/gnu-linux/bin/powerpc-bgp-linux-g++ -I/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/sys/include -g -Wall -O3 -mcpu=450fp2 -mstrict-align -fno-strict-aliasing
MPICH2 F77: /bgsys/drivers/V1R4M2_200_2010-100508P/ppc/gnu-linux/bin/powerpc-bgp-linux-gfortran -fno-underscoring -I/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/sys/include -g -Wall -O3 -mcpu=450fp2 -mstrict-align -fno-strict-aliasing
MPICH2 F90: /bgsys/drivers/V1R4M2_200_2010-100508P/ppc/gnu-linux/bin/powerpc-bgp-linux-gfortran -fno-underscoring -I/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/sys/include -g -Wall -O3 -mcpu=450fp2 -mstrict-align -fno-strict-aliasing

[Jeff Hammond]

In that case, I will revert my changeset and apply your patch, after the following issue is resolved: in your patch you are assuming that the constant MPI_THREAD_SINGLE is defined even if MPI_Init_thread/MPI_Query_thread are not provided. Is this actually the case?

This is an error. Do you want a new patch or will you #define it to 0? I believe it's 0 in implementations as well. 

Jeff

[Jack Poulson]

On Wed, Jun 12, 2013 at 5:18 AM, Jeff Hammond <jeff.s...@gmail.com> wrote:

In that case, I will revert my changeset and apply your patch, after the following issue is resolved: in your patch you are assuming that the constant MPI_THREAD_SINGLE is defined even if MPI_Init_thread/MPI_Query_thread are not provided. Is this actually the case?

This is an error. Do you want a new patch or will you #define it to 0? I believe it's 0 in implementations as well. 

Thanks Jeff. I just committed a slightly modified version of your changeset:
https://github.com/poulson/Elemental/commit/5089a73e956ede8a1caabaae8a40ab10bf3eeceb

Jack