CMake extra libraries?

Victor Eijkhout

unread,

Jun 8, 2017, 10:42:47 AM6/8/17

to Deal.II Googlegroup

Is there a cmake flag for specifying some extra libraries? Trilinos seems to have been built with a python interface, but deal is somehow not linking the libpython, so the cmake fails.

It would probably be easiest if I can explicitly give cmake a couple of extra libraries to link. What’s the flag for taht?

Victor.

Tobi Young

unread,

Jun 8, 2017, 11:42:40 AM6/8/17

to dealii

Is there a cmake flag for specifying some extra libraries?

Does: cmake -D extra_flags work?

Best,

Toby

Denis Davydov

unread,

Jun 8, 2017, 6:10:40 PM6/8/17

to deal.II User Group

U need to build with current master branch, look at commit message here https://github.com/dealii/dealii/pull/4437 essentially u would get the location of Trilinos used by deal.ii and link in extra libs

Regards,
Denis

Victor Eijkhout

unread,

Jun 15, 2017, 1:38:26 PM6/15/17

to deal.II User Group

That reply is way too cryptic for me. You seem to be saying that I need to add something to a file that is generated by cmake. But my cmake crashes to begin with.

And even then, it seems to be about a file deal.IIFeatureConfig.cmake which I can't find anywhere.

Victor

PS so there is no simple flag for "add these libraries"?

Denis Davydov

unread,

Jun 15, 2017, 1:56:14 PM6/15/17

to dea...@googlegroups.com

On 15 Jun 2017, at 19:38, Victor Eijkhout <eijk...@tacc.utexas.edu> wrote:

You seem to be saying that I need to add something to a file that is generated by cmake.

Nope. You add things to **your** CMakeList.txt

And even then, it seems to be about a file deal.IIFeatureConfig.cmake which I can't find anywhere.

You need to install the current development version of deal.II (from GitHub master branch).

Then in CMake of **your** project after deal.II is found you do

INCLUDE(${DEAL_II_FEATURE_CONFIG})

to have DEAL_II_TRILINOS_DIR available, which will point to the Trilinos used to build deal.II.

Then you use https://cmake.org/cmake/help/v3.0/command/find_package.html to re-find the Trilinos with extra libraries you need for your project.

Something like

find_package(Trilinos COMPONENTS <extra_libs_you_need> HINTS ${DEAL_II_TRILINOS_DIR})

alternatively you can do

FIND_LIBRARY(Trilinos_extra
NAMES <libname>
HINTS ${DEAL_II_TRILINOS_DIR}/lib
NO_DEFAULT_PATH
)
TARGET_LINK_LIBRARIES(${YOUR_TARGET_NAME}
"${Trilinos_extra}"
)

Victor

PS so there is no simple flag for "add these libraries”?

I don’t know a simpler way with CMake.

Regards,

Denis.

Victor Eijkhout

unread,

Jun 15, 2017, 4:19:28 PM6/15/17

to deal.II User Group

On Thursday, June 15, 2017 at 12:56:14 PM UTC-5, Denis Davydov wrote:

Nope. You add things to **your** CMakeList.txt

You mean that I should edit **your** CMakeLists.txt, right? Not make an extra one of my own?

(Speaking of your: it has a line

# See doc/readme.html and doc/development/cmake.html for further #

That second link does not exist.

You need to install the current development version of deal.II (from GitHub master branch).

Got that.

Then in CMake of **your** project after deal.II is found you do

INCLUDE(${DEAL_II_FEATURE_CONFIG})

You mean "add this line to your CMakeList.txt"?

to have DEAL_II_TRILINOS_DIR available, which will point to the Trilinos used to build deal.II.

Then you use https://cmake.org/cmake/help/v3.0/command/find_package.html to re-find the Trilinos with extra libraries you need for your project.
Something like

find_package(Trilinos COMPONENTS <extra_libs_you_need> HINTS ${DEAL_II_TRILINOS_DIR})

so I put

find_package(Trilinos COMPONENTS /usr/lib/libpython.so HINTS ${DEAL_II_TRILINOS_DIR})

in the CMakeLists.txt file. That leads to:

"-- TRILINOS_INCLUDE_DIRS: *** Required variable "Trilinos_INCLUDE_DIRS" empty ***"

so it seems to break something in what I've specified for Trilinos.

Victor.

Bruno Turcksin

unread,

Jun 15, 2017, 4:40:35 PM6/15/17

to deal.II User Group

Victor,

On Thursday, June 8, 2017 at 10:42:47 AM UTC-4, Victor Eijkhout wrote:

Is there a cmake flag for specifying some extra libraries? Trilinos seems to have been built with a python interface, but deal is somehow not linking the libpython, so the cmake fails.

I am not sure that you and Denis are talking about the same problem. Do you get this problem when you compile deal.II or when you compile your own project on top of deal.II?

Best,

Bruno

Victor Eijkhout

unread,

Jun 15, 2017, 5:19:56 PM6/15/17

to deal.II User Group

On Thursday, June 15, 2017 at 3:40:35 PM UTC-5, Bruno Turcksin wrote:

I am not sure that you and Denis are talking about the same problem. Do you get this problem when you compile deal.II or when you compile your own project on top of deal.II?

Ha. Good point.

I'm trying to install deal. The cmake tries to test the link to trilinos, which bombs on a lot of unresolved reference to python stuff.

Victor.

Bruno Turcksin

unread,

Jun 15, 2017, 5:33:08 PM6/15/17

to dea...@googlegroups.com

2017-06-15 17:19 GMT-04:00 Victor Eijkhout <eijk...@tacc.utexas.edu>:
> I'm trying to install deal. The cmake tries to test the link to trilinos,
> which bombs on a lot of unresolved reference to python stuff.

That's what I thought, don't listen to Denis :p The problem is that
Trilinos botched their TrilinosConfig.cmake. What we do is load the
TrilinosConfig.cmake to get their flags but if they don't put them in,
there is little we can do. What I would do is to add the flags needed
myself. Edit the file
/trilinos_install_directory/lib/cmake/Trilinos/TrilinosConfig.cmake
and add the flag that you need in SET(Trilinos_CXX_COMPILER_FLAGS) or
SET(Trilinos_EXTRA_LD_FLAGS). You should probably back up the file
first...

Best,

Bruno

Wolfgang Bangerth

unread,

Jun 16, 2017, 11:43:27 AM6/16/17

to dea...@googlegroups.com

On 06/15/2017 03:32 PM, Bruno Turcksin wrote:
> The problem is that
> Trilinos botched their TrilinosConfig.cmake. What we do is load the
> TrilinosConfig.cmake to get their flags but if they don't put them in,
> there is little we can do.

To be more specific: When the Trilinos cmake configuration builds the
Trilinos python interfaces, it forgets to either (i) record in the .so
file for that component of Trilinos that it depends on the python libs,
so that whenever you link with that .so the dynamic linker automatically
also pulls in the python libs, or (ii) record somewhere in the
TrilinosConfig.cmake that is installed along with the Trilinos libs that
whenever someone links with the Trilinos .so files that they *also* need
to link with the python libs.

There is little one can do to work around this: the deal.II cmake
configuration would have to figure out that Trilinos was configured to
include the python interfaces, and then it would have to figure out what
specific libraries the Trilinos .so files depend on. That's asking for
too much.

That's why Bruno is recommending this:

> What I would do is to add the flags needed
> myself. Edit the file
> /trilinos_install_directory/lib/cmake/Trilinos/TrilinosConfig.cmake
> and add the flag that you need in SET(Trilinos_CXX_COMPILER_FLAGS) or
> SET(Trilinos_EXTRA_LD_FLAGS).

This would add to the *installed* Trilinos configuration the parts that
are missing, so that the deal.II cmake configuration files know what
else to link with.

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

Denis Davydov

unread,

Jun 17, 2017, 7:13:29 AM6/17/17

to deal.II User Group, bang...@colostate.edu

On Friday, June 16, 2017 at 5:43:27 PM UTC+2, Wolfgang Bangerth wrote:

On 06/15/2017 03:32 PM, Bruno Turcksin wrote:
> The problem is that
> Trilinos botched their TrilinosConfig.cmake. What we do is load the
> TrilinosConfig.cmake to get their flags but if they don't put them in,
> there is little we can do.

What is done in both Spack (https://github.com/LLNL/spack/blob/develop/var/spack/repos/builtin/packages/trilinos/package.py#L578-L590) and Homebrew (https://github.com/Homebrew/homebrew-science/blob/master/trilinos.rb#L286-L289) is to manually remove PyTrilinos from COMPONENTS_LIST in <trilinos_prefix>/lib/cmake/Trilinos/TrilinosConfig.cmake

This issue is known to Trilinos for some time already https://github.com/trilinos/Trilinos/issues/866 , to my knowledge the first mentioning is May 2015

https://github.com/Homebrew/homebrew-science/issues/2148#issuecomment-103614509 .

Regards,

Denis.

Victor Eijkhout

unread,

Jul 6, 2017, 3:22:04 PM7/6/17

to deal.II User Group

Denis:

I can not remove pytrilinos because I actually need it in trilinos.

Bruno:

> add the flag that you need in SET(Trilinos_CXX_COMPILER_FLAGS)

build@build-BLDCHROOT:12.10.1> find . -name \*.cmake -exec grep COMPILER_FLAGS {} \; -print                                                                                                                                                                           SET(Trilinos_CXX_COMPILER_FLAGS "-g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -mkl -L${TACC_PYTHON_LIB} -lpython2.7 -DMPICH_SKIP_MPICXX -std=c++11 -O3 -DNDEBUG")
SET(Trilinos_C_COMPILER_FLAGS "-g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -mkl -L${TACC_PYTHON_LIB} -lpython2.7 -O3 -DNDEBUG")
SET(Trilinos_Fortran_COMPILER_FLAGS " -O3")
./lib/cmake/Trilinos/TrilinosConfig.cmake

but that does not make itself into the deal link line. (Never mind any doubts about that dollar getting expanded correctly; the whole python2.7 is not there in the link line.)

SET(Trilinos_EXTRA_LD_FLAGS)

I'm not finding that in the trilinos documentation, but I've added

  -D Trilinos_EXTRA_LINK_FLAGS="${TACC_PYTHON_LIB}/libpython2.7.so" \
  -D Trilinos_EXTRA_LD_FLAGS="${TACC_PYTHON_LIB}/libpython2.7.so" \

to the cmake line.....

...... and still those python libraries are not showing up. Here is where the CMakeError crashes:

Run Build Command:"/usr/bin/gmake" "cmTC_dd9de/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_dd9de.dir/build.make CMakeFiles/cmTC_dd9de.dir/build
gmake[1]: Entering directory `/tmp/dealii-build/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_dd9de.dir/src.cxx.o
/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin/mpicxx    -DDEAL_II_HAVE_USABLE_FLAGS_DEBUG -fpic -ansi -w2 -wd21 -wd68 -wd135 -wd175 -wd177 -wd191 -wd193 -wd279 -wd327 -wd383 -wd981 -wd1418 -wd1478 -wd1572 -wd2259 -wd\
2536 -wd3415 -wd15531 -wd111 -wd128 -wd185 -wd280 -qopenmp-simd -std=c++14 -std=c++14 -Wno-return-type -Wno-parentheses -O0 -g -gdwarf-2 -grecord-gcc-switches -std=c++14 -g -O0   -o CMakeFiles/cmTC_dd9de.dir/src.cxx.o -c /tmp/dealii-build\
/CMakeFiles/CMakeTmp/src.cxx
Linking CXX executable cmTC_dd9de
/opt/apps/cmake/3.8.2/bin/cmake -E cmake_link_script CMakeFiles/cmTC_dd9de.dir/link.txt --verbose=1

(followed by a link line and then unresolved references).

Victor.

Bruno Turcksin

unread,

Jul 6, 2017, 4:50:01 PM7/6/17

to deal.II User Group

Victor,

On Thursday, July 6, 2017 at 3:22:04 PM UTC-4, Victor Eijkhout wrote:

> add the flag that you need in SET(Trilinos_CXX_COMPILER_FLAGS)

build@build-BLDCHROOT:12.10.1> find . -name \*.cmake -exec grep COMPILER_FLAGS {} \; -print SET(Trilinos_CXX_COMPILER_FLAGS "-g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -mkl -L${TACC_PYTHON_LIB} -lpython2.7 -DMPICH_SKIP_MPICXX -std=c++11 -O3 -DNDEBUG") SET(Trilinos_C_COMPILER_FLAGS "-g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -mkl -L${TACC_PYTHON_LIB} -lpython2.7 -O3 -DNDEBUG") SET(Trilinos_Fortran_COMPILER_FLAGS " -O3") ./lib/cmake/Trilinos/TrilinosConfig.cmake

but that does not make itself into the deal link line. (Never mind any doubts about that dollar getting expanded correctly; the whole python2.7 is not there in the link line.)

Sorry, that's my bad. We actually don't read their linking flags instead we look for the TPLs in SET(Trilinos_TPL_LIBRARIES) That's where we pick up BLAS and Lapack from them so that's where you want to add libpython2.7.so

Best,

Bruno

Victor Eijkhout

unread,

Jul 7, 2017, 3:49:16 AM7/7/17

to deal.II User Group

Yay! Finally compiling.

Mind if I just continue in this ticket even though the question is moving on?

I think I'm correctly specifying a bunch of AVX flags to trilinos. In the deal build they show up in compile lines such as:

[ 63%] Building CXX object source/physics/CMakeFiles/obj_physics_release.dir/transformations.cc.o
cd /tmp/dealii-build/source/physics && /opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin/mpicxx   -I/tmp/dealii-build/source/physics -I/tmp/dealii-build/include -I/admin/build/rpms/BUILD/dealii-git20170615/inclu\
de -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/tbb41_20130401oss/include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/umfpack/UMFPACK/Include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/umfpack/AMD/Inclu\
de -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/muparser_v2_2_4/include -I/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/include -I/home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/include -I/home1/ap\
ps/intel17/impi17_0/trilinos/12.10.1/include -I/opt/apps/intel17/boost/1.64/include -I/opt/apps/intel17/impi17_0/parallel-netcdf/4.3.3.1/x86_64/include -I/opt/apps/intel17/impi17_0/phdf5/1.8.16/x86_64/include -I/opt/intel/compile\
rs_and_libraries/linux/mkl/include -I/home1/apps/intel17/impi17_0/petsc/3.7/include -I/home1/apps/intel17/impi17_0/p4est/2.0/include -I/home1/apps/intel17/impi17_0/slepc/3.7/knightslanding/include -I/home1/apps/intel17/impi17_0/s\
lepc/3.7/include  -fpic -ansi -w2 -wd21 -wd68 -wd135 -wd175 -wd177 -wd191 -wd193 -wd279 -wd327 -wd383 -wd981 -wd1418 -wd1478 -wd1572 -wd2259 -wd2536 -wd3415 -wd15531 -wd111 -wd128 -wd185 -wd280 -qopenmp-simd -std=c++14 -std=c++14\
 -Wno-return-type -Wno-parentheses -O2 -no-ansi-alias -ip -funroll-loops -std=c++14 -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -o CMakeFiles/obj_physics_release.dir/transformations.cc.o -c /admin/build/rpms/BUILD/dealii-git2017061\
5/source/physics/transformations.cc

but then the build bombs with:

[ 63%] Building CXX object source/numerics/CMakeFiles/obj_numerics_debug.dir/vector_tools_boundary.cc.o
cd /tmp/dealii-build/source/numerics && /opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin/mpicxx  -DDEBUG -DTBB_DO_ASSERT=1 -DTBB_USE_DEBUG -I/tmp/dealii-build/source/numerics -I/tmp/dealii-build/include -I/admi\
n/build/rpms/BUILD/dealii-git20170615/include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/tbb41_20130401oss/include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/umfpack/UMFPACK/Include -I/admin/build/rpms/BUILD/d\
ealii-git20170615/bundled/umfpack/AMD/Include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/muparser_v2_2_4/include -I/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/include -I/home1/apps/intel17/impi17_0/p\
etsc/3.7/knightslanding/include -I/home1/apps/intel17/impi17_0/trilinos/12.10.1/include -I/opt/apps/intel17/boost/1.64/include -I/opt/apps/intel17/impi17_0/parallel-netcdf/4.3.3.1/x86_64/include -I/opt/apps/intel17/impi17_0/phdf5\
/1.8.16/x86_64/include -I/opt/intel/compilers_and_libraries/linux/mkl/include -I/home1/apps/intel17/impi17_0/petsc/3.7/include -I/home1/apps/intel17/impi17_0/p4est/2.0/include -I/home1/apps/intel17/impi17_0/slepc/3.7/knightslandi\
ng/include -I/home1/apps/intel17/impi17_0/slepc/3.7/include  -fpic -ansi -w2 -wd21 -wd68 -wd135 -wd175 -wd177 -wd191 -wd193 -wd279 -wd327 -wd383 -wd981 -wd1418 -wd1478 -wd1572 -wd2259 -wd2536 -wd3415 -wd15531 -wd111 -wd128 -wd185\
 -wd280 -qopenmp-simd -std=c++14 -std=c++14 -Wno-return-type -Wno-parentheses -O0 -g -gdwarf-2 -grecod-gcc-switches -std=c++14 -g -O0 -o CMakeFiles/obj_numerics_debug.dir/vector_tools_boundary.cc.o -c /admin/build/rpms/BUILD/dea\
lii-git20170615/source/numerics/vector_tools_boundary.cc
In file included from /admin/build/rpms/BUILD/dealii-git20170615/include/deal.II/matrix_free/matrix_free.h(24),
                 from /admin/build/rpms/BUILD/dealii-git20170615/include/deal.II/numerics/vector_tools.templates.h(43),
                 from /admin/build/rpms/BUILD/dealii-git20170615/source/numerics/vector_tools_boundary.cc(17):
/admin/build/rpms/BUILD/dealii-git20170615/include/deal.II/base/vectorization.h(46): error: #error directive: "Mismatch in vectorization capabilities: AVX was detected during configuration of deal.II and switched on, but it is ap\
parently not available for the file you are trying to compile at the moment. Check compilation flags controlling the instruction set, such as -march=native."
  #error "Mismatch in vectorization capabilities: AVX was detected during configuration of deal.II and switched on, but it is apparently not available for the file you are trying to compile at the moment. Check compilation flags \
controlling the instruction set, such as -march=native."

Message has been deleted

Bruno Turcksin

unread,

Jul 7, 2017, 8:22:14 AM7/7/17

to dea...@googlegroups.com, Martin Kronbichler

This is very strange. Do you set the AVX flags yourself?

@Martin have you ever seen something like that?

Bruno

2017-07-07 3:50 GMT-04:00 Victor Eijkhout <eijk...@tacc.utexas.edu>:
> Hm. That second compile line should be:

> -Wno-parentheses -O0 -g -gdwarf-2 -grecord-gcc-switches -std=c++14 -g -O0 -o

> CMakeFiles/obj_numerics_debug.dir/vector_tools_boundary.cc.o -c
> /admin/build/rpms/BUILD/dea\
>
>
> lii-git20170615/source/numerics/vector_tools_boundary.cc
>
>
> In file included from
> /admin/build/rpms/BUILD/dealii-git20170615/include/deal.II/matrix_free/matrix_free.h(24),
>
>
> from
> /admin/build/rpms/BUILD/dealii-git20170615/include/deal.II/numerics/vector_tools.templates.h(43),
>
>
> from
> /admin/build/rpms/BUILD/dealii-git20170615/source/numerics/vector_tools_boundary.cc(17):
>
>
> /admin/build/rpms/BUILD/dealii-git20170615/include/deal.II/base/vectorization.h(46):
> error: #error directive: "Mismatch in vectorization capabilities: AVX was
> detected during configuration of deal.II and switched on, but it is ap\
>
>
> parently not available for the file you are trying to compile at the moment.
> Check compilation flags controlling the instruction set, such as
> -march=native."
>
>
> #error "Mismatch in vectorization capabilities: AVX was detected during
> configuration of deal.II and switched on, but it is apparently not available
> for the file you are trying to compile at the moment. Check compilation
> flags \
>
>
> controlling the instruction set, such as -march=native."
>
>
>

> --
> The deal.II project is located at http://www.dealii.org/
> For mailing list/forum options, see
> https://groups.google.com/d/forum/dealii?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "deal.II User Group" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/dealii/msEQFzuoxf0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> dealii+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Victor Eijkhout

unread,

Jul 7, 2017, 10:17:28 AM7/7/17

to deal.II User Group, kronbichl...@gmail.com

On Friday, July 7, 2017 at 7:22:14 AM UTC-5, Bruno Turcksin wrote:

This is very strange. Do you set the AVX flags yourself?

Yes. The CMakeOutput.log starts with:

The system is: Linux - 3.10.0-514.21.2.el7.x86_64 - x86_64

Compiling the C compiler identification source file "CMakeCCompilerId.c" succeeded.

Compiler: /opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin/mpicc 

Build flags: -g;-xCORE-AVX2;-axMIC-AVX512,CORE-AVX512;-mkl

Victor Eijkhout

unread,

Jul 7, 2017, 3:42:57 PM7/7/17

to deal.II User Group

It's weird. I just noticed taht I didn't have any -O2 flags in trilinos, so I added. And now the deal install has compile lines with O2 and the AVX flags, but also a bunch of compile lines with O0. And only sometimes those blow up:

[ 64%] Building CXX object source/numerics/CMakeFiles/obj_numerics_debug.dir/solution_transfer_inst3.cc.o

cd /tmp/dealii-build/source/numerics && /opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin/mpicxx  -DDEBUG -DTBB_DO_ASSERT=1 -DTBB_USE_DEBUG -I/tmp/dealii-build/source/numerics -I/tmp/dealii-build/include -I/admin/build/rpms/BU\


ILD/dealii-git20170615/include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/tbb41_20130401oss/include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/umfpack/UMFPACK/Include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/umfp\

ack/AMD/Include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/muparser_v2_2_4/include -I/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/include -I/home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/include -I/home1/apps\

/intel17/impi17_0/trilinos/12.10.1/include -I/opt/apps/intel17/boost/1.64/include -I/opt/apps/intel17/impi17_0/parallel-netcdf/4.3.3.1/x86_64/include -I/opt/apps/intel17/impi17_0/phdf5/1.8.16/x86_64/include -I/opt/intel/compilers_and_libraries/\

linux/mkl/include -I/home1/apps/intel17/impi17_0/petsc/3.7/include -I/home1/apps/intel17/impi17_0/p4est/2.0/include -I/home1/apps/intel17/impi17_0/slepc/3.7/knightslanding/include -I/home1/apps/intel17/impi17_0/slepc/3.7/include  -fpic -ansi -w\

2 -wd21 -wd68 -wd135 -wd175 -wd177 -wd191 -wd193 -wd279 -wd327 -wd383 -wd981 -wd1418 -wd1478 -wd1572 -wd2259 -wd2536 -wd3415 -wd15531 -wd111 -wd128 -wd185 -wd280 -qopenmp-simd -std=c++14 -std=c++14 -Wno-return-type -Wno-parentheses -O0 -g -gdwa\

rf-2 -grecord-gcc-switches -std=c++14 -g -O0 -o CMakeFiles/obj_numerics_debug.dir/solution_transfer_inst3.cc.o -c /admin/build/rpms/BUILD/dealii-git20170615/source/numerics/solution_transfer_inst3.cc

[ 64%] Building CXX object source/numerics/CMakeFiles/obj_numerics_debug.dir/solution_transfer_inst4.cc.o

cd /tmp/dealii-build/source/numerics && /opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin/mpicxx  -DDEBUG -DTBB_DO_ASSERT=1 -DTBB_USE_DEBUG -I/tmp/dealii-build/source/numerics -I/tmp/dealii-build/include -I/admin/build/rpms/BU\


ILD/dealii-git20170615/include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/tbb41_20130401oss/include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/umfpack/UMFPACK/Include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/umfp\

ack/AMD/Include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/muparser_v2_2_4/include -I/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/include -I/home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/include -I/home1/apps\

/intel17/impi17_0/trilinos/12.10.1/include -I/opt/apps/intel17/boost/1.64/include -I/opt/apps/intel17/impi17_0/parallel-netcdf/4.3.3.1/x86_64/include -I/opt/apps/intel17/impi17_0/phdf5/1.8.16/x86_64/include -I/opt/intel/compilers_and_libraries/\

linux/mkl/include -I/home1/apps/intel17/impi17_0/petsc/3.7/include -I/home1/apps/intel17/impi17_0/p4est/2.0/include -I/home1/apps/intel17/impi17_0/slepc/3.7/knightslanding/include -I/home1/apps/intel17/impi17_0/slepc/3.7/include  -fpic -ansi -w\

2 -wd21 -wd68 -wd135 -wd175 -wd177 -wd191 -wd193 -wd279 -wd327 -wd383 -wd981 -wd1418 -wd1478 -wd1572 -wd2259 -wd2536 -wd3415 -wd15531 -wd111 -wd128 -wd185 -wd280 -qopenmp-simd -std=c++14 -std=c++14 -Wno-return-type -Wno-parentheses -O0 -g -gdwa\

rf-2 -grecord-gcc-switches -std=c++14 -g -O0 -o CMakeFiles/obj_numerics_debug.dir/solution_transfer_inst4.cc.o -c /admin/build/rpms/BUILD/dealii-git20170615/source/numerics/solution_transfer_inst4.cc

[ 64%] Building CXX object source/numerics/CMakeFiles/obj_numerics_debug.dir/time_dependent.cc.o

cd /tmp/dealii-build/source/numerics && /opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin/mpicxx  -DDEBUG -DTBB_DO_ASSERT=1 -DTBB_USE_DEBUG -I/tmp/dealii-build/source/numerics -I/tmp/dealii-build/include -I/admin/build/rpms/BU\


ILD/dealii-git20170615/include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/tbb41_20130401oss/include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/umfpack/UMFPACK/Include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/umfp\

ack/AMD/Include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/muparser_v2_2_4/include -I/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/include -I/home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/include -I/home1/apps\

/intel17/impi17_0/trilinos/12.10.1/include -I/opt/apps/intel17/boost/1.64/include -I/opt/apps/intel17/impi17_0/parallel-netcdf/4.3.3.1/x86_64/include -I/opt/apps/intel17/impi17_0/phdf5/1.8.16/x86_64/include -I/opt/intel/compilers_and_libraries/\

linux/mkl/include -I/home1/apps/intel17/impi17_0/petsc/3.7/include -I/home1/apps/intel17/impi17_0/p4est/2.0/include -I/home1/apps/intel17/impi17_0/slepc/3.7/knightslanding/include -I/home1/apps/intel17/impi17_0/slepc/3.7/include  -fpic -ansi -w\

2 -wd21 -wd68 -wd135 -wd175 -wd177 -wd191 -wd193 -wd279 -wd327 -wd383 -wd981 -wd1418 -wd1478 -wd1572 -wd2259 -wd2536 -wd3415 -wd15531 -wd111 -wd128 -wd185 -wd280 -qopenmp-simd -std=c++14 -std=c++14 -Wno-return-type -Wno-parentheses -O0 -g -gdwa\

rf-2 -grecord-gcc-switches -std=c++14 -g -O0 -o CMakeFiles/obj_numerics_debug.dir/time_dependent.cc.o -c /admin/build/rpms/BUILD/dealii-git20170615/source/numerics/time_dependent.cc

[ 64%] Building CXX object source/numerics/CMakeFiles/obj_numerics_debug.dir/vector_tools_boundary.cc.o

cd /tmp/dealii-build/source/numerics && /opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin/mpicxx  -DDEBUG -DTBB_DO_ASSERT=1 -DTBB_USE_DEBUG -I/tmp/dealii-build/source/numerics -I/tmp/dealii-build/include -I/admin/build/rpms/BU\


ILD/dealii-git20170615/include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/tbb41_20130401oss/include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/umfpack/UMFPACK/Include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/umfp\

ack/AMD/Include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/muparser_v2_2_4/include -I/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/include -I/home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/include -I/home1/apps\

/intel17/impi17_0/trilinos/12.10.1/include -I/opt/apps/intel17/boost/1.64/include -I/opt/apps/intel17/impi17_0/parallel-netcdf/4.3.3.1/x86_64/include -I/opt/apps/intel17/impi17_0/phdf5/1.8.16/x86_64/include -I/opt/intel/compilers_and_libraries/\

linux/mkl/include -I/home1/apps/intel17/impi17_0/petsc/3.7/include -I/home1/apps/intel17/impi17_0/p4est/2.0/include -I/home1/apps/intel17/impi17_0/slepc/3.7/knightslanding/include -I/home1/apps/intel17/impi17_0/slepc/3.7/include  -fpic -ansi -w\

2 -wd21 -wd68 -wd135 -wd175 -wd177 -wd191 -wd193 -wd279 -wd327 -wd383 -wd981 -wd1418 -wd1478 -wd1572 -wd2259 -wd2536 -wd3415 -wd15531 -wd111 -wd128 -wd185 -wd280 -qopenmp-simd -std=c++14 -std=c++14 -Wno-return-type -Wno-parentheses -O0 -g -gdwa\

rf-2 -grecord-gcc-switches -std=c++14 -g -O0 -o CMakeFiles/obj_numerics_debug.dir/vector_tools_boundary.cc.o -c /admin/build/rpms/BUILD/dealii-git20170615/source/numerics/vector_tools_boundary.cc

Bruno Turcksin

unread,

Jul 7, 2017, 3:49:17 PM7/7/17

to dea...@googlegroups.com

2017-07-07 15:42 GMT-04:00 Victor Eijkhout <eijk...@tacc.utexas.edu>:
> It's weird. I just noticed taht I didn't have any -O2 flags in trilinos, so
> I added. And now the deal install has compile lines with O2 and the AVX
> flags, but also a bunch of compile lines with O0. And only sometimes those
> blow up:

Can you configure deal.II in Release mode only. I wonder if there is a
problem with the vectorization flags in debug mode.

Bruno

Martin Kronbichler

unread,

Jul 7, 2017, 5:03:56 PM7/7/17

to dea...@googlegroups.com

This code triggers a safety mechanism that I introduced recently (April 26). We also see this on one of the test machines: https://cdash.kyomu.43-1.org/viewBuildError.php?buildid=8758
but I previously thought this is only due to a leftover in the configuration and I did not see it in real life before.

Victor, are you on a clean deal.II build directory without previous content?

Could you also post the last 20 lines or so regarding the instruction extension in the output of the intel compiler you are using with

icpc   -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -dM -E -x c++ /dev/null

(see also the discussion we had a short while ago here: https://github.com/dealii/dealii/pull/4440 )

Best,
Martin

Victor Eijkhout

unread,

Jul 10, 2017, 10:28:50 AM7/10/17

to deal.II User Group

Ok, this was my error. I thought you guys picked u the compiler flags from Trilinos, but turns out I specify them. If I add the AVX instructions to the debug flags (which I don't like!) it compiles fine. Until here:









make -f examples/CMakeFiles/step-53.release.dir/build.make examples/CMakeFiles/step-53.release.dir/build

make[2]: Entering directory `/tmp/dealii-build'

[ 84%] Linking CXX executable ../bin/step-53.release

cd /tmp/dealii-build/examples && /opt/apps/cmake/3.8.2/bin/cmake -E cmake_link_script CMakeFiles/step-53.release.dir/link.txt --verbose=2

/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin/mpicxx    -rdynamic -Wl,--as-needed -shared-intel -rdynamic -qopenmp -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker /opt/intel/compilers_and_libraries_2017\

.4.196/linux/mpi/intel64/lib/release_mt -Xlinker -rpath -Xlinker /opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/lib -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/2017.0.0/intel64/lib/release_mt -Xlinker -rpath -Xlin\

ker /opt/intel/mpi-rt/2017.0.0/intel64/lib  CMakeFiles/step-53.release.dir/step-53/step-53.cc.o  -o ../bin/step-53.release -Wl,-rpath,/tmp/dealii-build/lib:/home1/apps/intel17/impi17_0/trilinos/12.10.1/lib:/opt/apps/intel17/boost\

/1.64/lib:/opt/apps/intel17/impi17_0/parallel-netcdf/4.3.3.1/x86_64/lib:/opt/apps/intel17/python/2.7.13/lib:/home1/apps/intel17/impi17_0/p4est/2.0/lib:/home1/apps/intel17/impi17_0/slepc/3.7/knightslanding/lib:/home1/apps/intel17/\

impi17_0/petsc/3.7/knightslanding/lib:/opt/apps/intel17/impi17_0/phdf5/1.8.16/x86_64/lib: ../lib/libdeal_II.so.9.0.0-pre -lbz2 /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libpytrilinos.so /home1/apps/intel17/impi17_0/trilin\

os/12.10.1/lib/libstokhos_muelu.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstokhos_ifpack2.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstokhos_amesos2.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/\

libstokhos_tpetra.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstokhos_sacado.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstokhos.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libmoochothyra.so /home\

1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libmoocho.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/librythmos.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libmuelu-adapters.so /home1/apps/intel17/impi17_0/trilino\

s/12.10.1/lib/libmuelu-interface.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libmuelu.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/liblocathyra.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/liblocaepetra.s\

o /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/liblocalapack.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libloca.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libnoxepetra.so /home1/apps/intel17/impi17_0/tril\

inos/12.10.1/lib/libnoxlapack.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libnox.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libphalanx.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstk_expreval.so /ho\

me1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstk_mesh_fixtures.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstk_search_util_base.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstk_search.so /home1/apps/in\

tel17/impi17_0/trilinos/12.10.1/lib/libstk_unit_test_utils.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstk_io_util.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstk_io.so /home1/apps/intel17/impi17_0/trilino\

s/12.10.1/lib/libstk_mesh_base.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstk_topology.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstk_util_use_cases.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/l\

ibstk_util_registry.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstk_util_parallel.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstk_util_diag.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstk_util_\

env.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstk_util_util.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_search_util.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_search.so\

 /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_rebalance_utils.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_rebalance.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_io_u\

til.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_io.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_expreval.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_algsup.so /h\

ome1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_mesh_fem.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_mesh_base.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_util_unit_test\

_support.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_util_parallel.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_util_diag.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkcla\

ssic_util_env.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstkclassic_util_util.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libintrepid.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libteko.so /home1/ap\

ps/intel17/impi17_0/trilinos/12.10.1/lib/libstratimikos.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstratimikosbelos.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstratimikosaztecoo.so /home1/apps/intel17/im\

pi17_0/trilinos/12.10.1/lib/libstratimikosamesos.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstratimikosml.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libstratimikosifpack.so /home1/apps/intel17/impi17_0/tril\

inos/12.10.1/lib/libifpack2-adapters.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libifpack2.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libanasazitpetra.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libMo\

deLaplace.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libanasaziepetra.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libanasazi.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libkomplex.so /home1/apps/intel1\

7/impi17_0/trilinos/12.10.1/lib/libmapvarlib.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libfastqlib.a /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libblotlib.a /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libp\

lt.a /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libsvdi_cgi.a /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libsvdi_cdr.a /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libsuplib_cpp.so /home1/apps/intel17/impi17_0/\

trilinos/12.10.1/lib/libsuplib_c.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libsuplib.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libsupes.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libaprepro_lib.so \

/home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libchaco.a /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libio_info_lib.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libIonit.so /home1/apps/intel17/impi17_0/trilinos/\

12.10.1/lib/libIotr.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libIohb.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libIogn.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libIopg.so /home1/apps/intel17/imp\

i17_0/trilinos/12.10.1/lib/libIoexo_fac.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libIofx.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libIoex.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libIoss.so /ho\

me1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libnemesis.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libexoIIv2for32.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libexodus_for.so /home1/apps/intel17/impi17_0/tri\

linos/12.10.1/lib/libexodus.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libamesos2.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libshylu.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libbelostpetra.so /hom\

e1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libbelosepetra.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libbelos.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libml.so /home1/apps/intel17/impi17_0/trilinos/12.10.\

1/lib/libifpack.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libzoltan2.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libpamgen_extras.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libpamgen.so /home1/apps/i\

ntel17/impi17_0/trilinos/12.10.1/lib/libamesos.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libgaleri-xpetra.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libgaleri-epetra.so /home1/apps/intel17/impi17_0/trilinos/1\

2.10.1/lib/libaztecoo.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libdpliris.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libisorropia.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libxpetra-sup.so /home1/\

apps/intel17/impi17_0/trilinos/12.10.1/lib/libxpetra.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libthyratpetra.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libthyraepetraext.so /home1/apps/intel17/impi17_0/trili\

nos/12.10.1/lib/libthyraepetra.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libthyracore.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libepetraext.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libtpetraext.\

so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libtpetrainout.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libtpetra.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libkokkostsqr.so /home1/apps/intel17/impi17_0\

/trilinos/12.10.1/lib/libtpetrakernels.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libtpetraclassiclinalg.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libtpetraclassicnodeapi.so /home1/apps/intel17/impi17_0/trili\

nos/12.10.1/lib/libtpetraclassic.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libtriutils.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libshards.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libzoltan.so /h\

ome1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libepetra.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libsacado.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/librtop.so /home1/apps/intel17/impi17_0/trilinos/12.10.\

1/lib/libteuchoskokkoscomm.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libteuchoskokkoscompat.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libteuchosremainder.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/\

libteuchosnumerics.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libteuchoscomm.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libteuchosparameterlist.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libteuchosco\

re.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libkokkosalgorithms.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libkokkoscontainers.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libkokkoscore.so /home1/app\

s/intel17/impi17_0/trilinos/12.10.1/lib/libtpi.so /home1/apps/intel17/impi17_0/trilinos/12.10.1/lib/libgtest.so /opt/apps/intel17/boost/1.64/lib/libboost_program_options.so -lX11 /opt/apps/intel17/impi17_0/parallel-netcdf/4.3.3.1\

/x86_64/lib/libnetcdf.so -lz /opt/apps/intel17/python/2.7.13/lib/libpython2.7.so /opt/apps/intel17/boost/1.64/lib/libboost_iostreams.so /opt/apps/intel17/boost/1.64/lib/libboost_serialization.so /opt/apps/intel17/boost/1.64/lib/l\

ibboost_system.so /opt/apps/intel17/boost/1.64/lib/libboost_thread.so /opt/apps/intel17/boost/1.64/lib/libboost_regex.so /opt/apps/intel17/boost/1.64/lib/libboost_chrono.so /opt/apps/intel17/boost/1.64/lib/libboost_date_time.so /\

opt/apps/intel17/boost/1.64/lib/libboost_atomic.so /home1/apps/intel17/impi17_0/p4est/2.0/lib/libp4est.so /home1/apps/intel17/impi17_0/p4est/2.0/lib/libsc.so -lm -lmkl_intel_thread /opt/intel/compilers_and_libraries/linux/lib/int\el64/libiomp5.so -lmpifort -lmpi -lmpigi -lifcoremt -limf -lipgo -lirc -lsvml -lirc_s -lc /home1/apps/intel17/impi17_0/slepc/3.7/knightslanding/lib/libslepc.so /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libpetsc.so\









 /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libsuperlu_dist.so /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libcmumps.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libdmumps.a /home1/apps\

/intel17/impi17_0/petsc/3.7/knightslanding/lib/libsmumps.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libzmumps.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libmumps_common.a /home1/apps/intel17/impi\

17_0/petsc/3.7/knightslanding/lib/libpord.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libparmetis.so /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libmetis.so /home1/apps/intel17/impi17_0/petsc/3.7/kni\

ghtslanding/lib/libsuperlu.so /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libHYPRE.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libscalapack.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib\

/libumfpack.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libklu.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libcholmod.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libbtf.a /home1/app\

s/intel17/impi17_0/petsc/3.7/knightslanding/lib/libccolamd.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libcolamd.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libcamd.a /home1/apps/intel17/impi17_0/p\

etsc/3.7/knightslanding/lib/libamd.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libsuitesparseconfig.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libspai.a /home1/apps/intel17/impi17_0/petsc/3.7/knig\

htslanding/lib/libsundials_cvode.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libsundials_nvecserial.a /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libsundials_nvecparallel.a -lmkl_intel_lp64 -lmkl_seq\

uential -lmkl_core -lhwloc /opt/apps/intel17/impi17_0/phdf5/1.8.16/x86_64/lib/libhdf5hl_fortran.so /opt/apps/intel17/impi17_0/phdf5/1.8.16/x86_64/lib/libhdf5_fortran.so /opt/apps/intel17/impi17_0/phdf5/1.8.16/x86_64/lib/libhdf5_h\

l.so /opt/apps/intel17/impi17_0/phdf5/1.8.16/x86_64/lib/libhdf5.so -lssl -lcrypto /home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/lib/libchaco.a -lifport -Wl,-Bstatic -lifcoremt_pic -Wl,-Bdynamic -lmpicxx -lmpifort -lmpi -W\

l,-Bstatic -lmpigi -Wl,-Bdynamic -lrt -lpthread -limf -lsvml -lirng -lm -Wl,-Bstatic -lipgo -ldecimal -Wl,-Bdynamic -lcilkrts -lirc -Wl,-Bstatic -lirc_s -Wl,-Bdynamic -ldl

CMakeFiles/step-53.release.dir/step-53/step-53.cc.o: In function `Step53::AfricaTopography::get_data()':

step-53.cc:(.text+0x1ac9): undefined reference to `boost::iostreams::zlib::default_strategy'

step-53.cc:(.text+0x1b28): undefined reference to `boost::iostreams::zlib::default_compression'

step-53.cc:(.text+0x1b2f): undefined reference to `boost::iostreams::zlib::deflated'

step-53.cc:(.text+0x1d2b): undefined reference to `boost::iostreams::detail::zlib_base::zlib_base()'

step-53.cc:(.text+0x1d49): undefined reference to `boost::iostreams::detail::zlib_base::do_init(boost::iostreams::zlib_params const&, bool, void* (*)(void*, unsigned int, unsigned int), void (*)(void*, void*), void*)'

step-53.cc:(.text+0x1e4d): undefined reference to `boost::iostreams::detail::gzip_header::reset()'

step-53.cc:(.text+0x1e5a): undefined reference to `boost::iostreams::detail::gzip_footer::reset()'

step-53.cc:(.text+0x261a): undefined reference to `boost::iostreams::detail::zlib_base::~zlib_base()'

CMakeFiles/step-53.release.dir/step-53/step-53.cc.o: In function `boost::iostreams::detail::zlib_decompressor_impl<std::allocator<char> >::~zlib_decompressor_impl()':

step-53.cc:(.text._ZN5boost9iostreams6detail22zlib_decompressor_implISaIcEED1Ev[_ZN5boost9iostreams6detail22zlib_decompressor_implISaIcEED1Ev]+0xd): undefined reference to `boost::iostreams::detail::zlib_base::reset(bool, bool)'

step-53.cc:(.text._ZN5boost9iostreams6detail22zlib_decompressor_implISaIcEED1Ev[_ZN5boost9iostreams6detail22zlib_decompressor_implISaIcEED1Ev]+0x15): undefined reference to `boost::iostreams::detail::zlib_base::~zlib_base()'

step-53.cc:(.text._ZN5boost9iostreams6detail22zlib_decompressor_implISaIcEED1Ev[_ZN5boost9iostreams6detail22zlib_decompressor_implISaIcEED1Ev]+0x3f): undefined reference to `boost::iostreams::detail::zlib_base::~zlib_base()'

CMakeFiles/step-53.release.dir/step-53/step-53.cc.o: In function `boost::iostreams::detail::indirect_streambuf<boost::iostreams::basic_gzip_decompressor<std::allocator<char> >, std::char_traits<char>, std::allocator<char>, boost:\

:iostreams::input>::strict_sync()':

step-53.cc:(.text._ZN5boost9iostreams6detail18indirect_streambufINS0_23basic_gzip_decompressorISaIcEEESt11char_traitsIcES4_NS0_5inputEE11strict_syncEv[_ZN5boost9iostreams6detail18indirect_streambufINS0_23basic_gzip_decompressorIS\

aIcEEESt11char_traitsIcES4_NS0_5inputEE11strict_syncEv]+0x76): undefined reference to `boost::iostreams::zlib::sync_flush'

Et cetera. We have boost 1.64. Any idea?

Victor.

Bruno Turcksin

unread,

Jul 10, 2017, 10:44:41 AM7/10/17

to dea...@googlegroups.com

Victor,

2017-07-10 10:28 GMT-04:00 Victor Eijkhout <eijk...@tacc.utexas.edu>:
> Et cetera. We have boost 1.64. Any idea?

Make sure that deal.II is compiled with only one version of boost (the
same than the one you used for Trilinos if you compiled Trilinos with
boost). It already happened to me that I had two versions of boost in
my path and deal mixed them. You can check using ldd on libdealii.so

Best,

Bruno

Victor Eijkhout

unread,

Jul 13, 2017, 3:58:17 PM7/13/17

to deal.II User Group

Our boost was broken. Having repaired it, I get the following:

[ 36%] Built target obj_numerics_inst

make -f source/numerics/CMakeFiles/obj_numerics_release.dir/build.make source/numerics/CMakeFiles/obj_numerics_release.dir/depend


make[2]: Entering directory `/tmp/dealii-build'

cd /tmp/dealii-build && /opt/apps/cmake/3.8.2/bin/cmake -E cmake_depends "Unix Makefiles" /admin/build/rpms/BUILD/dealii-git20170615 /admin/build/rpms/BUILD/dealii-git20170615/source/numerics /tmp/dealii-build /tmp/dealii-build/s\

ource/numerics /tmp/dealii-build/source/numerics/CMakeFiles/obj_numerics_release.dir/DependInfo.cmake --color=

Dependee "/opt/apps/intel17/boost/1.64/include/boost/aligned_storage.hpp" does not exist for depender "source/numerics/CMakeFiles/obj_numerics_release.dir/data_out.cc.o".

Dependee "/opt/apps/intel17/boost/1.64/include/boost/any.hpp" does not exist for depender "source/numerics/CMakeFiles/obj_numerics_release.dir/data_out.cc.o".

Followed by many thousands of lines of "does not exist", all on files that do actually exist:

root@build-BLDCHROOT:SPECS # ls -l /opt/apps/intel17/boost/1.64/include/boost/aligned_storage.hpp

-rw-r--r-- 1 root install 653 Jul  3 16:00 /opt/apps/intel17/boost/1.64/include/boost/aligned_storage.hpp

Any ideas?

We probably have more than one boost in the system, but cmake is finding the right one:

--   BOOST_VERSION: 1.64.0

--   BOOST_LIBRARIES: /opt/apps/intel17/boost/1.64/lib/libboost_iostreams.so;/opt/apps/intel17/boost/1.64/lib/libboost_serialization.so;/opt/apps/intel17/boost/1.64/lib/libboost_system.so;/opt/apps/intel17/boost/1.64/lib/libboost\

_thread.so;/opt/apps/intel17/boost/1.64/lib/libboost_regex.so;/opt/apps/intel17/boost/1.64/lib/libboost_chrono.so;/opt/apps/intel17/boost/1.64/lib/libboost_date_time.so;/opt/apps/intel17/boost/1.64/lib/libboost_atomic.so

--   BOOST_INCLUDE_DIRS: /opt/apps/intel17/boost/1.64/include

--   BOOST_USER_INCLUDE_DIRS: /opt/apps/intel17/boost/1.64/include

-- Found BOOST

Victor Eijkhout

unread,

Jul 13, 2017, 4:09:27 PM7/13/17

to deal.II User Group

PS where it actually crashes is this:









Dependee "/opt/apps/intel17/boost/1.64/include/boost/visit_each.hpp" does not exist for depender "source/numerics/CMakeFiles/obj_numerics_release.dir/vector_tools_rhs.cc.o".

Dependee "/opt/apps/intel17/boost/1.64/include/boost/weak_ptr.hpp" does not exist for depender "source/numerics/CMakeFiles/obj_numerics_release.dir/vector_tools_rhs.cc.o".

Clearing dependencies in "/tmp/dealii-build/source/numerics/CMakeFiles/obj_numerics_release.dir/depend.make".

Scanning dependencies of target obj_numerics_release

make[2]: Leaving directory `/tmp/dealii-build'

make -f source/numerics/CMakeFiles/obj_numerics_release.dir/build.make source/numerics/CMakeFiles/obj_numerics_release.dir/build


make[2]: Entering directory `/tmp/dealii-build'

[ 36%] Building CXX object source/numerics/CMakeFiles/obj_numerics_release.dir/data_out.cc.o

cd /tmp/dealii-build/source/numerics && /opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin/mpicxx   -I/tmp/dealii-build/source/numerics -I/tmp/dealii-build/include -I/admin/build/rpms/BUILD/dealii-git20170615/include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/tbb41_20130401oss/include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/umfpack/UMFPACK/Include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/umfpack/AMD/Include -I/admin/build/rpms/BUILD/dealii-git20170615/bundled/muparser_v2_2_4/include -I/opt/intel/compilers_and_libraries_2017.4.196/linux/mpi/intel64/include -I/home1/apps/intel17/impi17_0/petsc/3.7/knightslanding/include -I/home1/apps/intel17/impi17_0/trilinos/12.10.1/include -I/opt/apps/intel17/boost/1.64/include -I/opt/apps/intel17/impi17_0/parallel-netcdf/4.3.3.1/x86_64/include -I/opt/apps/intel17/impi17_0/phdf5/1.8.16/x86_64/include -I/opt/intel/compilers_and_libraries/linux/mkl/include -I/home1/apps/intel17/impi17_0/petsc/3.7/include -I/home1/apps/intel17/impi17_0/p4est/2.0/include -I/home1/apps/intel17/impi17_0/slepc/3.7/knightslanding/include -I/home1/apps/intel17/impi17_0/slepc/3.7/include  -fpic -ansi -w2 -wd21 -wd68 -wd135 -wd175 -wd177 -wd191 -wd193 -wd279 -wd327 -wd383 -wd981 -wd1418 -wd1478 -wd1572 -wd2259 -wd2536 -wd3415 -wd15531 -wd111 -wd128 -wd185 -wd280 -qopenmp-simd -std=c++14 -std=c++14 -Wno-return-type -Wno-parentheses -O2 -no-ansi-alias -ip -funroll-loops -std=c++14 -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -o CMakeFiles/obj_numerics_release.dir/data_out.cc.o -c /admin/build/rpms/BUILD/dealii-git20170615/source/numerics/data_out.cc

In file included from /admin/build/rpms/BUILD/dealii-git20170615/include/deal.II/base/work_stream.h(21),

                 from /admin/build/rpms/BUILD/dealii-git20170615/source/numerics/data_out.cc(16):

/admin/build/rpms/BUILD/dealii-git20170615/include/deal.II/base/graph_coloring.h(23): catastrophic error: cannot open source file "boost/unordered_map.hpp"

  #include <boost/unordered_map.hpp>

                                    ^

and this file exists:

root@build-BLDCHROOT:SPECS # ls -l /opt/apps/intel17/boost/1.64/include/boost/unordered_map.hpp 

-rw-r--r-- 1 root install 568 Jul 13 20:03 /opt/apps/intel17/boost/1.64/include/boost/unordered_map.hpp

Bruno Turcksin

unread,

Jul 13, 2017, 4:58:43 PM7/13/17

to dea...@googlegroups.com

Victor,

I have never seen anything like that before :-/ The only thing I found
is this (very) old thread without a reply
https://cmake.org/pipermail/cmake/2006-April/008893.html Someone got a
similar problem than yours when updating cmake. So maybe try another
version of cmake.

Best,

Bruno

Victor Eijkhout

unread,

Jul 14, 2017, 11:23:12 AM7/14/17

to deal.II User Group

Thanks for the hint. My cmake was indeed shifting under me. After completely removing the install it all went fine.