deal.ii Installation : expand_instantiations Error

[Aaditya Lakshmanan]

Hi Everyone,

                I have been trying to install deal.ii on NERSC Cori cluster for use on the KNL compute nodes using the following procedure :

 

Clone and setup :

git clone --single-branch --branch dealii-9.1 https://github.com/dealii/dealii.git

mkdir build

mkdir dealii_trial

cd build 

 

Build and install on an interactive node :

cmake -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment  -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn -DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_PETSC=ON -DDEAL_II_WITH_P4EST=ON -DDEAL_II_WITH_LAPACK=ON  -DCMAKE_INSTALL_PREFIX=/global/project/projectdirs/m2360/packagesCPFE/dealii_trial/ ../dealii

which completes without any issues after which  

 

make -j 32 install 

 eventually yields the following error :

--------------------------------------------------------------------------

[ 27%] Building CXX object bundled/umfpack/AMD/Source/CMakeFiles/obj_amd_long_debug.dir/amd_defaults.cc.o

/bin/sh: expand_instantiations: command not found

make[2]: *** [source/numerics/CMakeFiles/obj_numerics_inst.dir/build.make:91: source/numerics/data_out_dof_data.inst] Error 127

make[2]: *** Waiting for unfinished jobs....

---------------------------------------------------------------------------

Details on procedure before cloning deal.ii are included below :

(i) I first loaded the necessary modules so that module list displayed 

Currently Loaded Modulefiles:

  1) modules/3.2.11.4                                                    16) rca/2.2.20-7.0.1.1_4.46__g8e3fb5b.ari

  2) altd/2.0                                                                     17) atp/2.1.3

  3) darshan/3.1.7                                                          18) PrgEnv-intel/6.0.5

  4) intel/19.0.3.199                                                       19) craype-mic-knl

  5) craype-network-aries                                              20) cray-mpich/7.7.14

  6) craype/2.6.2                                                             21) craype-hugepages2M

  7) udreg/2.3.2-7.0.1.1_3.31__g8175d3d.ari           22) valgrind/3.15.0

  8) ugni/6.0.14.0-7.0.1.1_7.33__ge78e5b0.ari         23) zlib/1.2.11

  9) pmi/5.0.14                                                                 24) metis/5.1.0

 10) dmapp/7.1.1-7.0.1.1_4.48__g38cf134.ari           25) parmetis/4.0.3

 11) gni-headers/5.0.12.0-7.0.1.1_6.28__g3b1768f.ari  26) superlu-dist/6.1.1

 12) xpmem/2.2.20-7.0.1.1_4.10__g0475745.ari            27) hypre/2.15.1

 13) job/2.2.4-7.0.1.1_3.36__g36b56f4.ari                       28) cmake/3.14.4

 14) dvs/2.12_2.2.156-7.0.1.1_8.9__g5aab709e             29) cray-hdf5/1.10.2.0

 15) alps/6.6.58-7.0.1.1_6.4__g437d88db.ari                  30) petsc-real-64/3.11.2

 

(ii) Set environment variables for dynamic linking 

export XTPE_LINK_TYPE=dynamic

export CRAYPE_LINK_TYPE=dynamic

 

(iii) p4est installation

./configure --prefix=/global/project/projectdirs/m2360/packagesCPFE/p4est --host=x86_64 --enable-vtk-zlib=yes  --enable-mpi=yes F77=ftn FC=ftn CC=cc CXX=CC

make -j 32

make -j 32 install 

export P4EST_DIR=/global/project/projectdirs/m2360/packagesCPFE/p4est

I did initially submit a ticket on NERSC support for assistance with the installation process and after a few iterations, the person(https://github.com/JBlaschke) helping me out asked me to contact the developers. I would really appreciate your help on this. 

Thank you.

 

 

[Jean-Paul Pelteret]

Hi Aaditaya,

I don’t think that I’ve ever seen anyone have this problem before. We build the expand_instantiations binary, and its supposed to be used when expanding files in which we define various combinations of template parameters to instantiate templates within the library. If you’re interested, you can read more about this here

https://dealii.org/current/doxygen/deal.II/CodingConventions.html

under the "Instantiation of templated functions" section.

I’m not at all sure what’s gone wrong, but what might be happening is that by building in parallel with so many processes you somehow manage to get to an object that requires some template parameter expansions before this expand_instantiations binary has finished building. Perhaps you can try building in serial (or with fewer process) for a little bit, and then cancel once you see that you’re definitely at a point where the main library object modules are being compiled.

Perhaps there’s something more that we can do from our side if this is indeed the cause of the problem, but that would be a question for Matthias (who manages our CMake configuration) to answer. I’m not sure if we can somehow delay all other tasks until expand_instantiations binary has been built.

I hope that this helps you to resolve the issue.

Best,

Jean-Paul

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/aaff8931-185f-46d0-8f2c-eb535ae9428en%40googlegroups.com.

[Aaditya Lakshmanan]

Hi Jean-Paul,

   Thank you for your suggestion. I shall try the installation process with fewer processors(perhaps 4) and if even that doesn't work, then I will try it in serial.

Best,

Aaditya

[Jean-Paul Pelteret]

Hi Aaditya,

Great, let us know if that works for you. You should see the expand_instantiations binary appear in the “bin” sub-directory with your build folder. If it still doesn’t appear there, then perhaps you could create a new build directory and then post the configuration and full build logs that come from a fresh build. It could also be that this utility program is failing to build.

Best,

Jean-Paul

To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/0b6d6ab9-f630-4517-a2a0-7106604aa391n%40googlegroups.com.

[Aaditya Lakshmanan]

Hi Jean-Paul,

   I cleaned the previous installation, deleted all the files generated in the build directory and attempted a fresh build process as follows : 

cmake -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn -DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_PETSC=ON -DDEAL_II_WITH_P4EST=ON -DDEAL_II_WITH_LAPACK=ON  -DCMAKE_INSTALL_PREFIX=/global/project/projectdirs/m2360/packagesCPFE/dealii_trial ../dealii

which ran successfully after which I tried installing deal.ii via

make -j 4 install 

which exited with the following error(just as before) :

[ 40%] Building CXX object bundled/muparser_v2_2_4/CMakeFiles/obj_muparser_release.dir/src/muParserTest.cpp.o

[ 40%] Generating data_out_dof_data.inst

/bin/sh: expand_instantiations: command not found

make[2]: *** [source/numerics/CMakeFiles/obj_numerics_inst.dir/build.make:91: source/numerics/data_out_dof_data.inst] Error 127

make[1]: *** [CMakeFiles/Makefile2:3074: source/numerics/CMakeFiles/obj_numerics_inst.dir/all] Error 2

make[1]: *** Waiting for unfinished jobs....

[ 41%] Built target obj_muparser_debug

[ 41%] Generating fe_abf.inst

In the build/bin directory there is a file expand_instantiations of the following type :

expand_instantiations: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, BuildID[sha1]=3ee25c761ad827a88f618b972735b201bc6a2bbe, for GNU/Linux 3.2.0, with debug_info, not stripped

Is that the file you were referring to earlier? I have attached the detailed.log file generated in the build directory. Let me know if you need any more information from me. Thank you for your help.

Best,

Aaditya

[Timo Heister]

Hi Aaditya,

it looks like you are cross compiling due to the cray system you are
on? I don't have first-hand experience, but I know some users have had
success with it in the past.

The error message stems from the fact that we build a small tool
"expand_instantiations" during the build process. If you are cross
compiling, this executable obviously won't work, so the rest of the
build fails.

Take a look at https://dealii.org/9.2.0/developers/porting.html
(search for expand_instantiations) and let us know if that helps.

Best,
Timo


On Fri, Aug 21, 2020 at 2:52 PM Aaditya Lakshmanan

> To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/a2e65ec8-9691-4fbc-9722-182304e229d8n%40googlegroups.com.



--
Timo Heister
http://www.math.clemson.edu/~heister/

[Aaditya Lakshmanan]

Hi Timo,

    Thank you for sharing the link for cross-compilation of dealii. In the same webpage there is a link to information about cmake toolchain information and in that webpage I found the following :

Cross Compiling for the Cray Linux Environment

Cross compiling for compute nodes in the Cray Linux Environment can be done without needing a separate toolchain file. Specifying -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment on the CMake command line will ensure that the appropriate build settings and search paths are configured. The platform will pull its configuration from the current environment variables and will configure a project to use the compiler wrappers from the Cray Programming Environment’s PrgEnv-* modules if present and loaded.

The default configuration of the Cray Programming Environment is to only support static libraries. This can be overridden and shared libraries enabled by setting the CRAYPE_LINK_TYPE environment variable to dynamic.

Running CMake without specifying CMAKE_SYSTEM_NAME will run the configure step in host mode assuming a standard Linux environment. If not overridden, the PrgEnv-* compiler wrappers will end up getting used, which if targeting the either the login node or compute node, is likely not the desired behavior. The exception to this would be if you are building directly on a NID instead of cross-compiling from a login node. If trying to build software for a login node, you will need to either first unload the currently loaded PrgEnv-* module or explicitly tell CMake to use the system compilers in /usr/bin instead of the Cray wrappers. If instead targeting a compute node is desired, just specify the CMAKE_SYSTEM_NAME as mentioned above.

In my case I checked that PrgEnv-intel is one of the modules loaded by default. I did specify CMAKE_SYSTEM_NAME earlier while running cmake, when attempting the installation process on the login node and on the interactive node. I will now try the following :

1) Compiling with PrgEnv-intel on the compute node with CMAKE_SYSTEM_NAME=CrayLinuxEnvironment. 

2) Unloading PrgEnv-intel and compiling on the login node.

Please let me know if you have any other suggestions. Thank you.

Best,

Aaditya

[Aaditya Lakshmanan]

I tried compilation on the compute node with PrgEnv-intel loaded and including CMAKE_SYSTEM_NAME=CrayLinuxEnvironment in the cmake line. Upon make install it yielded the same error that I got earlier :

Scanning dependencies of target obj_numerics_inst

[ 41%] Generating data_out_dof_data.inst

/bin/sh: expand_instantiations: command not found

make[2]: *** [source/numerics/CMakeFiles/obj_numerics_inst.dir/build.make:91: source/numerics/data_out_dof_data.inst] Error 127

make[1]: *** [CMakeFiles/Makefile2:3074: source/numerics/CMakeFiles/obj_numerics_inst.dir/all] Error 2

make[1]: *** Waiting for unfinished jobs....

[ 41%] Building CXX object bundled/muparser_v2_2_4/CMakeFiles/obj_muparser_release.dir/src/muParserInt.cpp.o

[ 41%] Building CXX object bundled/muparser_v2_2_4/CMakeFiles/obj_muparser_release.dir/src/muParserTest.cpp.o

[ 41%] Building CXX object bundled/muparser_v2_2_4/CMakeFiles/obj_muparser_release.dir/src/muParserTokenReader.cpp.o

[ 41%] Built target obj_muparser_debug

[ 41%] Built target obj_muparser_release

make: *** [Makefile:130: all] Error 2

[Matthias Maier]

Hi,

CMake's cross compilation features are unfortunately a bit limited. To
work around this you have to create an executable
"expand_instantiations" separately by doing for example the following in
the source directory:

$ mkdir build-executable
$ cd build-executable
$ cmake ..
$ make expand_instantiations_exe

After that export the executable to your path:

$ export PATH="$PWD/bin:$PATH"

when you now run "expand_instantiations" in your command prompt you
should get a help message:

$ expand_instantiations
Usage:
expand_instantiations class_list_files < in_file > out_file

If all that works out switch to the actual build directory (in the same
terminal) and compile.

This should take care of the following error:

> /bin/sh: expand_instantiations: command not found
> make[2]: ***

Best,
Matthias

[Aaditya Lakshmanan]

Hi Matthias,

    I followed the steps you suggested so that expand_instantiations gave the output :

Usage:
expand_instantiations class_list_files < in_file > out_file 

after which I changed to the regular build directory for building(all performed on an interactive compute node ) as 

cmake -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn -DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_PETSC=ON -DDEAL_II_WITH_P4EST=ON -DDEAL_II_WITH_LAPACK=ON -DCMAKE_INSTALL_PREFIX=/global/project/projectdirs/m2360/packagesCPFE/dealii_trial ../dealii

which ran fine, after which the installation process

make -j 8 install 

resulted in the following error :

[100%] Built target obj_fe_debug
[100%] Linking CXX shared library ../lib/libdeal_II.g.so
/usr/bin/ld: cannot find -lAtpSigHandler
/usr/bin/ld: cannot find -lAtpSigHCommData
make[2]: *** [source/CMakeFiles/deal_II.dir/build.make:1283: lib/libdeal_II.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:2978: source/CMakeFiles/deal_II.dir/all] Error 2

make[1]: *** Waiting for unfinished jobs....

I am not sure what the above means. I contacted support at NERSC for help and he responded as follows :

-----------------------------------------------------------------------------------------------------

The problem is that somewhere in the cmake files it's missing:
```
find_library(ATP_SIGHANDLER_LIBRARY NAMES AtpSigHandler)
```
and then you need to add
```
target_link_libraries(<Target> ${ATP_SIGHANDLER_LIBRARY})
```
to the file that specifies the link step for `libdeal_II.g.so`

I had a look at the cmake files, but it's a pretty complex setup. I think it might be best to reach out to the developers again, and get them to incorporate those changes for cray systems. 

-----------------------------------------------------------------------------------------------------

Thank you for your help.

Best,

Aaditya

 

[Wolfgang Bangerth]

On 8/24/20 3:40 PM, Aaditya Lakshmanan wrote:
>
> I am not sure what the above means. I contacted support at NERSC for help and
> he responded as follows :
> -----------------------------------------------------------------------------------------------------
> The problem is that somewhere in the cmake files it's missing:
> ```
> find_library(ATP_SIGHANDLER_LIBRARY NAMES AtpSigHandler)
> ```
> and then you need to add
> ```
> target_link_libraries(<Target> ${ATP_SIGHANDLER_LIBRARY})
> ```
> to the file that specifies the link step for `libdeal_II.g.so`
>
> I had a look at the cmake files, but it's a pretty complex setup. I think it
> might be best to reach out to the developers again, and get them to
> incorporate those changes for cray systems.
> -----------------------------------------------------------------------------------------------------
>

Aaditya -- that would help, but the question is why that is necessary. What
does this library do, and why do we need to link with it? If this kind of
change is necessary for *every* project you want to install there, then the
system setup is somehow broken -- so I suspect that there is a reason why the
compiler requires linking with this project, but I don't know why. Can you ask
you system support why this is necessary and why the compiler doesn't link
these libraries in itself?

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Johannes Blaschke]

Hi all y'all

I am the consultant helping Aaditya compile deal.II on NERSC's Cori system. Let me start by trying to address Wolfgang's questions:

> that would help, but the question is why that is necessary

Good question! I've never had to do this myself (the cray compiler wrappers usually take care of things like this). I had a look around -- and a chat with some of my colleagues -- and our working theory is that:

1) the cmake script is interfering with the cray compiler wrappers

2) the cmake script is adding libraries that it's detecting in the environment

> What does this library do

AtpSigHandler is part of Cray's ATP support -- Abnormal Termination Processing (ATP) is a system that monitors Cray system user applications. (taken from `man atp_intro`)

> If this kind of change is necessary for *every* project you want to install there, then the system setup is somehow broken

I think if there is something broken with Cori, loads of other users would run into the same problem -- that doesn't mean that everything's right either, for example: one of the libraries used by deal.II might be broken. If there is a way find out where deal.II's build system gets its link-time flags (i.e. is there a verbose mode? can be get more information about how cmake detects its environment?)

> so I suspect that there is a reason why the compiler requires linking with this project, but I don't know why

This is the first build where we needed to add this library explicitly. And looking at past tickets, I can only find a hand full of occurrences of this error.

The ATP module is loaded by default (and the cray compiler wrappers link to it automatically), so I am also confused why this step is necessary. The only things that I can think of are listed above. I agree with them that it makes sense to try and debug why this and find out why their build needs something special. I was hoping that you (i.e. the deal.II developers) would know that this is an issue with Cray systems already. Alas, it seems that this is a new problem for you also.

But I am available to work with you directly -- however I cannot debug this on my own (as the cmake setup is pretty huge!).

Cheers,
Johannes

[Matthias Maier]

Would you mind quickly sending us the file `detailed.log` that you can
find in the build directory? This would help us a lot.

On Mon, Aug 24, 2020, at 16:40 CDT, Aaditya Lakshmanan <aadit...@gmail.com> wrote:

> [100%] Built target obj_fe_debug
> [100%] Linking CXX shared library ../lib/libdeal_II.g.so
> /usr/bin/ld: cannot find -lAtpSigHandler
> /usr/bin/ld: cannot find -lAtpSigHCommData

This implies that the linker (in this case "/usr/bin/ld" was invoked)
was not able to find the library "AtpSigHandler". So, your compiler
toolchain seems to do the right thing by adding

-lAtpSigHandler -lAtpSigHCommData

to the link line but the linker does not know the correct library
location (which would typically be specified by -L<...>). This begs the
question whether "/usr/bin/ld" is actually the correct linker.

> The problem is that somewhere in the cmake files it's missing:
> ```
> find_library(ATP_SIGHANDLER_LIBRARY NAMES AtpSigHandler)

> target_link_libraries(<Target> ${ATP_SIGHANDLER_LIBRARY})
> ```

It looks like that these two commands might work around the problem by
letting cmake add a "-L<...>" to the command line. You can give this a
shot with the following changes to source/CMakeLists.txt:


diff --git a/source/CMakeLists.txt b/source/CMakeLists.txt
index a6d85d8b68..86ca17d073 100644
--- a/source/CMakeLists.txt
+++ b/source/CMakeLists.txt
@@ -158,10 +158,12 @@ FOREACH(build ${DEAL_II_BUILD_TYPES})
)
ENDIF()

+ find_library(ATP_SIGHANDLER_LIBRARY NAMES AtpSigHandler)

TARGET_LINK_LIBRARIES(${DEAL_II_BASE_NAME}${DEAL_II_${build}_SUFFIX}
${DEAL_II_LIBRARIES_${build}}
${DEAL_II_LIBRARIES}
+ ${ATP_SIGHANDLER_LIBRARY}
)

FILE(MAKE_DIRECTORY



But I would rather prefer to figure out why our cmake configurations
(probably?) selects the wrong linker. In any case, please show us the
full contents of detailed.log

Best,
Matthias

[Matthias Maier]

On Tue, Aug 25, 2020, at 00:07 CDT, Johannes Blaschke <jpbla...@lbl.gov> wrote:

> But I am available to work with you directly -- however I cannot debug this
> on my own (as the cmake setup is pretty huge!).

Please have a look at the file `detailed.log` found in the build
directory. It spells out the compiler, all compile flags and the full
configured link interface. Do you see anything odd there?

Aaditya, Johannes: Another point to check quickly is whether CMake
itself does the right thing. Would you mind adding the following three
lines to the main CMakeLists.txt file and examining the output?

diff --git a/CMakeLists.txt b/CMakeLists.txt
index ab24b8f477..dce1dcac08 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -95,6 +95,10 @@ VERBOSE_INCLUDE(${CMAKE_SOURCE_DIR}/cmake/setup_cached_variables.cmake)
PROJECT(deal.II CXX C)
ENABLE_LANGUAGE_OPTIONAL(Fortran)

+message("Link directories: ${CMAKE_CXX_IMPLICIT_LINK_DIRECTORIES}")
+message("Link libraries: ${CMAKE_CXX_IMPLICIT_LINK_LIBRARIES}")
+message(FATAL_ERROR "exit")
+
VERBOSE_INCLUDE(${CMAKE_SOURCE_DIR}/cmake/setup_deal_ii.cmake)

VERBOSE_INCLUDE(${CMAKE_SOURCE_DIR}/cmake/setup_compiler_flags.cmake)


I expect that the AtpSigHandler library should be listed there.

Best,
Matthias

[Aaditya Lakshmanan]

Hi Matthias,

   I have attached the 'detailed.log' file here. I will attempt what you suggested. Thank you.

Best,

Aaditya

[Matthias Maier]

Thanks.

This looks mostly good.

In particular, the Lapack CMake module seems to pick up "AtpSigHandler"
and "AtpSigHCommData":

55 # DEAL_II_WITH_LAPACK set up with external dependencies
56 # LAPACK_WITH_64BIT_BLAS_INDICES = OFF
57 # LAPACK_LINKER_FLAGS =
58 # LAPACK_INCLUDE_DIRS =
59 # LAPACK_USER_INCLUDE_DIRS =
60 # LAPACK_LIBRARIES = /opt/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64/libmkl_intel_lp64.so;/opt/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel6 4/libmkl_intel_thread.so;/opt/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64/libmkl_core.so;/opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64/libiom p5.so;-lm;-ldl;hugetlbfs;AtpSigHandler;AtpSigHCommData;z;hdf5_hl;hdf5;hdf5hl_fortran;hdf5_fortran;mpich_intel;mpichf90_intel;ifport;ifcoremt;imf;m;ipgo;irc;pthread;svml;gcc;gcc_s;irc_s;dl ;c

One thing is peculiar:

108 # DEAL_II_WITH_ZLIB set up with external dependencies
109 # ZLIB_VERSION = 1.2.11
110 # ZLIB_INCLUDE_DIRS = /usr/include
111 # ZLIB_LIBRARIES = /usr/lib64/libz.so

This library does not reside under /global and I doubt it has the right
architecture. Would you mind disabling zlib support for the time being
(-DDEAL_II_WITH_ZLIB=OFF)? Or reconfigure such that the correct zlib
library is picked up?

Maybe this explains why the system linker /usr/bin/ld is used instead of
one from the toolchain under /opt/cray/pe/craype/2.6.2/bin ...

Best,
Matthias

[Johannes Blaschke]

HI Matthias,

thanks for your input -- you might be onto something RE ld

We use a wrapper script for ld:

```

blaschke@cori06:~> which ld
/global/common/cori/software/altd/2.0/bin/ld

```

which is managed by the ALTD module (Automatic Library Tracking Database). Using the system linker (which is what cmake is picking up) isn't usually a problem, but in this case it might be. I checked Aaditya's loaded modules and ALTD is in there, so CMake should be picking it up. How does CMake detect the linker? Is it just a question of adding `-DCMAKE_LINKER=$(which ld)`?

RE zlib: we have a zlib module. So maybe it's just a question of loading that. Does CMake just detect zlib in the PATH? If so, doing a `module load zlib` should be enough. Otherwise, is there a way to point CMake to the zlib module directory?

Cheers,

Johannes

[Aaditya Lakshmanan]

Hi, 

  zlib(version 1.2.11) was already loaded as a module in all my previous compilation attempts. I will try it by setting DEAL_II_WITH_ZLIB=OFF but I am not sure why the path to zlib libraries is picked up from /usr . Thanks for the suggestion.

Best,

Aaditya

[Matthias Maier]

On Wed, Aug 26, 2020, at 13:01 CDT, Aaditya Lakshmanan <aadit...@gmail.com> wrote:

> Hi,
> zlib(version 1.2.11) was already loaded as a module in all my previous
> compilation attempts. I will try it by setting DEAL_II_WITH_ZLIB=OFF but I
> am not sure why the path to zlib libraries is picked up from /usr . Thanks
> for the suggestion.

This is probably simply a faulty zlib module shipped with zlib. In order
to force cmake to pick up the correct library it might be easiest to use
"CMAKE_PREFIX_PATH" like so:

-DCMAKE_PREFIX_PATH=/path/to/the/zlib/module/

If that doesn't help then you can specify zlib by hand:

-DDEAL_II_WITH_ZLIB=ON -DZLIB_INCLUDE_DIRS=/[...]/include -DZLIB_LIBRARIES=/[...]/libz.so

Best,
Matthias

[Matthias Maier]

On Wed, Aug 26, 2020, at 12:46 CDT, Johannes Blaschke <jpbla...@lbl.gov> wrote:

> HI Matthias,
>
> thanks for your input -- you might be onto something RE ld
>
> We use a wrapper script for ld:
> ```
> blaschke@cori06:~> which ld
> /global/common/cori/software/altd/2.0/bin/ld
> ```
> which is managed by the ALTD module (Automatic Library Tracking Database).
> Using the system linker (which is what cmake is picking up) isn't usually a
> problem, but in this case it might be. I checked Aaditya's loaded modules
> and ALTD is in there, so CMake should be picking it up. How does CMake
> detect the linker? Is it just a question of adding `-DCMAKE_LINKER=$(which
> ld)`?

Yes, setting CMAKE_LINKER would be an option. But typically CMake just
invokes the compiler wrapper which should do the right thing...

Best,
Matthias

[Aaditya Lakshmanan]

Hi,

  Compiling deal.ii with DEAL_II_WITH_ZLIB=OFF and subsequently make -j 32 install gives the same error as earlier :

[100%] Building CXX object source/CMakeFiles/deal_II.dir/dummy.cc.o

[100%] Linking CXX shared library ../lib/libdeal_II.so

/usr/bin/ld: cannot find -lAtpSigHandler

/usr/bin/ld: cannot find -lAtpSigHCommData

make[2]: *** [source/CMakeFiles/deal_II.dir/build.make:1274: lib/libdeal_II.so] Error 1

make[1]: *** [CMakeFiles/Makefile2:2879: source/CMakeFiles/deal_II.dir/all] Error 2

make[1]: *** Waiting for unfinished jobs....

So I will try with another zlib module than the one originally used. I have attached the 'detalied.log' file for my most recent attempt.

Best,

Aaditya

[Aaditya Lakshmanan]

Hi Matthias,

   I tried the cmake and make on the login and interactive compute nodes with other versions of zlib modules. In all cases I obtain the same error as I did earlier :

[100%] Building CXX object source/CMakeFiles/deal_II.dir/dummy.cc.o

[100%] Linking CXX shared library ../lib/libdeal_II.so

/usr/bin/ld: cannot find -lAtpSigHandler

/usr/bin/ld: cannot find -lAtpSigHCommData

make[2]: *** [source/CMakeFiles/deal_II.dir/build.make:1274: lib/libdeal_II.so] Error 1

make[1]: *** [CMakeFiles/Makefile2:2879: source/CMakeFiles/deal_II.dir/all] Error 2

make[1]: *** Waiting for unfinished jobs....

For all the versions of zlib modules loaded, in the 'detailed.log' file the lines involving zlib dependencies are as follows :

# DEAL_II_WITH_ZLIB set up with external dependencies

# ZLIB_VERSION = 1.2.11
# ZLIB_INCLUDE_DIRS = /usr/include
# ZLIB_LIBRARIES = /usr/lib64/libz.so 

I performed a make clean and deleted the files in the build directory before running cmake with a different zlib module.

Based on one of your earlier suggestions I included the following lines(with help from Johannes) in 'CMakeLists.txt' :

---------------------------------------------------------------------------------------------------

ENABLE_LANGUAGE_OPTIONAL(Fortran)

# Modification start - Lines included based on Matthias suggestion

message("Link directories: ${CMAKE_CXX_IMPLICIT_LINK_DIRECTORIES}")

message("Link libraries: ${CMAKE_CXX_IMPLICIT_LINK_LIBRARIES}")

message(FATAL_ERROR "exit")

# Modification end

#-------------------------------------------

VERBOSE_INCLUDE(${CMAKE_SOURCE_DIR}/cmake/setup_deal_ii.cmake)

--------------------------------------------------------------------------------------------------- 

and then executing cmake from the main build directory resulted in the following output just before exiting :

Link directories: /opt/cray/pe/hdf5/1.10.2.0/intel/16.0/lib;/opt/cray/dmapp/default/lib64;/opt/cray/pe/mpt/7.7.14/gni/mpich-intel/16.0/lib;/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/lib;/opt/cray/pe/atp/2.1.3/libApp;/global/common/sw/cray/cnl7/haswell/petsc/3.11.2/intel/19.0.3.199/hocbntp/lib;/global/common/sw/cray/cnl7/haswell/hypre/2.15.1/intel/19.0.3.199/2jbygqt/lib;/global/common/sw/cray/cnl7/haswell/superlu-dist/6.1.1/intel/19.0.3.199/vnkv5xn/lib;/global/common/sw/cray/cnl7/haswell/metis/5.1.0/intel/19.0.3.199/fpy6uzj/lib;/global/common/sw/cray/cnl7/haswell/hypre/2.15.1/intel/19.0.3.199/j73evxt/lib;/global/common/sw/cray/cnl7/haswell/superlu-dist/6.1.1/intel/19.0.3.199/4m7cu62/lib;/global/common/sw/cray/cnl7/haswell/metis/5.1.0/intel/19.0.3.199/dkb63mv/lib;/opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64;/opt/intel/compilers_and_libraries_2019.3.199/linux/mkl/lib/intel64;/opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64_lin;/usr/lib64/gcc/x86_64-suse-linux/7;/usr/lib64;/lib64;/usr/x86_64-suse-linux/lib;/lib;/usr/lib

Link libraries: hugetlbfs;AtpSigHandler;AtpSigHCommData;z;hdf5_hl;hdf5;hdf5_hl_cpp;hdf5_cpp;mpich_intel;mpichcxx_intel;stdc++;imf;m;ifcoremt;ifport;pthread;imf;svml;irng;stdc++;m;ipgo;decimal;cilkrts;stdc++;gcc;gcc_s;irc;svml;c;gcc;gcc_s;irc_s;dl;c

CMake Error at CMakeLists.txt:102 (message):

  exit

-- Configuring incomplete, errors occurred!

 

From the Link libraries output it looks like AtpSigHandler and AtpSigHCommData are picked up. Is there anything else I can try? Thank you.

Best,

Aaditya

[Johannes Blaschke]

Hi y'all,

I just wanted to update y'all on what we've learned so far:

1) message("Link directories: ${CMAKE_CXX_IMPLICIT_LINK_DIRECTORIES}")

Shows `/opt/cray/pe/atp/2.1.3/libApp` as one of the entries -- this is the home of AtpSigHandler.so, so (in theory) cmake should find it.

2) Including `find_library(ATP_SIGHANDLER_LIBRARY NAMES AtpSigHandler)` gives the following error:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
ATP_SIGHANDLER_LIBRARY
linked by target "deal_II.g" in directory /global/project/projectdirs/m2360/packagesCPFE/dealii/source
linked by target "deal_II" in directory /global/project/projectdirs/m2360/packagesCPFE/dealii/source

-- Configuring incomplete, errors occurred

When adding `ATP_SIGHANDLER_LIBRARY` to target_link_library in source/CMakeLists.txt

My theory is that find_library is not detecting libAtpSigHandler.so -- but why?

Cheers,

Johannes

[vachanpo...@gmail.com]

Hi Aaditya and Johannes,

My reply might be too late to be of help. About an year ago, I ran into a host of problems for installing dealii on Cray X50. I have posted all the details to get a working installation here https://groups.google.com/g/dealii/c/MCYyPrZNyjg/m/bVS1woSXBgAJ.

The workaround I figured for the issue here (atp libs) is to unload the atp module before configuring. I faced the exact same error and this worked for me. I hope this would be of help.

Thanks,
Vachan