Problems compiling examples with deal.ii configured with PETSc

[David White]

Hello all,

Here is a short preface: I am attempting to create a package of many libraries that will be portable to from a smaller to a larger scale supercomputer. Currently these libraries need to all be static where possible. I have nearly every required library installed locally in an EXTLIB directory.

I have successfully compiled all libraries, but I have not been able to compile any of the examples due to some errors pertaining to PETSc. Here is how I configured PETSc:

./configure --with-mpi-dir=$(PWD)/../MPICH/mpich-static --with-shared-libraries=0 --with-single-library --with-blas-lapack-dir=/opt/intel/mkl/lib/intel64

And here is the configuration for deal.ii:

#!/bin/bash

./configure \

--enable-mpi \

--disable-shared \

--disable-threads \

--with-blas='lapack -lrefblas' \

--with-lapack LDFLAGS=-L$EXTLIB/LAPACK/lapack-static/lib \

--with-umfpack \

--with-trilinos=$EXTLIB/TRILINOS/trilinos-static \

--with-petsc=$PETSC_DIR \

--with-petsc-arch=$PETSC_ARCH \

--with-mumps=$EXTLIB/MUMPS/MUMPS_4.10.0 \

--with-scalapack=$EXTLIB/ScaLAPACK/scalapack-2.0.2 \

--with-blacs=$EXTLIB/BLACS/BLACS \

--with-metis=$EXTLIB/METIS/metis-static \

I have also set some environmental variables, as the FAQ section recommended:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:\

$EXTLIB/BLACS/BLACS/LIB:\

$EXTLIB/CMAKE/bin:\

$EXTLIB/LAPACK/lapack-static/lib:\

$EXTLIB/METIS/metis-static/lib:\

$EXTLIB/MPICH/mpich-static/lib:\

$EXTLIB/MUMPS/MUMPS_4.10.0/lib:\

$EXTLIB/PETSc/petsc-3.4.2/linux-gnu/lib:\

$EXTLIB/ScaLAPACK/scalapack-2.0.2:\

$EXTLIB/TRILINOS/trilinos-static/lib

export CC=$EXTLIB/MPICH/mpich-static/bin/mpicc

export CXX=$EXTLIB/MPICH/mpich-static/bin/mpicxx

export PETSC_DIR=$EXTLIB/PETSc/petsc-3.4.2/

export PETSC_ARCH=linux-gnu

The errors I get while compiling the deal.ii examples look something like this: http://pastebin.com/7PM40Exw

Any help is appreciated, please let me know if you have more questions for me.

THANKS!

David

[Matthias Maier]

Am 03. Jul 2013, 17:02 schrieb David White <david.w...@gmail.com>:

>
> Any help is appreciated, please let me know if you have more
> questions for me.
>

When you link a program to static deal.II and PETSc it is crucial to get
the complete link interface.

I spent quite some time to get this right with the new CMake build
system of the current development version.

If you're a bit flexible with the deal.II version you can try the
current development version.


Alternatively, ensure that the resulting link line for the example
steps contains the content of the variable

PETSC_WITH_EXTERNAL_LIB

found in $PETSC_ARCH/conf/petscvariables. This is something, the old
build system does wrong...


Best
Matthias

[david.w...@gmail.com]

Matthias,

 

I attempted to add the contents of that variable to step-1’s link line and there are now many more errors created, each complaining of multiple definitions. We are not able to work with the developmental version either. Any other ideas?

 

Best,

David

 

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "deal.II User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dealii/7kPvGoE6z54/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dealii+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Matthias Maier]

Hi David,

it is hard to help without any further information.

But as a starting point: Is it possible to run a PETSc example code with
static linkage? Is static linkage with deal.II (and without PETSc) fine?

Best,
Matthias

> You received this message because you are subscribed to the Google

> Groups "deal.II User Group" group.

> To unsubscribe from this group and stop receiving emails from it,

[David White]

Hi Matthias,

To answer your question, this is the output after running make test in
the PETSc install directory:
>Running test examples to verify correct installation
>Using
PETSC_DIR=/home/sumuser1/IEL_exec/Curr_versions/David_v3/EXTLIB/PETSc/petsc-3.4.2/
and >PETSC_ARCH=linux-gnu
>C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1
MPI process
>C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2
MPI processes
>Fortran example src/snes/examples/tutorials/ex5f run successfully with
1 MPI process
>Completed test examples

I have a few more questions to ask that you may be able to help me with
as well. Do you know of any successful installs of static deal.II on a
CRAY machine? If not, what working installs are you aware of with static
libraries, and perhaps you could show me some of the configuration
necessary for those.

I have read a little about using CMAKE instead of the configure script.
Perhaps this would be a good alternative since we are using so many
parameters that are not installed typically. Any opinion on this?

I have tried compiling deal.II shared (along with all of the other
libraries shared--LAPACK, BLACS, Trilinos, PETSc, etc), and I am
encountering an error as follows:

>=====deal.II==========optimized====== Linking library: libdeal_II.so
>/usr/bin/ld:
/home/sumuser1/EXTLIB/deal.II/deal.II/lib/contrib/umfpack/amd_i_control.o:
relocation >R_X86_64_32 against `.rodata.str1.1' can not be used when
making a shared object; recompile with >-fPIC
>/home/sumuser1/EXTLIB/deal.II/deal.II/lib/contrib/umfpack/amd_i_control.o: could not read symbols: >Bad value
>collect2: ld returned 1 exit status

I have added -fPIC to the CFLAGS variables in AMD, UMFPACK and BLACS. I
am worried that this is not the issue with the error. Perhaps you could
guide me on how to edit those variables for compiling deal.II? Or
perhaps that is the wrong direction to take for this error?

I hope you can help me with all of this.

Thank you!
David

On 07/05/2013 05:49 PM, Matthias Maier wrote:
> Hi David,
>
> it is hard to help without any further information.
>
> But as a starting point: Is it possible to run a PETSc example code with
> static linkage? Is static linkage with deal.II (and without PETSc) fine?
>
> Best,
> Matthias
>
>
>
> Am 05. Jul 2013, 21:49 schrieb <david.w...@gmail.com>:
>
>> Matthias,
>>

>> I attempted to add the contents of that variable to step-1�s link

[Matthias Maier]

Am 10. Jul 2013, 21:50 schrieb David White <david.w...@gmail.com>:

> Hi Matthias,

>
> I have a few more questions to ask that you may be able to help me
> with as well. Do you know of any successful installs of static deal.II
> on a CRAY machine?

I'm not aware of a successful installation of static deal.II on a CRAY
machine. But maybe Gudio, Timo or Wolfgang know better. :-D

> If not, what working installs are you aware of with
> static libraries, and perhaps you could show me some of the
> configuration necessary for those.
>
> I have read a little about using CMAKE instead of the configure
> script. Perhaps this would be a good alternative since we are using so
> many parameters that are not installed typically. Any opinion on this?

Well, as the one who has actually written the CMake stuff for deal.II my
opinion is a bit biased, isn't it? ;-)

Btw, do you only want to compile deal.II as a static library or is your
goal to compile a completely static executable (using a static deal.II
library)?


The former works quite well with the new CMake build system. It requires
some tweaking by hand, though. Have a look at a small test I just did [1].


For the latter, there is a last remaining issue to mitigate - namely -
CMake does search for shared libs prior to static libs.

I'll test a small patch that introduces a configuration toggle that will

- force CMake to only find static libs
- Build executables (when the cmake functionality provided by deal)
with -static.
- bail out if an external (static) library has a link interface with
dynamic libraries.

>
> I have tried compiling deal.II shared (along with all of the other
> libraries shared--LAPACK, BLACS, Trilinos, PETSc, etc), and I am
> encountering an error as follows:
>
>>=====deal.II==========optimized====== Linking library: libdeal_II.so
>> /usr/bin/ld:
> /home/sumuser1/EXTLIB/deal.II/deal.II/lib/contrib/umfpack/amd_i_control.o:
> relocation >R_X86_64_32 against `.rodata.str1.1' can not be used when
> making a shared object; recompile with >-fPIC
>>/home/sumuser1/EXTLIB/deal.II/deal.II/lib/contrib/umfpack/amd_i_control.o: could not read symbols: >Bad value
>>collect2: ld returned 1 exit status
>
> I have added -fPIC to the CFLAGS variables in AMD, UMFPACK and
> BLACS. I am worried that this is not the issue with the error. Perhaps
> you could guide me on how to edit those variables for compiling
> deal.II? Or perhaps that is the wrong direction to take for this
> error?

This is bundled AMD/UMFPACK, isn't it? The old build system should
already compile these object files with -fPIC (for later linkage to a
shared library.)

Have you changed any compile flags in the build system? Is this a
"vanilla" tarball?

Best,
Matthias




[1] A quick test with external, static openmpi, blas/lapack and petsc:


* Installed a static openmpi version:

./configure --disable-shared --enable-static --without-libnuma --without-memory-manager --prefix=$HOME/temp/test/openmpi-1.4.3-install
make
make install

* Installed PETSc with static blas/lapack and above openmpi:

PATH=$HOME/temp/test/openmpi-1.4.3-install/bin:$PATH PETSC_DIR=$PWD ./configure --download-f-blas-lapack
make PETSC_DIR=/home/tamiko/temp/test/petsc-3.3-p5 PETSC_ARCH=linux-gnu-cxx-opt all

* Configured and installed deal.II with the above libraries:

cmake -DBUILD_SHARED_LIBS=OFF \
-DDEAL_II_ALLOW_AUTODETECTION=OFF
-DDEAL_II_FORCE_BUNDLED_BOOST=ON \
-DWITH_THREADS=ON -DDEAL_II_FORCE_BUNDLED_THREADS=ON \
-DWITH_MPI=ON -DMPI_CXX_COMPILER=/home/tamiko/temp/test/openmpi-1.4.3-install/bin/mpicxx \
-DWITH_LAPACK=ON -DLAPACK_LIBRARIES="/home/tamiko/temp/test/petsc-3.3-p5/linux-gnu-cxx-opt/lib/libflapack.a;/home/tamiko/temp/test/petsc-3.3-p5/linux-gnu-cxx-opt/lib/libfblas.a;/usr/lib64/gcc/x86_64-pc-linux-gnu/4.6.3/libgfortran.a;/usr/lib64/gcc/x86_64-pc-linux-gnu/4.6.3/libquadmath.a;/usr/lib64/libm.a" \
-DWITH_PETSC=ON -DPETSC_DIR=/home/tamiko/temp/test/petsc-3.3-p5 -DPETSC_ARCH=linux-gnu-cxx-opt \
[...]


The resulting link interface (minus some very nasty dynamic linkage from
the link interface from PETSc... *sigh*):

tamiko@northernraven step-4 % ldd step-4
linux-vdso.so.1 (0x00007fffa2bff000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007fdafcf76000)
libm.so.6 => /lib64/libm.so.6 (0x00007fdafcc80000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fdafca7c000)
libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.1/libstdc++.so.6 (0x00007fdafc51c000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdafc305000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdafc0e8000)
libc.so.6 => /lib64/libc.so.6 (0x00007fdafbd3f000)
/lib64/ld-linux-x86-64.so.2 (0x00007fdafd179000)
librt.so.1 => /lib64/librt.so.1 (0x00007fdafbb36000)

[Wolfgang Bangerth]

>> I have a few more questions to ask that you may be able to help me
>> with as well. Do you know of any successful installs of static deal.II
>> on a CRAY machine?
>
> I'm not aware of a successful installation of static deal.II on a CRAY
> machine. But maybe Gudio, Timo or Wolfgang know better. :-D

We tried last year with the old ./configure-based system but failed miserably.
The problem was that that particular machine had a different front end system
than the compute nodes and so one needed to cross-compile -- something
./configure wasn't prepared to do.

Best
W.
--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@math.tamu.edu
www: http://www.math.tamu.edu/~bangerth/

[Matthias Maier]

Hi,

Am 11. Jul 2013, 01:32 schrieb Matthias Maier <matthia...@iwr.uni-heidelberg.de>:

>
> For the latter, there is a last remaining issue to mitigate - namely -
> CMake does search for shared libs prior to static libs.
>
> I'll test a small patch that introduces a configuration toggle that will
>
> - force CMake to only find static libs

Just a small update: I've checked in a small patch that prefers static
libraries over shared libraries if BUILD_SHARED_LIBS is set to off.
This works very well on my system in producing a static library with a
link interface that contains only a minimum of dynamic (system)
libraries.

I've discarded the last two ideas (ensuring that the link interface is
completely static; build static executables). It is not feasible to
support this atm.


Best,
Matthias

[David White]

On 07/10/2013 07:32 PM, Matthias Maier wrote:

If not, what working installs are you aware of with
static libraries, and perhaps you could show me some of the
configuration necessary for those.

I have read a little about using CMAKE instead of the configure
script. Perhaps this would be a good alternative since we are using so
many parameters that are not installed typically. Any opinion on this?

Well, as the one who has actually written the CMake stuff for deal.II my
opinion is a bit biased, isn't it? ;-)

I suppose so! CMake seems to be the best option here for the static libs. I will get back to you after I make some progress there.

Btw, do you only want to compile deal.II as a static library or is your
goal to compile a completely static executable (using a static deal.II
library)?

We hope to have a completely static executable at the end of all of this so that we don't have to worry about executables attempting to reference shared libraries on different nodes of our Cray system.

This is bundled AMD/UMFPACK, isn't it? The old build system should
already compile these object files with -fPIC (for later linkage to a
shared library.)

Have you changed any compile flags in the build system? Is this a
"vanilla" tarball?

I am using a vanilla tarball and the bundles UMFPACK. I don't quite know how we did it, but we don't see that error anymore. Must have been something latent from another install. Currently we have every module working except MUMPS now. We are getting an undefined reference to 'pb_topget' and 'pb_topset'. It looks like an issue with the -DAdd_ flag in ScaLAPACK. Any experience with that error?

Besides that, we are now able to run examples without error with our shared libraries. We will still eventually need to be able to run a static library. We are currently running on a small scale 12-core Linux Red Hat system with gcc 4.4.6 using MPICH2. Is there perhaps a PETSc version that is favorable to use? Maybe a way to use deal.II without PETSc?

Best,
David

[Matthias Maier]

Have a look at the CMake build system, then ;-)

>
> We hope to have a completely static executable at the end of all of
> this so that we don't have to worry about executables attempting to
> reference shared libraries on different nodes of our Cray system.
>

I've commited a series of patches and small bugfixes for the new CMake
build system. It is now possible to configure for a completely static
build with

cmake -DDEAL_II_STATIC_EXECUTABLES=ON <...>

This will change the command line used to produce static binaries and
will prefer static archives for dependency resolution.

After a configuration run have a look at the link interfaces for enabled
features in the generated file 'detailed.log'. Obviously there must no
be any shared libraries around.. (The build system will still detect
shared libraries - if a transitive requirement cannot be fulfilled
otherwise).

Furthermore, have a look at deal.II's CMake documentation for how to
hint for library locations.

With that I was able to link a completely static binary with boost,
blas, lapack, mumps and umfpack yesterday.

>
> the -DAdd_ flag in ScaLAPACK. Any experience with that error?
>

I haven't seen this unresolved symbol, yet. For MUMPS I have
the following link interface:

86 SET(MUMPS_LIBRARIES
87 ${DMUMPS_LIBRARY}
88 ${MUMPS_COMMON_LIBRARY}
89 ${PORD_LIBRARY}
90 ${SCALAPACK_LIBRARIES}
91 ${METIS_LIBRARIES}
92 ${MPI_Fortran_LIBRARIES}
93 )

> Besides that, we are now able to run examples without error with our
> shared libraries. We will still eventually need to be able to run a
> static library. We are currently running on a small scale 12-core
> Linux Red Hat system with gcc 4.4.6 using MPICH2.

> Is there perhaps a PETSc version that is favorable to use?

> Maybe a way to use deal.II without PETSc?

PETSc is not a mandatory build time dependency of deal. If you don't
want to use PETSc have a look at example step 32 that uses Trilinos and
p4est with MPI.

Best,
Matthias

[Timo Heister]

> We tried last year with the old ./configure-based system but failed
> miserably. The problem was that that particular machine had a different
> front end system than the compute nodes and so one needed to cross-compile
> -- something ./configure wasn't prepared to do.

I regularly configure and compile using an interactive session to a)
get on a node with the correct architecture and b) avoid
memory/runtime restictions on the frontend node.


--
Timo Heister
http://www.math.tamu.edu/~heister/

[Wolfgang Bangerth]

> I regularly configure and compile using an interactive session to a)
> get on a node with the correct architecture and b) avoid
> memory/runtime restictions on the frontend node.

But then you need the compilers on the compute nodes. I don't think that was
the case on that machine -- they had a very stripped down OS on the nodes.

[David White]

On 07/12/2013 07:59 AM, Matthias Maier wrote:
> Have a look at the CMake build system, then ;-)

This system is MUCH better. The warning and error information is much
more in depth and there does seem to be much more control.

> After a configuration run have a look at the link interfaces for enabled
> features in the generated file 'detailed.log'. Obviously there must no
> be any shared libraries around.. (The build system will still detect
> shared libraries - if a transitive requirement cannot be fulfilled
> otherwise).
>

> With that I was able to link a completely static binary with boost,
> blas, lapack, mumps and umfpack yesterday.
>> the -DAdd_ flag in ScaLAPACK. Any experience with that error?
> I haven't seen this unresolved symbol, yet. For MUMPS I have
> the following link interface:
>
> 86 SET(MUMPS_LIBRARIES
> 87 ${DMUMPS_LIBRARY}
> 88 ${MUMPS_COMMON_LIBRARY}
> 89 ${PORD_LIBRARY}
> 90 ${SCALAPACK_LIBRARIES}
> 91 ${METIS_LIBRARIES}
> 92 ${MPI_Fortran_LIBRARIES}
> 93 )

Forgive my inexperience, but where did you put this interface? Is this
something that can follow the cmake command? I am now noticing some
issues with my MUMPS installation, so that may be the issue as well, but
I am still curious how to set these variables. Currently my only issue
with my static CMake installation is the following error:
CMake Error: The following variables are used in this project, but they
are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the
CMake files:
MPI_Fortran_LIBRARIES (ADVANCED)
linked by target "report_features" in directory
/home/sumuser1/IEL_exec/Curr_versions/David_v3/EXTLIB/deal.II/deal.II/cmake/scripts
linked by target "deal_II" in directory
/home/sumuser1/IEL_exec/Curr_versions/David_v3/EXTLIB/deal.II/deal.II/source
linked by target "deal_II.g" in directory
/home/sumuser1/IEL_exec/Curr_versions/David_v3/EXTLIB/deal.II/deal.II/source

-- Configuring incomplete, errors occurred!
make: *** [make] Error 1

Any advice here?

> PETSc is not a mandatory build time dependency of deal. If you don't
> want to use PETSc have a look at example step 32 that uses Trilinos and
> p4est with MPI.

If we don't use PETSc or p4est (we don't need dynamic meshing), will we
still be able to build the deal.II libraries?

Thanks,
David

[Wolfgang Bangerth]

>> PETSc is not a mandatory build time dependency of deal. If you don't
>> want to use PETSc have a look at example step 32 that uses Trilinos and
>> p4est with MPI.
> If we don't use PETSc or p4est (we don't need dynamic meshing), will we still
> be able to build the deal.II libraries?

Yes, deal.II is quite happy as a standalone without all of the external
packages it can interface with. The new cmake-based configuration system
simply links with whatever it can find on your system, but you can switch this
off.

Best

[Matthias Maier]

Am 12. Jul 2013, 21:28 schrieb David White <david.w...@gmail.com>:

>> 86 SET(MUMPS_LIBRARIES
>> 87 ${DMUMPS_LIBRARY}
>> 88 ${MUMPS_COMMON_LIBRARY}
>> 89 ${PORD_LIBRARY}
>> 90 ${SCALAPACK_LIBRARIES}
>> 91 ${METIS_LIBRARIES}
>> 92 ${MPI_Fortran_LIBRARIES}
>> 93 )
> Forgive my inexperience, but where did you put this interface? Is this
> something that can follow the cmake command?

This is the interface that is build up in cmake/modules/FindMUMPS.cmake
(the CMake module for finding MUMPS).

If you want to specify your own link interface (e.g. because
autodetection messes up badly), you can provide a comma separated list
on the command line:

cmake -DMUMPS_LIBRARIES="/.../libdmumps.a;/.../libmumps_common.a;etc.,etc." \
-DMUMPS_INCLUDE_DIRS="..."
[...]

Have a look at the documentation

http://www.dealii.org/developer/development/cmake.html#configureoverride

and

http://www.dealii.org/developer/development/cmake.html#advanced
http://www.dealii.org/developer/development/Config.sample

> I am now noticing some
> issues with my MUMPS installation, so that may be the issue as well,
> but I am still curious how to set these variables. Currently my only
> issue with my static CMake installation is the following error:


> CMake Error: The following variables are used in this project, but
> they are set to NOTFOUND.
> Please set them or make sure they are set and tested correctly in the
> CMake files:
> MPI_Fortran_LIBRARIES (ADVANCED)
> linked by target "report_features" in directory
> /home/sumuser1/IEL_exec/Curr_versions/David_v3/EXTLIB/deal.II/deal.II/cmake/scripts
> linked by target "deal_II" in directory
> /home/sumuser1/IEL_exec/Curr_versions/David_v3/EXTLIB/deal.II/deal.II/source
> linked by target "deal_II.g" in directory
> /home/sumuser1/IEL_exec/Curr_versions/David_v3/EXTLIB/deal.II/deal.II/source
>
> -- Configuring incomplete, errors occurred!
> make: *** [make] Error 1
>
> Any advice here?

Good catch. A bug in the build system :-]

The variable "MPI_Fortran_LIBRARIES" was not cleaned up if no MPI
libraries for Fortran could be found.

Please try again with revision 29988

Best,
Matthias

[David White]

> I have tried compiling deal.II shared (along with all of the other
> libraries shared--LAPACK, BLACS, Trilinos, PETSc, etc), and I am
> encountering an error as follows:
>
>>=====deal.II==========optimized====== Linking library: libdeal_II.so
>> /usr/bin/ld:
> /home/sumuser1/EXTLIB/deal.II/deal.II/lib/contrib/umfpack/amd_i_control.o:
> relocation >R_X86_64_32 against `.rodata.str1.1' can not be used when
> making a shared object; recompile with >-fPIC
>>/home/sumuser1/EXTLIB/deal.II/deal.II/lib/contrib/umfpack/amd_i_control.o: could not read symbols: >Bad value
>>collect2: ld returned 1 exit status
>
> I have added -fPIC to the CFLAGS variables in AMD, UMFPACK and
> BLACS. I am worried that this is not the issue with the error. Perhaps
> you could guide me on how to edit those variables for compiling
> deal.II? Or perhaps that is the wrong direction to take for this
> error?

This is bundled AMD/UMFPACK, isn't it? The old build system should
already compile these object files with -fPIC (for later linkage to a
shared library.)

Have you changed any compile flags in the build system? Is this a
"vanilla" tarball?

 I am still having trouble with this error. This is the bundled UMFPACK/AMD and a completely unmodified, "vanilla" tarball. I have tried adding the -fPIC flag all over the deal.II install--in the configure script, the makefiles for UMFPACK, and the Make.global_options.in. No luck. Of course I have also tried not adding the flag anywhere or messing with anything--just using the ./configure script. Perhaps that flag isn't the issue?

Thanks,

David

[David White]

I fixed this error. It turns out adding -fPIC to CFLAGS and CXXFLAGS or even CFLAGSO, etc would not place it in the compile line for UMFPACK and AMD, so I added it to the line directly in the AMD/Source/Makefile.