Compiled example programs fail with segfault

[develo...@googlemail.com]

Hei,

I am currently trying to run some example programs with a down-scaled version of deal.II (i.e. no additional external libraries, internal boost).
I freshly compiled deal.II with the intel-compiler without issues, but when compiling and running any program which links to deal.II (including the example programs) the program directly crashes with a segfault. Running it in gdb results in

(gdb) backtrace
#0  0x00007fffd22edcee in ?? () from /usr/lib64/libstdc++.so.6
#1  0x00007fffd49a3259 in start_thread () from /usr/lib64/libpthread.so.0
#2  0x00007fffd19532b3 in clone () from /usr/lib64/libc.so.6
What could be the reason, and how can I narrow the issue down?

Thanks!

[Wolfgang Bangerth]

On 9/21/21 11:16 PM, 'develo...@googlemail.com' via deal.II User Group
wrote:

> I am currently trying to run some example programs with a down-scaled
> version of deal.II (i.e. no additional external libraries, internal boost).
> I freshly compiled deal.II with the intel-compiler without issues, but
> when compiling and running any program which links to deal.II (including
> the example programs) the program directly crashes with a segfault.
> Running it in gdb results in

> /(gdb) backtrace
> #0 0x00007fffd22edceein ??() from /usr/lib64/libstdc++.so.6
> #1 0x00007fffd49a3259in start_thread() from /usr/lib64/libpthread.so.0
> #2 0x00007fffd19532b3in clone() from /usr/lib64/libc.so.6/

> What could be the reason, and how can I narrow the issue down?

One problem we often see is when running an executable compiled on one
platform on a different processor. So the question here would be how you
installed deal.II?

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Roland Richter]

Hei, the interesting point is that I'm installing deal.II directly using cmake/make/make install on that machine (after it's my development machine), there is no cross-compilation or something similar involved.

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "deal.II User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dealii/TE43i5-jp2U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dealii+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/1a7c7b0f-87cf-30dd-ec14-aefe515e7025%40colostate.edu.

[Wolfgang Bangerth]

On 9/22/21 2:13 PM, 'Roland Richter' via deal.II User Group wrote:
> Hei, the interesting point is that I'm installing deal.II directly using
> cmake/make/make install on that machine (after it's my development machine),
> there is no cross-compilation or something similar involved.

In that case, I don't really have any good suggestions. This seems to happen
before main() is even called, which may be in the initialization of deal.II --
but in the backtrace there is no deal.II function or initializer. Is the
internet any source of help?

[Matthias Maier]

Would you mind sharing your `detailed.log` (in the build directory) with
us?

Best,
Matthias

[develo...@googlemail.com]

Hei,

I will do that as soon as my re-compilation is done. I re-compiled deal.II with gcc 11.2, and in that case I did not get a segfault. Thus, I will now try to narrow down the issue if it might be related to the intel compiler, or not.

Regards

[develo...@googlemail.com]

Hei,

I did some testing now, with different compilers. I did not get a segfault for GCC 11.2, but it segfaults with ICC 2021.3.0. I attached the detailed.log-files for both deal.II-versions. Could the issue be that some libraries which have been compiled with GCC are linked into my executable, even when being compiled with ICC? ldd for step-1 compiled with ICC gives

linux-vdso.so.1 (0x00007ffe509a2000)
       libdeal_II.g.so.10.0.0-pre => /opt/dealii/lib/libdeal_II.g.so.10.0.0-pre (0x00007f596e345000)
       libz.so.1 => /usr/lib64/libz.so.1 (0x00007f596e302000)
       libumfpack.so.5 => /usr/lib64/libumfpack.so.5 (0x00007f596e23d000)
       libcholmod.so.3 => /usr/lib64/libcholmod.so.3 (0x00007f596e138000)
       libccolamd.so.2 => /usr/lib64/libccolamd.so.2 (0x00007f596e12b000)
       libcolamd.so.2 => /usr/lib64/libcolamd.so.2 (0x00007f596e122000)
       libcamd.so.2 => /usr/lib64/libcamd.so.2 (0x00007f596e114000)
       libsuitesparseconfig.so.5 => /usr/lib64/libsuitesparseconfig.so.5 (0x00007f596e10f000)
       libamd.so.2 => /usr/lib64/libamd.so.2 (0x00007f596e103000)
       libmetis.so.5 => /usr/lib64/libmetis.so.5 (0x00007f596e093000)
       librt.so.1 => /usr/lib64/librt.so.1 (0x00007f596e088000)
       libarpack.so.2 => /usr/lib64/libarpack.so.2 (0x00007f596e03d000)
       libmkl_intel_lp64.so.1 => /opt/intel/mkl/2021.3.0/lib/intel64/libmkl_intel_lp64.so.1 (0x00007f596d2a5000)
       libmkl_intel_thread.so.1 => /opt/intel/mkl/2021.3.0/lib/intel64/libmkl_intel_thread.so.1 (0x00007f596998b000)
       libmkl_core.so.1 => /opt/intel/mkl/2021.3.0/lib/intel64/libmkl_core.so.1 (0x00007f596531c000)
       libiomp5.so => /opt/intel/compiler/2021.3.0/linux/compiler/lib/intel64_lin/libiomp5.so (0x00007f5964f05000)
       libimf.so => /opt/intel/compiler/2021.3.0/linux/compiler/lib/intel64_lin/libimf.so (0x00007f596487d000)
       libm.so.6 => /usr/lib64/libm.so.6 (0x00007f5964739000)
       libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f5964732000)
       libgsl.so.25 => /usr/lib64/libgsl.so.25 (0x00007f596446e000)
       libgslcblas.so.0 => /usr/lib64/libgslcblas.so.0 (0x00007f596442c000)
       libmuparser.so.2.3.2 => /usr/lib64/libmuparser.so.2.3.2 (0x00007f59643c8000)
       libmpicxx.so.12 => /opt/intel/mpi/2021.3.0//lib/libmpicxx.so.12 (0x00007f59641a8000)
       libmpifort.so.12 => /opt/intel/mpi/2021.3.0//lib/libmpifort.so.12 (0x00007f5963dea000)
       libmpi.so.12 => /opt/intel/mpi/2021.3.0//lib/release/libmpi.so.12 (0x00007f59629f3000)
       libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f59629d2000)
       libsvml.so => /opt/intel/compiler/2021.3.0/linux/compiler/lib/intel64_lin/libsvml.so (0x00007f5960e51000)
       libirng.so => /opt/intel/compiler/2021.3.0/linux/compiler/lib/intel64_lin/libirng.so (0x00007f5960ae7000)
       libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f59608ce000)
       libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f59608b3000)
       libintlc.so.5 => /opt/intel/compiler/2021.3.0/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x00007f596063b000)
       libc.so.6 => /usr/lib64/libc.so.6 (0x00007f596046c000)
       /lib64/ld-linux-x86-64.so.2 (0x00007f59936c4000)
       libopenblas_pthreads.so.0 => /usr/lib64/libopenblas_pthreads.so.0 (0x00007f595e6d4000)
       liblapack.so.3 => /usr/lib64/liblapack.so.3 (0x00007f595dfb9000)
       libgomp.so.1 => /usr/lib64/libgomp.so.1 (0x00007f595df72000)
       libblas.so.3 => /usr/lib64/libblas.so.3 (0x00007f595dee6000)
       libgfortran.so.5 => /usr/lib64/libgfortran.so.5 (0x00007f595dc37000)
       libquadmath.so.0 => /usr/lib64/libquadmath.so.0 (0x00007f595dbed000)

while ldd for step-1 compiled with GCC gives

       linux-vdso.so.1 (0x00007ffea18d9000)
       libdeal_II.g.so.10.0.0-pre => /opt/dealii_gcc/lib/libdeal_II.g.so.10.0.0-pre (0x00007fd6d071b000)
       libz.so.1 => /usr/lib64/libz.so.1 (0x00007fd6d06d8000)
       libboost_iostreams.so.1.76.0 => /usr/lib64/libboost_iostreams.so.1.76.0 (0x00007fd6d06c3000)
       libboost_serialization.so.1.76.0 => /usr/lib64/libboost_serialization.so.1.76.0 (0x00007fd6d0682000)
       libboost_system.so.1.76.0 => /usr/lib64/libboost_system.so.1.76.0 (0x00007fd6d067d000)
       libboost_thread.so.1.76.0 => /usr/lib64/libboost_thread.so.1.76.0 (0x00007fd6d065f000)
       libboost_regex.so.1.76.0 => /usr/lib64/libboost_regex.so.1.76.0 (0x00007fd6d0619000)
       libboost_chrono.so.1.76.0 => /usr/lib64/libboost_chrono.so.1.76.0 (0x00007fd6d060f000)
       libboost_date_time.so.1.76.0 => /usr/lib64/libboost_date_time.so.1.76.0 (0x00007fd6d060a000)
       libboost_atomic.so.1.76.0 => /usr/lib64/libboost_atomic.so.1.76.0 (0x00007fd6d0600000)
       libumfpack.so.5 => /usr/lib64/libumfpack.so.5 (0x00007fd6d053b000)
       libcholmod.so.3 => /usr/lib64/libcholmod.so.3 (0x00007fd6d0438000)
       libccolamd.so.2 => /usr/lib64/libccolamd.so.2 (0x00007fd6d0429000)
       libcolamd.so.2 => /usr/lib64/libcolamd.so.2 (0x00007fd6d0420000)
       libcamd.so.2 => /usr/lib64/libcamd.so.2 (0x00007fd6d0412000)
       libsuitesparseconfig.so.5 => /usr/lib64/libsuitesparseconfig.so.5 (0x00007fd6d040d000)
       libamd.so.2 => /usr/lib64/libamd.so.2 (0x00007fd6d0401000)
       libmetis.so.5 => /usr/lib64/libmetis.so.5 (0x00007fd6d0393000)
       librt.so.1 => /usr/lib64/librt.so.1 (0x00007fd6d0386000)
       libarpack.so.2 => /usr/lib64/libarpack.so.2 (0x00007fd6d033b000)
       libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007fd6d0334000)
       libopenblas_pthreads.so.0 => /usr/lib64/libopenblas_pthreads.so.0 (0x00007fd6ce59c000)
       libgsl.so.25 => /usr/lib64/libgsl.so.25 (0x00007fd6ce2d8000)
       libgslcblas.so.0 => /usr/lib64/libgslcblas.so.0 (0x00007fd6ce296000)
       libmuparser.so.2.3.2 => /usr/lib64/libmuparser.so.2.3.2 (0x00007fd6ce232000)
       libmpi.so.40 => /usr/lib64/mpi/gcc/openmpi3/lib64/libmpi.so.40 (0x00007fd6ce132000)
       libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fd6cdf19000)
       libm.so.6 => /usr/lib64/libm.so.6 (0x00007fd6cddd5000)
       libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007fd6cddba000)
       libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007fd6cdd99000)
       libc.so.6 => /usr/lib64/libc.so.6 (0x00007fd6cdbc8000)
       /lib64/ld-linux-x86-64.so.2 (0x00007fd6e71f6000)
       libbz2.so.1 => /usr/lib64/libbz2.so.1 (0x00007fd6cdba9000)
       liblzma.so.5 => /usr/lib64/liblzma.so.5 (0x00007fd6cdb76000)
       liblapack.so.3 => /usr/lib64/liblapack.so.3 (0x00007fd6cd45b000)
       libgomp.so.1 => /usr/lib64/libgomp.so.1 (0x00007fd6cd414000)
       libblas.so.3 => /usr/lib64/libblas.so.3 (0x00007fd6cd386000)
       libgfortran.so.5 => /usr/lib64/libgfortran.so.5 (0x00007fd6cd0d7000)
       libopen-rte.so.40 => /usr/lib64/mpi/gcc/openmpi3/lib64/libopen-rte.so.40 (0x00007fd6cd020000)
       libopen-pal.so.40 => /usr/lib64/mpi/gcc/openmpi3/lib64/libopen-pal.so.40 (0x00007fd6ccf2d000)
       libquadmath.so.0 => /usr/lib64/libquadmath.so.0 (0x00007fd6ccee1000)
       libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00007fd6cced3000)
       libutil.so.1 => /usr/lib64/libutil.so.1 (0x00007fd6ccece000)

Thanks!

[Matthias Maier]

Hi,

On Thu, Sep 23, 2021, at 07:57 CDT, "'develo...@googlemail.com' via deal.II User Group" <dea...@googlegroups.com> wrote:

> Hei,
> I did some testing now, with different compilers. I did not get a segfault
> for GCC 11.2, but it segfaults with ICC 2021.3.0. I attached the
> detailed.log-files for both deal.II-versions.

> Could the issue be that some libraries which have been compiled with
> GCC are linked into my executable, even when being compiled with ICC?

That shouldn't be an issue. The Intel compiler tries to be ABI compliant
with the GCC compiler, so it is perfectly fine to link against libraries
compiled with GCC - in fact, Intel is using the C++ standard library
(libstdc++) shipped by GCC.

But, would you mind quickly checking compilation and execution with ICC
again *without* the "-march=native -mavx2" that you have set somewhere
(CXXFLAGS?)? (After configure and before you can check 'detailed.log"
quickly whether the flags still show up in the line DEAL_II_CXX_FLAGS.)

We had issues with ICC miscompiling with "-march=native" in the past.

Best,
Matthias