Segmentation with DEAL_II_WITH_THREADS=ON

[Riku Suzuki]

Hi

I'm trying to build the latest deal.II on a cluster consists of 64 nodes that each has two eight-core Intel Xeon E5 processors and 128GB of system memory. I used gcc/7.3.0 and openmpi 3.0.0. Everything is OK if I set DEAL_II_WITH_THREADS=OFF. However, when I use DEAL_II_WITH_THREADS=ON, I get segmentation error with any executable, including step-1, the building completes successfully though. I tried to use DEAL_II_FORCE_BUNDLED_THREADS=ON, but did not change anything.

"gdb bin/step-1.debug" output:

[Thread debugging using libthread_db enabled]

Program received signal SIGSEGV, Segmentation fault.
0x00007fffefe11f68 in tbb::interface9::global_control::active_value(int) ()
    at /gpfs/u/home/barn-shared/dealii/bundled/tbb-2018_U2/src/tbb/tbb_main.cpp:563
563         return controls[param]->active_value();
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 glibc-2.12-1.212.el6.x86_64 hdf5-1.8.5.patch1-10.el6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-65.el6.x86_64 libX11-1.6.4-3.el6.x86_64 libXau-1.0.6-4.el6.x86_64 libcom_err-1.41.12-24.el6.x86_64 libcurl-7.19.7-53.el6_9.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-7.el6.x86_64 libssh2-1.4.2-2.el6_7.1.x86_64 libxcb-1.12-4.el6.x86_64 lua-5.1.4-4.1.el6.x86_64 netcdf-4.1.1-3.el6.5.x86_64 nspr-4.19.0-1.el6.x86_64 nss-3.36.0-8.el6.x86_64 nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64 nss-util-3.36.0-1.el6.x86_64 numactl-2.0.9-2.el6.x86_64 openldap-2.4.40-16.el6.x86_64 openssl-1.0.1e-57.el6.x86_64 zlib-1.2.3-29.el6.x86_64

Could anybody figure out what was wrong? I've attached the logs for your references.

Thank you

Riku

[Bruno Turcksin]

Riku,

What happens if you use only one thread (use export DEAL_II_NUM_THREADS=1) ? Also, can you post the full backtrace from gdb (type bt once the code has crashed)

Best,

Bruno

[Riku Suzuki]

Hi Bruno

Setting DEAL_II_NUM_THREADS=1 in the environment doesn't help. The backtrace is like this:

#0  0x00007fffefe11f68 in tbb::interface9::global_control::active_value(int) ()               [0/70]
    at /gpfs/u/home/barn-shared/dealii/bundled/tbb-2018_U2/src/tbb/tbb_main.cpp:563
#1  0x00007fffefdfd6f3 in tbb::internal::market::global_market(bool, unsigned int, unsigned long)
    ()
    at /gpfs/u/home/barn-shared/dealii/bundled/tbb-2018_U2/include/tbb/global_control.
h:69
#2  0x00007fffefdfd85b in tbb::internal::market::create_arena(int, int, unsigned long) ()
    at /gpfs/u/home/barn-shared/dealii/bundled/tbb-2018_U2/src/tbb/market.cpp:304
#3  0x00007fffefdf85a1 in tbb::internal::governor::init_scheduler(int, unsigned long, bool) ()
    at /gpfs/u/home/barn-shared/dealii/bundled/tbb-2018_U2/src/tbb/governor.cpp:204
#4  0x00007fffefdf868d in tbb::task_scheduler_init::initialize(int, unsigned long) ()
    at /gpfs/u/home/barn-shared/dealii/bundled/tbb-2018_U2/src/tbb/governor.cpp:325
#5  0x00007fffefdf8746 in tbb::task_scheduler_init::initialize(int) ()
    at /gpfs/u/home/barn-shared/dealii/bundled/tbb-2018_U2/src/tbb/governor.cpp:313
#6  0x00007ffff40cc6bb in dealii::MultithreadInfo::set_thread_limit(unsigned int) ()
    at /gpfs/u/home/barn-shared/dealii/source/base/multithread_info.cc:164
#7  0x00007ffff40cca01 in dealii::MultithreadInfo::initialize_multithreading() ()
    at /gpfs/u/home/barn-shared/dealii/source/base/multithread_info.cc:226
#8  0x00007ffff2059fe2 in dealii::FE_Q_Base<dealii::TensorProductPolynomials<3, dealii::Polynomials:
:Polynomial<double> >, 3, 3>::initialize(std::vector<dealii::Point<1, double>, std::allocator<dealii
::Point<1, double> > > const&) ()
    at /gpfs/u/home/barn-shared/dealii/include/deal.II/base/thread_management.h:1961
#9  0x00007ffff205f9ca in dealii::FE_Q<3, 3>::FE_Q(unsigned int) ()
    at /gpfs/u/home/barn-shared/dealii/source/fe/fe_q.cc:66
#10 0x00007ffff30b946a in dealii::MappingQGeneric<3, 3>::MappingQGeneric(unsigned int) ()
    at /gpfs/u/home/barn-shared/dealii/source/fe/mapping_q_generic.cc:2261
#11 0x00007ffff21d175c in dealii::MappingQ1<3, 3>::MappingQ1() ()
    at /gpfs/u/home/barn-shared/dealii/source/fe/mapping_q1.cc:49
#12 0x00007ffff48a6e99 in dealii::MappingQGeneric<3, 3>& dealii::hp::(anonymous namespace)::get_stat
ic_mapping_q1<3, 3>() ()
    at /gpfs/u/home/barn-shared/dealii/source/hp/mapping_collection.cc:93
#13 0x00007ffff48a70b9 in __static_initialization_and_destruction_0 ()
    at /gpfs/u/home/barn-shared/dealii/source/hp/mapping_collection.cc:101
#14 0x00007ffff48a712e in _GLOBAL__sub_I_mapping_collection.cc ()
    at /gpfs/u/home/barn-shared/dealii/source/hp/mapping_collection.cc:111
#15 0x00007ffff5047116 in __do_global_ctors_aux ()
   from /gpfs/u/home/barn-shared/dealii-build/lib/libdeal_II.g.so.9.1.0-pre
#16 0x00007fffffffd9b8 in ?? ()
#17 0x0000000000000001 in ?? ()
#18 0x00007fffffffd9b8 in ?? ()
#19 0x00007fffefcc12eb in _init ()
   from /gpfs/u/home/barn-shared/dealii-build/lib/libdeal_II.g.so.9.1.0-pre
#20 0x000000000000001c in ?? ()
#21 0x000000395c00e985 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#23 0x0000000000000001 in ?? ()
#24 0x00007fffffffdcfa in ?? ()
#25 0x0000000000000000 in ?? ()

Thank you

Riku

[Bruno Turcksin]

Riku,

I don't what is going on here :-/ You should try to install the latest version of TBB and see if that works.

Best,

Bruno

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "deal.II User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dealii/X8wT3X2-NKM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dealii+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Riku Suzuki]

Hi Bruno

I built a TBB from source and linked deal.II to it, the problem still existed. But after I got rid of all the third-party libraries including lapack, petsc, p4est etc. from deal.II expect for TBB, deal.II worked fine! (basic version of deal.II + TBB -> work!) Therefore I am suspect that there were conflicts among deal.II and those dependencies I used (lapack, p4est, petsc, metis). I'll add them back one by one and see what was causing the problem.

Thank you for your suggestion.

Riku

[Riku Suzuki]

Hi Bruno

A manually-installed TBB and a clean build of deal.II solves the problem. Now everything works as expected!

Thank you for your advice.

Riku

[Bruno Turcksin]

Riku,

I am glad it's finally working!

Best,

Bruno

--