Issue with boost serialization and spack?

[Konrad Simon]

Dear deal.ii community,

I am having a little problem and I was wondering if this issue is known. I installed deal.ii on our cluster  and all dependencies build nicely with my chosen compiler (gcc v7.3). BLAS and LAPACK are being built as well as openmpi (the versions on the cluster are not suitable at the moment). However, when building deal.ii I get the output below (I only paste the relevant part). It also happens when I ask preck to build deal.ii with (externally spack built) boost v1.62 and v1.68. 

Could there be a configuration issue with boost in the spack build class of deal.ii? When looking at it I see that some version dependent boost patches are applied but I am not familiar with the details.

Anyone knows this? Is there a workaround?

Thanks and best regards,
Konrad

-- Include /tmp/u290231/spack-stage/dealii-9.1.1-gfk33pgt5rojhujhmuruaahuuzyzq2zm/spack-src/cmake/configure/configure_2_boost.cmake

-- Found Boost: /scratch/cen/numgeo/spack-lib/linux-debian8-broadwell/gcc-7.3.0/boost-1.62.0/include (found suitable version "1.62.0", minimum required is "1.59") found components:  iostreams serialization system thread regex chrono date_time atomic 

--   BOOST_VERSION: 1.62.0

--   BOOST_LIBRARIES: /scratch/cen/numgeo/spack-lib/linux-debian8-broadwell/gcc-7.3.0/boost-1.62.0/lib/libboost_iostreams-mt.so;/scratch/cen/numgeo/spack-lib/linux-debian8-broadwell/gcc-7.3.0/boost-1.62.0/lib/libboost_serialization-mt.so;/scratch/cen/numgeo/spack-lib/linux-debian8-broadwell/gcc-7.3.0/boost-1.62.0/lib/libboost_system-mt.so;/scratch/cen/numgeo/spack-lib/linux-debian8-broadwell/gcc-7.3.0/boost-1.62.0/lib/libboost_thread-mt.so;/scratch/cen/numgeo/spack-lib/linux-debian8-broadwell/gcc-7.3.0/boost-1.62.0/lib/libboost_regex-mt.so;/scratch/cen/numgeo/spack-lib/linux-debian8-broadwell/gcc-7.3.0/boost-1.62.0/lib/libboost_chrono-mt.so;/scratch/cen/numgeo/spack-lib/linux-debian8-broadwell/gcc-7.3.0/boost-1.62.0/lib/libboost_date_time-mt.so;/scratch/cen/numgeo/spack-lib/linux-debian8-broadwell/gcc-7.3.0/boost-1.62.0/lib/libboost_atomic-mt.so

--   BOOST_INCLUDE_DIRS: /scratch/cen/numgeo/spack-lib/linux-debian8-broadwell/gcc-7.3.0/boost-1.62.0/include

--   BOOST_USER_INCLUDE_DIRS: /scratch/cen/numgeo/spack-lib/linux-debian8-broadwell/gcc-7.3.0/boost-1.62.0/include

-- Found BOOST

-- Performing Test BOOST_IOSTREAMS_USABLE

-- Performing Test BOOST_IOSTREAMS_USABLE - Success

-- Performing Test BOOST_SERIALIZATION_USABLE

-- Performing Test BOOST_SERIALIZATION_USABLE - Failed

-- The externally provided Boost.Serialization library failed to pass a crucial test. 

Therefore, the bundled boost package is used. 

The configured testing project can be found at 

/tmp/u290231/spack-stage/dealii-9.1.1-gfk33pgt5rojhujhmuruaahuuzyzq2zm/spack-build/cmake/configure/TestBoostBugWorkdir

-- DEAL_II_WITH_BOOST has unmet external dependencies.

CMake Error at cmake/macros/macro_configure_feature.cmake:112 (MESSAGE):

  

  Could not find the boost library!

  The externally provided Boost.Serialization library failed to pass a

  crucial test.Please ensure that a suitable boost library is installed on

  your computer.

  If the library is not at a default location, either provide some hints for

  autodetection,

      $ BOOST_DIR="..." cmake <...>

      $ cmake -DBOOST_DIR="..." <...>

  or set the relevant variables by hand in ccmake.

  Alternatively you may choose to compile the bundled library of boost by

  setting DEAL_II_ALLOW_BUNDLED=on or DEAL_II_FORCE_BUNDLED_BOOST=on.

Call Stack (most recent call first):

  cmake/macros/macro_configure_feature.cmake:269 (FEATURE_ERROR_MESSAGE)

  cmake/configure/configure_2_boost.cmake:237 (CONFIGURE_FEATURE)

  cmake/macros/macro_verbose_include.cmake:19 (INCLUDE)

  CMakeLists.txt:124 (VERBOSE_INCLUDE)

[Denis Davydov]

Hi,

Boost serializstion was broken in several versions of boost, that’s why it’s blacklisted in Spack package for some and has a patch for others. It looks like your setup of compiler/platform reveal that it’s broken in another case as well. To be sure you can check CMake log in build folder where a small test is run to check serialization. If the output looks valid (it is indeed broken), I would try a different version of boost.

Denis.

[Denis Davydov]

p.s. long discussion on serialization bug in Boost is here https://github.com/boostorg/serialization/pull/79
Frankly, I would not be surprized that 1.62 has issues as well...

[Konrad Simon]

Thank you, Denis. I use a pretty stupid (but simple) workaround: I setup and compile deal.ii myself since all dependencies are compiled and use the cmake command used by spack. That works. And I do not get the serialization error.

However, now my code runs on the machine I installed it on. But once I use slurm to distribute the job across nodes I get "illegal instruction" erros. Frustrating.

Nevertheless, thanks for your hint.

Best,
Konrad

[Denis Davydov]

Hi Konrad,

On Monday, September 30, 2019 at 3:55:20 PM UTC+2, Konrad Simon wrote:

Thank you, Denis. I use a pretty stupid (but simple) workaround: I setup and compile deal.ii myself since all dependencies are compiled and use the cmake command used by spack. That works. And I do not get the serialization error.

now that is strange. Are you sure you pick up the same boost?

Could you post the original error from CMake error logs that shows how serialization test fails?

 

However, now my code runs on the machine I installed it on. But once I use slurm to distribute the job across nodes I get "illegal instruction" erros. Frustrating.

With HPC I used to have access to they did not use slurm, so I can't really comment here.

If that's during running quick tests or so, it could be related to a wrong MPIEXEC pickedup by deal.II config.

I wanted to fix that in Spack https://github.com/spack/spack/pull/11142 

but apparently this solution may not be fully functional for Slurm.

Denis.

[Konrad Simon]

Thank you, Denis. I use a pretty stupid (but simple) workaround: I setup and compile deal.ii myself since all dependencies are compiled and use the cmake command used by spack. That works. And I do not get the serialization error.

now that is strange. Are you sure you pick up the same boost?

Could you post the original error from CMake error logs that shows how serialization test fails?

 

Cmake in my little modification seems to pick up the right boost. (for error log with boost see above)

 

DEAL_II_WITH_BOOST set up with external dependencies
#            BOOST_VERSION = 1.70.0
#            BOOST_DIR = /scratch/cen/numgeo/spack-lib/linux-debian8-x86_64/gcc-7.3.0/boost-1.70.0
#            BOOST_CXX_FLAGS = -Wno-unused-local-typedefs
#            BOOST_DEFINITIONS = BOOST_NO_AUTO_PTR
#            BOOST_USER_DEFINITIONS = BOOST_NO_AUTO_PTR
#            BOOST_INCLUDE_DIRS = /scratch/cen/numgeo/spack-lib/linux-debian8-x86_64/gcc-7.3.0/boost-1.70.0/include
#            BOOST_USER_INCLUDE_DIRS = /scratch/cen/numgeo/spack-lib/linux-debian8-x86_64/gcc-7.3.0/boost-1.70.0/include
#            BOOST_LIBRARIES = /scratch/cen/numgeo/spack-lib/linux-debian8-x86_64/gcc-7.3.0/boost-1.70.0/lib/libboost_iostreams-mt.so;/scratch/cen/numgeo/spack-lib/lin
ux-debian8-x86_64/gcc-7.3.0/boost-1.

 

However, now my code runs on the machine I installed it on. But once I use slurm to distribute the job across nodes I get "illegal instruction" erros. Frustrating.

With HPC I used to have access to they did not use slurm, so I can't really comment here.

If that's during running quick tests or so, it could be related to a wrong MPIEXEC pickedup by deal.II config.

I wanted to fix that in Spack https://github.com/spack/spack/pull/11142 

but apparently this solution may not be fully functional for Slurm.

I guess I have a clue why I get the error. My backend nodes run on a different architecture. The openmpi version (i.e. mpirun) is the correct one. This is made sure in the slurm script.

 

Best,
Konrad

[Bruno Turcksin]

Konrad

On Wednesday, October 2, 2019 at 5:50:53 AM UTC-4, Konrad Simon wrote:

 

However, now my code runs on the machine I installed it on. But once I use slurm to distribute the job across nodes I get "illegal instruction" erros. Frustrating.

With HPC I used to have access to they did not use slurm, so I can't really comment here.

If that's during running quick tests or so, it could be related to a wrong MPIEXEC pickedup by deal.II config.

I wanted to fix that in Spack https://github.com/spack/spack/pull/11142 

but apparently this solution may not be fully functional for Slurm.

I guess I have a clue why I get the error. My backend nodes run on a different architecture. The openmpi version (i.e. mpirun) is the correct one. This is made sure in the slurm script.

That's always a pain to deal with. If you can, I would get an interactive session on a compute node and compile everything on the compute node. You can also compile everything in batch mode or compute on the login node and pass the flag for the architecture of the compute node.

Best,

Bruno

[Konrad Simon]

I guess I have a clue why I get the error. My backend nodes run on a different architecture. The openmpi version (i.e. mpirun) is the correct one. This is made sure in the slurm script.

That's always a pain to deal with. If you can, I would get an interactive session on a compute node and compile everything on the compute node. You can also compile everything in batch mode or compute on the login node and pass the flag for the architecture of the compute node.

Thank you, that's what I am trying. I will find a solution the question is jsut how much time it will eat...

Konrad

[Denis Davydov]

On Wednesday, October 2, 2019 at 11:50:53 AM UTC+2, Konrad Simon wrote:

Thank you, Denis. I use a pretty stupid (but simple) workaround: I setup and compile deal.ii myself since all dependencies are compiled and use the cmake command used by spack. That works. And I do not get the serialization error.

now that is strange. Are you sure you pick up the same boost?

Could you post the original error from CMake error logs that shows how serialization test fails?

 

Cmake in my little modification seems to pick up the right boost. (for error log with boost see above)

For the error log I meant  CMakeError.log which should contain the failing serialization test. 

What you posted is the output of CMake config which only states that the serialization test failed, but does not give details as to why.

But that's not iimportant as 1.70 works for you.

 

DEAL_II_WITH_BOOST set up with external dependencies
#            BOOST_VERSION = 1.70.0
#            BOOST_DIR = /scratch/cen/numgeo/spack-lib/linux-debian8-x86_64/gcc-7.3.0/boost-1.70.0
#            BOOST_CXX_FLAGS = -Wno-unused-local-typedefs
#            BOOST_DEFINITIONS = BOOST_NO_AUTO_PTR
#            BOOST_USER_DEFINITIONS = BOOST_NO_AUTO_PTR
#            BOOST_INCLUDE_DIRS = /scratch/cen/numgeo/spack-lib/linux-debian8-x86_64/gcc-7.3.0/boost-1.70.0/include
#            BOOST_USER_INCLUDE_DIRS = /scratch/cen/numgeo/spack-lib/linux-debian8-x86_64/gcc-7.3.0/boost-1.70.0/include
#            BOOST_LIBRARIES = /scratch/cen/numgeo/spack-lib/linux-debian8-x86_64/gcc-7.3.0/boost-1.70.0/lib/libboost_iostreams-mt.so;/scratch/cen/numgeo/spack-lib/lin
ux-debian8-x86_64/gcc-7.3.0/boost-1.

but that is different boos version from what you posted above, namely 1.70.0 instead of 1.62. I guess that would also work if you install dealii + 1.70 via Spack without manual labour. 

 

Regards,

Denis.

[Konrad Simon]

Hi Denis,

I don't have the build folders any more so I can not post the error log. But the error (using spack) occurred with both boost versions. I will post something once I will find a solution.

At any rate, thank you for your help.

Best,
Konrad