Compilation with ICC18 leads to several undefined AVX/SSE-functions

Maxi Miller

unread,

Feb 1, 2019, 3:29:43 AM2/1/19

to deal.II User Group

I tried to compile deal.II using ICC 18, but during compilation I got several errors, such as

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1489): error: no operator "+=" matches these operands
            operand types are: __m256d += const __m256d

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1504): error: no operator "-=" matches these operands
            operand types are: __m256d -= const __m256d

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1518): error: no operator "*=" matches these operands
            operand types are: __m256d *= const __m256d

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1533): error: no operator "/=" matches these operands
            operand types are: __m256d /= const __m256d

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1549): error: identifier "_mm256_loadu_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1562): error: identifier "_mm256_storeu_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1574): error: identifier "_mm256_stream_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1598): error: identifier "_mm_loadu_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1600): error: identifier "_mm256_i32gather_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1644): error: identifier "_mm256_sqrt_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1659): error: identifier "_mm256_set1_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1661): error: identifier "_mm256_andnot_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1674): error: identifier "_mm256_max_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1687): error: identifier "_mm256_min_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1728): error: identifier "_mm256_loadu_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1732): error: identifier "_mm256_permute2f128_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1736): error: identifier "_mm256_unpacklo_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1737): error: identifier "_mm256_unpackhi_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1770): error: identifier "_mm256_permute2f128_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1774): error: identifier "_mm256_unpacklo_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1775): error: identifier "_mm256_unpackhi_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1784): error: identifier "_mm256_loadu_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1784): error: identifier "_mm256_add_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1785): error: identifier "_mm256_storeu_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1795): error: identifier "_mm256_storeu_pd" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1832): error: identifier "_mm256_set1_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1869): error: no operator "+=" matches these operands
            operand types are: __m256 += const __m256

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1884): error: no operator "-=" matches these operands
            operand types are: __m256 -= const __m256

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1898): error: no operator "*=" matches these operands
            operand types are: __m256 *= const __m256

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1913): error: no operator "/=" matches these operands
            operand types are: __m256 /= const __m256

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1929): error: identifier "_mm256_loadu_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1942): error: identifier "_mm256_storeu_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1954): error: identifier "_mm256_stream_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1978): error: identifier "_mm256_loadu_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(1980): error: identifier "_mm256_i32gather_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2024): error: identifier "_mm256_sqrt_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2039): error: identifier "_mm256_set1_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2041): error: identifier "_mm256_andnot_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2054): error: identifier "_mm256_max_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2067): error: identifier "_mm256_min_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2103): error: identifier "_mm_loadu_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2113): error: identifier "_mm256_set1_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2114): error: identifier "_mm256_insertf128_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2122): error: identifier "_mm256_shuffle_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2156): error: identifier "_mm256_shuffle_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2164): error: identifier "_mm256_extractf128_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2178): error: identifier "_mm_loadu_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2178): error: identifier "_mm_add_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2179): error: identifier "_mm_storeu_ps" is undefined

~/Downloads_ICC/dealii/include/deal.II/base/vectorization.h(2197): error: identifier "_mm_storeu_ps" is undefined

Compilation with GCC works fine. Why did I get those errors, and how can I fix them? I attached the detailed.log-file for debugging

detailed.log

Martin Kronbichler

unread,

Feb 1, 2019, 3:46:44 AM2/1/19

to dea...@googlegroups.com

Dear Maxi,

I have not yet look at all details, but what strikes my attention is that you added `-march=native` to your C++ compile flags. That is a gcc thing, so I don't know what Intel makes out of that. Can you try what happens if you replace it by `-xhost` which is (mostly) the Intel equivalent?

Another thing that I still do not understand is why this passed the configuration step as an AVX compile (DEAL_II_VECTORIZATION_LEVEL = 2), since it only seems that SSE2 is available. It might be that there is something left from an old installation and you need to delete `CMakeCache.txt` and `CMakeFiles/`.

Best,
Martin

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Maxi Miller

unread,

Feb 1, 2019, 4:27:30 AM2/1/19

to deal.II User Group

I removed those files, and rebuild using -xhost, but I still get the same errors. AVX should be available, after I am compiling on a Xeon E5-2695 v4, which definitely has AVX (and the intel compiler should have that, too). Test programs using those functions work fine.

Maxi Miller

unread,

Feb 1, 2019, 4:57:22 AM2/1/19

to deal.II User Group

It looks like as if this compilation line is responsible:

cd ~/Downloads_ICC/dealii/build/source/lac && /share/apps/software/Core/CUDA/10.0.130/bin/nvcc -ccbin=/share/apps/software/Compiler/intel/2018.3.222-GCC-7.3.0-2.30/impi/2018.3.222/bin64/mpiicpc  -I~/Downloads_ICC/dealii/build/source/lac -I~/Downloads_ICC/dealii/build/include -I~/Downloads_ICC/dealii/include -I~/Downloads_ICC/dealii/bundled/umfpack/UMFPACK/Include -I~/Downloads_ICC/dealii/bundled/umfpack/AMD/Include -I~/Downloads_ICC/dealii/bundled/muparser_v2_2_4/include -isystem=/share/apps/software/Core/CUDA/10.0.130/include -isystem=/share/apps/software/Core/icc/2018.3.222-GCC-7.3.0-2.30/compilers_and_libraries_2018.3.222/linux/tbb/include -isystem=~/local_opt_ICC/boost/include -isystem=~/local_opt_ICC/ginkgo/include -isystem=/share/apps/software/Compiler/intel/2018.3.222-GCC-7.3.0-2.30/METIS/5.1.0/include -isystem=~/local_opt_ICC/trilinos/include -isystem=/share/apps/software/MPI/intel/2018.3.222-GCC-7.3.0-2.30/impi/2018.3.222/HDF5/1.10.2/include -isystem=/share/apps/software/MPI/intel/2018.3.222-GCC-7.3.0-2.30/impi/2018.3.222/imkl/2018.3.222/mkl/include -isystem=/share/apps/software/Compiler/GCCcore/7.3.0/zlib/1.2.11/include -isystem=/share/apps/software/Compiler/GCCcore/7.3.0/Szip/2.1.1/include -isystem=/share/apps/software/MPI/intel/2018.3.222-GCC-7.3.0-2.30/impi/2018.3.222/ParMETIS/4.0.3/include -isystem=~/local_opt_ICC/petsc/include -isystem=/share/apps/software/Compiler/intel/2018.3.222-GCC-7.3.0-2.30/impi/2018.3.222/include -isystem=~/local_opt_ICC/adol-c/include -isystem=/share/apps/software/Compiler/GCC/7.3.0-2.30/GSL/2.5/include -isystem=~/local_opt_ICC/nanoflann/include -isystem=~/local_opt_ICC/p4est/include -isystem=~/local_opt_ICC/slepc/include  -Xcompiler "-fpic -ansi -w2 -diag-disable=remark -wd21 -wd68 -wd135 -wd175 -wd177 -wd191 -wd193 -wd279 -wd327 -wd383 -wd981 -wd1418 -wd1478 -wd1572 -wd2259 -wd2536 -wd2651 -wd3415 -wd15531 -wd111 -wd128 -wd185 -wd186 -wd280 -qopenmp-simd -std=c++14 -xhost -march=core-avx2 -O2 -no-ansi-alias -ip -funroll-loops" -arch=sm_35 -std=c++14 -x cu -c ~/Downloads_ICC/dealii/source/lac/la_parallel_vector.cu -o CMakeFiles/obj_lac_release.dir/la_parallel_vector.cu.o

Apparently nvcc does not know the intrinsics, and thus is throwing an error. Everything works fine when disabling the cuda part of deal.II

Daniel Arndt

unread,

Feb 1, 2019, 5:16:29 AM2/1/19

to deal.II User Group

Maxi,

[...]

Apparently nvcc does not know the intrinsics, and thus is throwing an error. Everything works fine when disabling the cuda part of deal.II

Yes, we disabled vectorization when compiling CUDA device code in https://github.com/dealii/dealii/pull/7655 in a more fine-grained way.

Best,

Daniel

Maxi Miller

unread,

Mar 29, 2019, 8:04:27 AM3/29/19

to deal.II User Group

I retested it using the latest version from git, and I still get similar errors. Which option should I disable for getting deal.II to work together with CUDA?

Thanks!

detailed.log

compilation_log.txt

Bruno Turcksin

unread,

Mar 29, 2019, 12:15:03 PM3/29/19

to deal.II User Group

Hi,

On Friday, March 29, 2019 at 8:04:27 AM UTC-4, Maxi Miller wrote:

I retested it using the latest version from git, and I still get similar errors. Which option should I disable for getting deal.II to work together with CUDA?
Thanks!

We don't test cuda+intel so it might be the reason you see this. There shouldn't be any problem with cuda+gcc (this is the only combination that we test). So you could try with gcc instead of intel or you can take a look here https://github.com/dealii/dealii/pull/7663 to see how to disable vectorization.