segmentation fault when calling make_vectorized_array

[Denis Davydov]

Hi all,

With the current master (182e974) I get a weird segmentation fault for float numbers when calling 

 a = make_vectorized_array<value_type>(0.);

inside a constructor of my class where 

VectorizedArray<value_type> a;

I see this only on a cluster (Intel Xeon Ivy Bridge), deal.II is compiled with -march=native with GCC 4.8.5 in Debug mode (obviously segmentation fault is there in Release as well).

Unfortunately, I wiped the old (working) installation and can't check it now,

but I think this used to work with -march=native a few months back.

Relevant config tests are:

-- Performing Test DEAL_II_HAVE_SSE2
-- Performing Test DEAL_II_HAVE_SSE2 - Success
-- Performing Test DEAL_II_HAVE_AVX
-- Performing Test DEAL_II_HAVE_AVX - Success
-- Performing Test DEAL_II_HAVE_AVX512
-- Performing Test DEAL_II_HAVE_AVX512 - Failed
-- Performing Test DEAL_II_HAVE_OPENMP_SIMD
-- Performing Test DEAL_II_HAVE_OPENMP_SIMD - Failed

If I try to replicate what make_vectorized_array does, then it works fine whereas calling it gives segfault:

      const value_type u = 0.;
      VectorizedArray<value_type> result;
      result = u;
      // ^^ copy paste of make_vectorized_array
      a = u;
      VectorizedArray<double> dummy = make_vectorized_array<double>(0.); // <- quick test for doubles
      a = make_vectorized_array<value_type>(0.);   <===== Segmentation fault here!

I am puzzled with this... Any ideas where to dig?

Cheers,

Denis

[Martin Kronbichler]

Hi Denis,

what is a? Is it an element of an array? If yes, is the array of type AlignedVector (or derived from that).

You get this kind of error when there is a load or store to an address that is not aligned by the size of the vectorized array, 32 bytes in your case. The data types VectorizedArray make the compiler insert 'aligned load/store' type of operations, which cannot always be guaranteed.

Can you try to look at the problem in a debugger, e.g. gdb and post the offending assembler line and a few before that to see the context?

Best,
Martin

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Denis Davydov]

Hi Martin,

On Wednesday, July 12, 2017 at 6:05:30 PM UTC+2, Martin Kronbichler wrote:

Hi Denis,

what is a? Is it an element of an array? If yes, is the array of type AlignedVector (or derived from that).

no, it's just a member variable VectorizedArray: 

VectorizedArray<value_type> a;

 

You get this kind of error when there is a load or store to an address that is not aligned by the size of the vectorized array, 32 bytes in your case. The data types VectorizedArray make the compiler insert 'aligned load/store' type of operations, which cannot always be guaranteed.

Can you try to look at the problem in a debugger, e.g. gdb and post the offending assembler line and a few before that to see the context?

here they are:

0x7ffff73913b1 <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+9>        mov    %rdi,%rbx 

   │0x7ffff73913b4 <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+12>       callq  0x7ffff71d1f90 <_ZN6dealii19MatrixFreeOperators4BaseILi3ENS_13LinearAlgebra11distributed6VectorIfEEEC2Ev@plt

   │0x7ffff73913b9 <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+17>       mov    0x9ecd60(%rip),%rax        # 0x7ffff7d7e120      

   │0x7ffff73913c0 <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+24>       lea    0x10(%rax),%rax  

   │0x7ffff73913c4 <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+28>       mov    %rax,(%rbx)      

   │0x7ffff73913c7 <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+31>       vxorps %xmm0,%xmm0,%xmm0

  >│0x7ffff73913cb <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+35>       vmovaps %ymm0,0xe0(%rbx)        

   │0x7ffff73913d3 <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+43>       vmovaps 0x64ffc5(%rip),%ymm0        # 0x7ffff79e13a0

   │0x7ffff73913db <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+51>       vmovaps %ymm0,0x100(%rbx)   

   │0x7ffff73913e3 <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+59>       mov    -0x8(%rbp),%rbx     

   │0x7ffff73913e7 <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+63>       leaveq                       

   │0x7ffff73913e8 <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+64>       retq

(I did a few replacement to ease the readability). Hopefully you can make sense out of it. 

Regards,

Denis. 

[Martin Kronbichler]

Hi Denis,

no, it's just a member variable VectorizedArray: 

VectorizedArray<value_type> a;

But then the class/struct that holds this member variable is not properly aligned, you probably have it inside an std::vector or allocated it with 'new'. You will need to put it into an AlignedVector or similar type. Alternatively you need to wrap the member that is of type VectorizedArray into an AlignedVector<VectorizedArray>.

  >│0x7ffff73913cb <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+35>       vmovaps %ymm0,0xe0(%rbx)       

This is the offending instruction. The compiler tries to store to the address given by an offset of 0xe0 (224 bytes) to the value in %rbx, using an 'aligned' store operation (vmovAps) that assumes 32 byte alignment. If you print the value of %rbx (in gdb with 'print $rbx'), I'm pretty sure you get an address that is divisible by 16 but not by 32. It is a pity that the compiler / API of the intrinsics forces the compiler to use an aligned access instruction whenever there is a variable of type VectorizedArray but there's nothing we can change and we have to live with that...

Best,
Martin

[Denis Davydov]

Hi Martin,

On 13 Jul 2017, at 09:13, Martin Kronbichler <kronbichl...@gmail.com> wrote:

no, it's just a member variable VectorizedArray: 

VectorizedArray<value_type> a;

But then the class/struct that holds this member variable is not properly aligned, you probably have it inside an std::vector or allocated it with 'new'. You will need to put it into an AlignedVector or similar type.

this is a matrix-free level operator which lives in MGLevelObject.

Indeed, internally it stores objects in std::vector of shared pointers.

Alternatively you need to wrap the member that is of type VectorizedArray into an t<VectorizedArray>.

This worked, thanks a lot!

While on the topic of alignment, do I need to be more careful about using

template <typename number>

struct QPData

{

   Table<2, VectorizedArray<number> > values;

}

QPData<double> qp_data;

as a member variable in my class? 

  >│0x7ffff73913cb <Operator<3, 2, 3, LA::d::Vector<float> >::Operator()+35>       vmovaps %ymm0,0xe0(%rbx)       

This is the offending instruction. The compiler tries to store to the address given by an offset of 0xe0 (224 bytes) to the value in %rbx, using an 'aligned' store operation (vmovAps) that assumes 32 byte alignment. If you print the value of %rbx (in gdb with 'print $rbx'), I'm pretty sure you get an address that is divisible by 16 but not by 32. It is a pity that the compiler / API of the intrinsics forces the compiler to

indeed:

(gdb) print $rbx

$1 = 22305328

Thanks a lot for detailed explanation.

Kind regards,

Denis.

[Martin Kronbichler]

Hi Denis,

While on the topic of alignment, do I need to be more careful about using

template <typename number>

struct QPData

{

   Table<2, VectorizedArray<number> > values;

}

QPData<double> qp_data;

as a member variable in my class?

Table<N,VectorizedArray> is fine because it is based on AlignedVector, so it will use the correct alignment internally.

Best,
Martin

[Denis Davydov]

Thanks a lot for your help, Martin!

Cheers,

Denis

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "deal.II User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dealii/xccZy4KU7Zc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dealii+un...@googlegroups.com.