ilp64 build -> odd ipiv vector

[Paul Karlshöfer]

Dear developers,

I am using Magma to compute the LU decomposition of a large matrix with the function 

magmaf_dgetrf_m(...).

The matrices are larger than the combined GPU memory - so to deal with a seg-fault once we exceed the memory size, we built Magma with ipl64. This works great.

However, we encounter a problem with the ipiv vector (permutation matrix) which is a integer(8), allocatable   :: ipiv(:)

In int8 representation, the return vector reads something like:

8589934593

17179869187

25769803781

[..]

I noticed that each value represents 2 int4s, as 8589934593 in hex is 0x200000001

so i tried to split each int8 into 2 int4s with some bitwise operations. However, there are also a lot of 0s or other random but small and reoccurring numbers in the ipiv vector which don't seem to represent anything meaningful in the ipiv vector.

Furthermore, I counted the numbers in the ipiv vector which exceed 0x100000000 and they precisely match the matrix dimension divided by 2.

So I conclude that all those numbers (split in 2 int4s) should somehow represent my ipiv vector but so far, I didn't manage to fully recover it.

The source of the issue seems to be, that Magma does not properly fill the ipiv vector for int8?!

Does anyone have an idea? I am grateful for all tips!

This is part of the code:

              integer, pointer        :: ipiv(:)

              integer(8), allocatable   :: ipiv_copy(:)

              call magmaf_dgetrf_m(1, d_dim, d_dim, asigma(id)%mat, d_dim, ipiv_copy, info_lapack )

              do i = 1, d_dim

if (ipiv_copy(i) .GE. 4294967296) then ! >0x100000000

  id3 = id3 + 1

  asigma(id)%ipiv(id3) = int(ISHFT(ISHFT(ipiv_copy(i),32), -32), kind=4)

  id3 = id3 + 1

  asigma(id)%ipiv(id3) = int(ISHFT(ipiv_copy(i),-32), kind=4)

end if

              end do

 

IPIV Vector

          int8                          int4 (least sig)   int4(most sig)

          8589934593           1           2

           17179869187           3           4

           25769803781           5           6

           34359738375           7           8

           42949672969           9          10

           51539607563          11          12

           60129542157          13          14

           68719476751          15          16

           77309411345          17          18

           85899345939          19          20

           94489280533          21          22

          103079215127          23          24

          111669149721          25          26

          120259084315          27          28

          128849018909          29          30

          352187318353          81          82

          360777252947          83          84

          369367187541          85          86

          377957122135          87          88

          386547056729          89          90

        28441273440733        6621        6622

        28449863375327        6623        6624

        28458453309921        6625        6626

        28467043244515        6627        6628

        28475633179109        6629        6630

          523986010233         121         122

          532575944827         123         124

[...]

                   256         ?                ?

                   256        ?                ?

                   256        ?                ?

                   256         ?                ?

                   256        ?                ?

                   256         ?                ?

                   256        ?                ?

                   256         ?                ?

                   256         ?                ?

[Mark Gates]

What BLAS & LAPACK library are you linking with? What is your make.inc file, or the list of libraries using `ldd your_application`? For example, using MKL, it should have ilp64 in the name: like -lmkl_gf_ilp64.

With ilp64, the ipiv vector should have 64-bit integers in it. You shouldn't need to unpack them as 32-bit ints in any way.

Mark