slate_scalapack_api and FHI-aims

[Yi Yao]

Dear SLATE developers,

We are at the 2021 SDSC GPU Hackathon. We are trying to link SLATE for distributed linear algebra operations with our FHI-aims code to perform GW calculation on GPU. We successfully linked SLATE but it gave us this error at the run time.

aims.210427.scalapack.mpi.x: scalapack_api/scalapack_slate.hh:64: slate::Matrix<scalar_t> slate::scalapack_api::slate_scalapack_submatrix(int, int, slate::Matrix<scalar_t>&, int, int, int*) [with scalar_t = std::complex<double>]: Assertion `An % desc_NB(desca) == 0' failed.

We noted two notes in your README_scalapack_api.txt file.

NOTE: A submatrix must start and end on a tile boundary.

Taking a submatrix of global matrix A of size Am,An starting at ia,ja.

and 

NOTE: The ScaLAPACK BLACS grid needs to be Column-major.

CALL BLACS_GRIDINIT( ICTXT, 'Col-major', NPROW, NPCOL )

Our usage do conflict with these two restrictions. We are using the row-major gridinit and the submatrix might not be dividable by the block size. 

When should we expect these two restrictions be eliminated? Or should we work on to reconstruct our code?

Best wishes,

Yi

[Mark Gates]

Hi Yi,

We should be able to relax the tile boundary condition somewhat. Are you using a submatrix of a larger matrix, so it really doesn't begin & end on tile boundaries, or do you just have an arbitrary matrix where m % mb != 0 and n % nb != 0. The later case is easily handled in SLATE. Most SLATE codes do assume the upper-left tile starts on a tile boundary.

It should be easy to support both row-major and column-major BLACS grids. We just need to construct matrices with a different lambda for the distribution.

Both of these issues are limitations in SLATE's ScaLAPACK API, not in SLATE itself. Note that using SLATE through its ScaLAPACK interface is meant as a transition aide, but the best performance will of course be using SLATE's native interface.

What routines are you interested in using from SLATE?

Mark

--

Innovative Computing Laboratory

University of Tennessee, Knoxville

[Yi Yao]

Post the email in the google group.

Hi Mark, 

Thank you so much for the quick response. 

We should be able to relax the tile boundary condition somewhat. Are you using a submatrix of a larger matrix, so it really doesn't begin & end on tile boundaries, or do you just have an arbitrary matrix where m % mb != 0 and n % nb != 0. The later case is easily handled in SLATE. Most SLATE codes do assume the upper-left tile starts on a tile boundary.

We are not using a submatrix, but instead our matrix are not start from 1. For example, we defined our matrix to something like mat(100:200,100:200) for convenience. Does that cause the problem?

It should be easy to support both row-major and column-major BLACS grids. We just need to construct matrices with a different lambda for the distribution.

Both of these issues are limitations in SLATE's ScaLAPACK API, not in SLATE itself. Note that using SLATE through its ScaLAPACK interface is meant as a transition aide, but the best performance will of course be using SLATE's native interface.

What routines are you interested in using from SLATE?

We are using the pzgemm, pzpotrf, pzpotri, and pztranc. Are you suggesting using the SLATE API directly? How much effort should we expect to take to do that?

Best wishes,

Yi

[Mark Gates]

On Thu, May 13, 2021 at 4:13 PM Yi Yao wrote:

We should be able to relax the tile boundary condition somewhat. Are you using a submatrix of a larger matrix, so it really doesn't begin & end on tile boundaries, or do you just have an arbitrary matrix where m % mb != 0 and n % nb != 0. The later case is easily handled in SLATE. Most SLATE codes do assume the upper-left tile starts on a tile boundary.

 

We are not using a submatrix, but instead our matrix are not start from 1. For example, we defined our matrix to something like mat(100:200,100:200) for convenience. Does that cause the problem?

Do you mean your code is in Fortran, and your array is defined with a lower bound of 100? It's unclear to me exactly the ramifications of that. SLATE is in C++, so the array always starts at (0, 0), which in the ScaLAPACK Fortran parlance is (ia=1, ja=1). It would be helpful to see a minimum code example to see how your array is defined and how you call ScaLAPACK.

It should be easy to support both row-major and column-major BLACS grids. We just need to construct matrices with a different lambda for the distribution.

Both of these issues are limitations in SLATE's ScaLAPACK API, not in SLATE itself. Note that using SLATE through its ScaLAPACK interface is meant as a transition aide, but the best performance will of course be using SLATE's native interface.

What routines are you interested in using from SLATE?

We are using the pzgemm, pzpotrf, pzpotri, and pztranc. Are you suggesting using the SLATE API directly? How much effort should we expect to take to do that?

Is your code in Fortran or C++? If in C++, yes, I would recommend using SLATE's native API rather than the ScaLAPACK API. I think the SLATE API is easier to use. All the parameters defining each matrix are encapsulated into the matrix objects, so the calling sequence is simplified, e.g.:

    slate::chol_solve( A, B );

vs.

    scalapack_pposv( uplo, n, nrhs, A, 1, 1, descA, B, 1, 1, descB, &info );

You can see examples in the SLATE Users Guide or SLATE Tutorial (below). If your code is in Fortran, we do have an experimental native SLATE Fortran API.

Users Guide

https://bitbucket.org/icl/slate/  # see README.md

https://www.icl.utk.edu/files/publications/2020/icl-utk-1437-2020.pdf

Tutorial

https://bitbucket.org/icl/slate/downloads/

https://bitbucket.org/icl/slate/downloads/slate-tutorial-2021.pdf

Mark

[Yi Yao]

Yes, our code is in Fortran. I will have a look at the Fortran interface. Thanks!