GPU-to-GPU communication

[bcsj]

I've been playing around with writing my own methods to run on slate::Matrix data on the GPU and have thus been looking more into the tile-management part of the slate code.

I seem to observe that when I send a tile (i,j) from its rank to another rank with tileSend and tileRecv I also get a copy of the tile on the Host on the receiving rank. 

Example code

if (A.mpiRank() == A.tileRank(1,0)) {

        A.tileSend(1, 0, 0);

    } else if (A.mpiRank() == 0) {

        A.tileRecv<slate::Target::Devices>(1, 0, A.tileRank(1,0), slate::Layout::ColMajor);

    }

I noticed the tileSend<Target> is being deprecated.

The tile (1,0) here is on the GPU on rank 1 and after the send rank 0 receives it tileExists(1,0,dev) tells me that the tile exists on both host -1 and device 0.

I have export MPICH_GPU_SUPPORT_ENABLED=1 so GPU-to-GPU should be enabled. This is on the LUMI infrastructure, so the hardware should support GPU-to-GPU.

I assume there is simply something I've misunderstood in how this is supposed to work?

Thanks!

[bcsj]

Here is my little test code.

My allocation:

salloc --account=<...> --partition=small-g --nodes=1 --ntasks=4 --gpus=1 --time=00:30:00

srun ./tile_test_simple.app

Code

#include <slate/slate.hh>

#include <iostream>

#include <iomanip>

template <typename scalar_t>

void printTileExists(slate::Matrix<scalar_t> &A, int device) {

    if (A.mpiRank() != 0) return;

    int64_t mt = A.mt();

    int64_t nt = A.nt();

    std::cout << "Exists (device = " << device << "):" << std::endl;

    for (int64_t i = 0; i < mt; i++) {

        for (int64_t j = 0; j < nt; j++) {

            std::cout << A.tileExists(i, j, device);

            if (j + 1 < nt)

                std::cout << " ";

        }

        std::cout << std::endl;

    }

}

int main(int argc, char **argv) {

    slate::gpu_aware_mpi(true);

    using scalar_t = double;

    int64_t  m = 128;

    int64_t  n = 128;

    int64_t  tilesize = 16;

    int64_t p = 2, q = 2;

    int err=0, mpi_provided=0;

    err = MPI_Init_thread( &argc, &argv, MPI_THREAD_MULTIPLE, &mpi_provided );

    assert( err == 0 && mpi_provided == MPI_THREAD_MULTIPLE );

    slate::Options opts = {{slate::Option::Target, slate::Target::Devices}};

    slate::Matrix<scalar_t> A(m, n, tilesize, p, q, MPI_COMM_WORLD);

    A.insertLocalTiles(slate::Target::Devices);

    slate::Matrix<scalar_t> B = A.emptyLike();

    B.insertLocalTiles(slate::Target::Devices);

    slate::copy(A, B, opts);

    /////////////////////////////////////////////////////////

    // GPU2GPU COMM TEST ////////////////////////////////////

    if (A.mpiRank() == A.tileRank(1,0)) {

        A.tileSend(1, 0, 0);

    } else if (A.mpiRank() == 0) {

        A.tileRecv<slate::Target::Devices>(1, 0, A.tileRank(1,0), slate::Layout::ColMajor);

        printTileExists(A, -1);

        printTileExists(A,  0);

    }

    return 0;

}

Output

Exists (device = -1):
0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
Exists (device = 0):
1 0 1 0 1 0 1 0
1 0 0 0 0 0 0 0
1 0 1 0 1 0 1 0
0 0 0 0 0 0 0 0
1 0 1 0 1 0 1 0
0 0 0 0 0 0 0 0
1 0 1 0 1 0 1 0
0 0 0 0 0 0 0 0

I've also tried putting slate::gpu_aware_mpi(true); after the MPI_Init_thread, just in case that could matter. And I've tried with and without the environment variables too, without any changes.

[Neil Lindquist]

Resenting to the listserve since I didn't have emails setup right.

On Thu, May 16, 2024, 9:26 AM Neil Lindquist <neillin...@gmail.com> wrote:

Oops, I guess I didn't hit reply all.

Are you using the development version or a release version?  The logic for GPU-aware tileSend/tileRecv was only merged in December or January, and I don't think there's been a subsequent release.

I can't think of any other possible issues off the top of my head.  Your code looks to me like it should do what you expect.

Neil

On Thu, May 16, 2024, 8:57 AM Bjørn Jensen <bjornje...@gmail.com> wrote:

Hi Neil,

Thanks for your reply! I think you might have replied to me directly and not to the User Forum(?)

I've tried setting the environment variable and separately setting the state in the code as you suggested, but I still seem to get a Host-copy of the tile I send.

Are there other steps I would need to take to avoid the host-copy?

Actually, let me put the example code on the forum, so you know exactly what I'm running.

On Thu, 16 May 2024, 15:40 Neil Lindquist, <neillin...@gmail.com> wrote:

SLATE doesn't currently have any logic to detect whether MPI is GPU aware.  You can tell SLATE to use GPU aware MPI by setting the environment variable SLATE_GPU_AWARE_MPI=1 or calling slate::gpu_aware_mpi(true).

Neil

--
You received this message because you are subscribed to the Google Groups "SLATE User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to slate-user+...@icl.utk.edu.
To view this discussion on the web visit https://groups.google.com/a/icl.utk.edu/d/msgid/slate-user/85a6e1f0-2b62-4c83-b7d1-1144088adc3an%40icl.utk.edu.

[bcsj]

Okay, that makes sense. I should be using the 2023.11.05-release version, so I guess I just missed that feature.