I'm a new user of MAGMA and new to the GPU game in general (but a long-time user of ScaLAPACK). I'm using a GPU node with the following:
- AMD EPYC 7452 32-Core
- 256 GB system memory
- 4 x Nvidia A100 GPU's with 40 GB memory each
I've been running some tests and, based on the results, I'm planning to upgrade the system memory from 256 GB to 512 GB. Before I make the investment, I'm hoping that someone can confirm that I'm thinking about this correctly.
I'd like to be able to find the eigenvalues and eigenvectors of a double precision, symmetric, Hermitian matrix that is about 110,000 x 110,000. The following test at 90,000 x 90,000 is just about the biggest matrix I can solve:
$ testing/testing_dsyevd -N 90000 -JV --ngpu 4
% MAGMA 2.6.1 64-bit magma_int_t, 64-bit pointer.
Compiled with CUDA support for 8.0
% CUDA runtime 11030, driver 11030. OpenMP threads 32.
% device 0: NVIDIA A100-PCIE-40GB, 1410.0 MHz clock, 40536.2 MiB memory, capability 8.0
% device 1: NVIDIA A100-PCIE-40GB, 1410.0 MHz clock, 40536.2 MiB memory, capability 8.0
% device 2: NVIDIA A100-PCIE-40GB, 1410.0 MHz clock, 40536.2 MiB memory, capability 8.0
% device 3: NVIDIA A100-PCIE-40GB, 1410.0 MHz clock, 40536.2 MiB memory, capability 8.0
% Fri Sep 17 17:01:09 2021
% Usage: testing/testing_dsyevd [options] [-h|--help]
% jobz = Vectors needed, uplo = Lower, ngpu = 4
% N CPU Time (sec) GPU Time (sec) |S-S_magma| |A-USU^H| |I-U^H U|
90000 --- 831.9553 --- --- --- ok
That test used about 250 GB of system memory (all of it!) and topped out at about 18 GB per GPU. I expect my larger matrix to therefore use about (110/90)^2 x 250 = 375 GB of system memory and (110/90)^2 x 18 = 27 GB of each GPU memory. And, thus, upgrading to 512 GB system memory should be sufficient.
Thanks for any advice!!
(And also thanks to the developers -- MAGMA seems fantastic so far!)
ps In case anyone might find this helpful, the trickiest part of the install was getting things set up correctly for ILP64. I used OpenBlas with
make USE_OPENMP=1 INTERFACE64=1 LIBNAMESUFFIX=64
Then I built MAGMA with the attached make.inc.