Solving eigenproblem of very large matrix(270k) with dsyevd: GPU mem alloc error

[KyungWon Park]

Hello

I want to get eigen values and vectors for

very large matrix (270k by 270k)

But with 4 * NVIDIA V100 GPUs(32GB VRAM) and

testing_dsyevd example in MAGMA testing directory,

I get "error -113: cannot allocate memory on GPU device" error

Should VRAM be big enough to hold the entire matrix?

I'd like to know about hardware requirements for MAMGA routines...

And if I can't solve this 270k matrix problem with MAGMA,

could you recommend any other choices?

Thank you in advance!

[Stanimire Tomov]

Hello,

MAGMA is for a single node. Multiple GPUs can be used. The memory requirement

is that the matrix can fit in VRAM but also that it can fit in the GPUs’ memory.

In particular, the requirement for a GPU is:

nb*(1 + n/(nb*ngpu)) * (ldda + 3*lddw*nb) + ldda*( magma_ceildiv( n, nb ) + 1),

where n is the matrix size, ldda is the leading dimension (n + possibly a little bit

to have better alignment), and nb is the block size, and ngpu is the number of GPUs.

So, you need about n^2/ngpu x 8 (for double precision) bytes memory per GPU. 

So, you would not be able to solve this with MAGMA as your matrix is about

580 GB and the 4 GPUs have only 128 GB. MAGMA has linear solvers that can be used

for problems exceeding the GPU memories but we haven’t developed these for eigensolvers.

An alternative that I would recommend is to use SLATE (https://icl.utk.edu/slate/), which is 

for distributed-memory systems and supports GPUs. You can scale and use a number of nodes so 

that the matrix fits into the GPU memories. SLATE is under development now and has eigenvalue 

computation for dsyevd, but SLATE doesn't yet compute vectors (coming up).

Best regards,

Stan

--
You received this message because you are subscribed to the Google Groups "MAGMA User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to magma-user+...@icl.utk.edu.
To view this discussion on the web visit https://groups.google.com/a/icl.utk.edu/d/msgid/magma-user/33f86ee4-5100-488b-a3fb-da44bba4d12fn%40icl.utk.edu.

[KyungWon Park]

Thank you so much for the kind reply!

I'll look into SLATE

Best