Solving eigenproblem of very large matrix(270k) with dsyevd: GPU mem alloc error

46 views
Skip to first unread message

KyungWon Park

unread,
Sep 28, 2020, 8:32:20 AM9/28/20
to MAGMA User
Hello

I want to get eigen values and vectors for
very large matrix (270k by 270k)

But with 4 * NVIDIA V100 GPUs(32GB VRAM) and
testing_dsyevd example in MAGMA testing directory,
I get "error -113: cannot allocate memory on GPU device" error

Should VRAM be big enough to hold the entire matrix?
I'd like to know about hardware requirements for MAMGA routines...

And if I can't solve this 270k matrix problem with MAGMA,
could you recommend any other choices?

Thank you in advance!

Stanimire Tomov

unread,
Oct 5, 2020, 6:13:59 PM10/5/20
to KyungWon Park, MAGMA User
Hello,

MAGMA is for a single node. Multiple GPUs can be used. The memory requirement
is that the matrix can fit in VRAM but also that it can fit in the GPUs’ memory.

In particular, the requirement for a GPU is:

nb*(1 + n/(nb*ngpu)) * (ldda + 3*lddw*nb) + ldda*( magma_ceildiv( n, nb ) + 1),

where n is the matrix size, ldda is the leading dimension (n + possibly a little bit
to have better alignment), and nb is the block size, and ngpu is the number of GPUs.
So, you need about n^2/ngpu x 8 (for double precision) bytes memory per GPU. 

So, you would not be able to solve this with MAGMA as your matrix is about
580 GB and the 4 GPUs have only 128 GB. MAGMA has linear solvers that can be used
for problems exceeding the GPU memories but we haven’t developed these for eigensolvers.

An alternative that I would recommend is to use SLATE (https://icl.utk.edu/slate/), which is 
for distributed-memory systems and supports GPUs. You can scale and use a number of nodes so 
that the matrix fits into the GPU memories. SLATE is under development now and has eigenvalue 
computation for dsyevd, but SLATE doesn't yet compute vectors (coming up).

Best regards,
Stan


--
You received this message because you are subscribed to the Google Groups "MAGMA User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to magma-user+...@icl.utk.edu.
To view this discussion on the web visit https://groups.google.com/a/icl.utk.edu/d/msgid/magma-user/33f86ee4-5100-488b-a3fb-da44bba4d12fn%40icl.utk.edu.

KyungWon Park

unread,
Oct 7, 2020, 2:20:50 AM10/7/20
to MAGMA User, to...@icl.utk.edu, MAGMA User, KyungWon Park
Thank you so much for the kind reply!
I'll look into SLATE

Best
Reply all
Reply to author
Forward
Message has been deleted
0 new messages