How to add MAGMA library into nvfortran compiler

[Amore]

Hello, 

I want to apply the function from MAGMA library  into my fortran codes. I use the nvfortran compiler of NVIDIA. I follow the official instruction to install the MAGMA-2.8.0 folder. Next, I need to create a correct make.inc file and put it into the home directory of MAGMA. Finally, some static library I need can be obtained after running the makefile using the command "make". 

Some good examples of .inc file are provided in the make.inc-examples folder. The program is similar to generating the intel LAPACK library. However, in our make.inc-examples, I am confused which file is the one I need?  In addition, I am not sure if I can still use the MAGMA library into my codes.
In my linux system, there are ifort, gfortran, gcc and nvfortran. But my codes decide that I must use the nvfortran compiler. And I use OpenACC. cuSparse, cuSolver, cudafor and OpenMP in my codes. 

[Natalie Beams]

You may not be able to use any of the make.inc examples directly, depending on your configuration, but hopefully we can find one that's close to what you want. What are you using for your CPU BLAS/LAPACK?
Many of the examples are for Intel MKL (make.inc.mkl-* files). Some of those are for using 64-bit integers (*-ilp64 versions). There are also some examples for using OpenBLAS. You mentioned Intel LAPACK, so perhaps you want to use MKL? Then there are some compiler options, like gcc or icx. Since you have gcc available, that would probably be a good choice. So maybe make.inc.mkl-gcc would be a good starting point for you, for example. You could then edit the file to use nvfortran instead of gfortran for the Fortran code, if you want (disclaimer: I have never personally tried this).

-- Natalie

[Amore]

Hi Natalie,

Thanks for your replies. I try to creat a make.inc for the Nvfortran compiler, such as:

#//////////////////////////////////////////////////////////////////////////////
#   -- MAGMA (version 2.8.0) --
#      Univ. of Tennessee, Knoxville
#      Univ. of California, Berkeley
#      Univ. of Colorado, Denver
#      @date March 2024
#//////////////////////////////////////////////////////////////////////////////

# --------------------
# configuration

BACKEND     ?= cuda

OPENBLASDIR ?= /usr/local/lib/cmake/
CUDADIR     ?= /usr/local/cuda

ifeq (,$(findstring $(BACKEND),hip cuda))
    $(error "'BACKEND' should be either 'cuda' or 'hip' (got $(BACKEND))")
endif

# --------------------
# programs

CC          ?= gcc
CXX         ?= g++
FORT        ?= nvfortran
HIPCC       ?= hipcc
NVCC        ?= nvcc
DEVCC       ?= NONE

ifeq ($(BACKEND),cuda)
    DEVCC = $(NVCC)
else ifeq ($(BACKEND),hip)
    DEVCC = $(HIPCC)
endif

ARCH      = ar
ARCHFLAGS = cr
RANLIB    = ranlib

# --------------------
# flags/settings

ifeq ($(BACKEND),cuda)
    GPU_TARGET = Volta Turing Ampere
else ifeq ($(BACKEND),hip)
    GPU_TARGET = gfx900 gfx906 gfx908
endif

FPIC      = -fPIC

CFLAGS      = -O3 $(FPIC) -DNDEBUG -DADD_ -Wall -fopenmp -std=c99
CXXFLAGS    = -O3 $(FPIC) -DNDEBUG -DADD_ -Wall -fopenmp -std=c++11
FFLAGS      = -O3 $(FPIC) -Mfree -Mbackslash -Mlarge_arrays -Minfo=all
F90FLAGS    = -O3 $(FPIC) -Mfree -Mbackslash -Mlarge_arrays -Minfo=all
LDFLAGS     =     $(FPIC)                       -fopenmp

DEVCCFLAGS  = -O3         -DNDEBUG -DADD_

# --------------------
# libraries

LIB       += -lopenblas

# --------------------
# directories

LIBDIR    += -L$(OPENBLASDIR)/lib
INC       +=

# --------------------
# checks

-include make.check-openblas

# --------------------
# backend-specific

ifeq ($(BACKEND),cuda)
    -include make.check-cuda

    DEVCCFLAGS += -Xcompiler "$(FPIC)" -std=c++11

    INC    += -I$(CUDADIR)/include
    LIBDIR += -L$(CUDADIR)/lib64
    LIB    += -lcublas -lcusparse -lcudart -lcudadevrt
endif

ifeq ($(BACKEND),hip)
    -include make.check-hip

    DEVCCFLAGS += $(FPIC) -std=c++11

    INC    += -I$(ROCM_PATH)/include
    LIBDIR += -L$(ROCM_PATH)/lib
    LIB    += -lhipblas -lhipsparse
endif

However, I put it into the home directoy of MAGMA and run the command "make". After a long time, I only obtain the libmagma.a and libmagma.ao. According to the offical ReadMe file, another static library should be generated as the same time. So I am not sure the static library I create is correct. I try to use this library to my codes. When I call the function magma_dgesv, the error is that this functiion can not be find. Do I need to add other command, such as USE MAGMA? 

Unfortunately, the my libmagma.a is too large to can not be uploded, but you can use the make.inc file mentioned above to generate one.

[Natalie Beams]

Do you see any error messages when you do `make`?

If libmagma.a/libmagma.so are created, another reason you may get an error about `magma_dgesv` not being found would be if the linking to MAGMA from your code is not happening correctly. Is the error a build error or linking error? Can you share your exact error output? 

--Natalie

[Amore]

This is the error when I directly do 'make'

[Natalie Beams]

Ah, I think I see...Those test files are in F77, and need the `-Mfixed` compiler flag instead of `-Mfree` with nvfortran.  What happens if you change the FFLAGS variable to use `-Mfixed`?
(F90FLAGS should remain `-Mfree`)

-- Natalie

[Mark Gates]

On Wed, Jan 15, 2025 at 9:23 AM Amore <zhaoqi...@163.com> wrote:

However, I put it into the home directoy of MAGMA and run the command "make". After a long time, I only obtain the libmagma.a and libmagma.ao. According to the offical ReadMe file, another static library should be generated as the same time.

What other static library are you expecting? Are you referring to this MAGMA README?

https://github.com/icl-utk-edu/magma/blob/master/README

 

So I am not sure the static library I create is correct. I try to use this library to my codes. When I call the function magma_dgesv, the error is that this functiion can not be find. Do I need to add other command, such as USE MAGMA?

Unfortunately, the my libmagma.a is too large to can not be uploded, but you can use the make.inc file mentioned above to generate one.

If you are calling from Fortran, you probably want magmaf_dgesv (with an f for Fortran). Otherwise, you would need to call magma_dgesv as a C function using Fortran's iso_c_binding. You can check what symbols are in the library using `nm`. For example (I highlighted the relevant C and Fortran routines):

sh methane magma> nm lib/libmagma.so  | grep dgesv
00000000009c61a0 W _Z26dgesv_batched_small_kernelILi10EEvPPdiPPiS1_iS2_
00000000009c60a0 W _Z26dgesv_batched_small_kernelILi11EEvPPdiPPiS1_iS2_
00000000009c5fa0 W _Z26dgesv_batched_small_kernelILi12EEvPPdiPPiS1_iS2_
00000000009c5ea0 W _Z26dgesv_batched_small_kernelILi13EEvPPdiPPiS1_iS2_
00000000009c5da0 W _Z26dgesv_batched_small_kernelILi14EEvPPdiPPiS1_iS2_
00000000009c5ca0 W _Z26dgesv_batched_small_kernelILi15EEvPPdiPPiS1_iS2_
00000000009c5ba0 W _Z26dgesv_batched_small_kernelILi16EEvPPdiPPiS1_iS2_
00000000009c5aa0 W _Z26dgesv_batched_small_kernelILi17EEvPPdiPPiS1_iS2_
00000000009c59a0 W _Z26dgesv_batched_small_kernelILi18EEvPPdiPPiS1_iS2_
00000000009c58a0 W _Z26dgesv_batched_small_kernelILi19EEvPPdiPPiS1_iS2_
00000000009c6aa0 W _Z26dgesv_batched_small_kernelILi1EEvPPdiPPiS1_iS2_
00000000009c57a0 W _Z26dgesv_batched_small_kernelILi20EEvPPdiPPiS1_iS2_
00000000009c56a0 W _Z26dgesv_batched_small_kernelILi21EEvPPdiPPiS1_iS2_
00000000009c55a0 W _Z26dgesv_batched_small_kernelILi22EEvPPdiPPiS1_iS2_
00000000009c54a0 W _Z26dgesv_batched_small_kernelILi23EEvPPdiPPiS1_iS2_
00000000009c53a0 W _Z26dgesv_batched_small_kernelILi24EEvPPdiPPiS1_iS2_
00000000009c52a0 W _Z26dgesv_batched_small_kernelILi25EEvPPdiPPiS1_iS2_
00000000009c51a0 W _Z26dgesv_batched_small_kernelILi26EEvPPdiPPiS1_iS2_
00000000009c50a0 W _Z26dgesv_batched_small_kernelILi27EEvPPdiPPiS1_iS2_
00000000009c4fa0 W _Z26dgesv_batched_small_kernelILi28EEvPPdiPPiS1_iS2_
00000000009c4ea0 W _Z26dgesv_batched_small_kernelILi29EEvPPdiPPiS1_iS2_
00000000009c69a0 W _Z26dgesv_batched_small_kernelILi2EEvPPdiPPiS1_iS2_
00000000009c4da0 W _Z26dgesv_batched_small_kernelILi30EEvPPdiPPiS1_iS2_
00000000009c4ca0 W _Z26dgesv_batched_small_kernelILi31EEvPPdiPPiS1_iS2_
00000000009c4ba0 W _Z26dgesv_batched_small_kernelILi32EEvPPdiPPiS1_iS2_
00000000009c68a0 W _Z26dgesv_batched_small_kernelILi3EEvPPdiPPiS1_iS2_
00000000009c67a0 W _Z26dgesv_batched_small_kernelILi4EEvPPdiPPiS1_iS2_
00000000009c66a0 W _Z26dgesv_batched_small_kernelILi5EEvPPdiPPiS1_iS2_
00000000009c65a0 W _Z26dgesv_batched_small_kernelILi6EEvPPdiPPiS1_iS2_
00000000009c64a0 W _Z26dgesv_batched_small_kernelILi7EEvPPdiPPiS1_iS2_
00000000009c63a0 W _Z26dgesv_batched_small_kernelILi8EEvPPdiPPiS1_iS2_
00000000009c62a0 W _Z26dgesv_batched_small_kernelILi9EEvPPdiPPiS1_iS2_
00000000009c4b90 T _Z29dgesv_batched_small_sm_kerneliiPPdiPPiS0_iS1_
00000000009c4a70 T _Z63__device_stub__Z29dgesv_batched_small_sm_kerneliiPPdiPPiS0_iS1_iiPPdiPPiS0_iS1_
                 U dgesvd_
0000000019c9be70 D dgesvd_path
0000000000571180 T magma_dgesv
0000000000690240 T magma_dgesv_batched
00000000009c3e80 T magma_dgesv_batched_small
00000000005601c0 T magma_dgesv_gpu
0000000000690660 T magma_dgesv_nopiv_batched
0000000000560540 T magma_dgesv_nopiv_gpu
0000000000572020 T magma_dgesv_rbt
00000000006915b0 T magma_dgesv_rbt_batched
0000000000296800 T magma_dgesvd
00000000001e2210 T magma_get_dgesvd_nb
00000000001ea0f0 T magmaf_dgesv_
00000000001ea110 T magmaf_dgesv_gpu_
00000000001ea130 T magmaf_dgesv_nopiv_gpu_
00000000001ea150 T magmaf_dgesv_rbt_
00000000001ea1c0 T magmaf_dgesvd_
00000000001e9520 T magmaf_get_dgesvd_nb_

If that doesn't work, can you share a minimal code example so we can see what exactly you're calling?

Mark

[Amore]

Again, only one libmagma.a file was generated. According to  mygpu.pdf (d29g4g2dyqv443.cloudfront.net),  the libmagma sparse.a library should be generated at the same time.  But the error is different after changing the '-Mfree' to '-Mfixed'.

[Amore]

After running ' nm lib/libmagma.so  | grep dgesv ', the same results are obtained. I try to use the magma_dgesv or  magmaf_dgesv  from the llibrary. The compiler ask me for "/home/featurize/backup/home/featurize/Git_Document/Inversion_GN/module/TESTLIBRARY.f90:122: undefined reference to `magmaf_dgesv_'".

Here is the code that I call the functiion. The parameters are correct, bucause them are used in DGESV from intel Lapack library.

        ! CALL DGESV( rows, NRHS,A, cols, IPIVs, B, cols, INFO )

        CALL magma_dgesv( rows, NRHS,A, cols, IPIVs, B, cols, INFO )

        CALL magmaf_dgesv (rows, NRHS,A, cols, IPIVs, B, cols, INFO)

[Mark Gates]

Thanks for the link to the document you are following. `make` would generally produce both libmagma.so and libmagma_sparse.so. `make lib` will produce just libmagma.so, which is all you need for call dgesv. `make sparse-lib` will produce libmagma_sparse.so, but you don't need that for solving dense systems. But if there's an error during the compilation, it may only get through building libmagma.so and not get to building libmagma_sparse.so.

Please include the complete input and output of your compile line, as below. A complete minimum example that we can compile is also helpful.

Here's a minimum example that works.

sh methane> gfortran -Wall -o main main.f90 -Wl,-rpath,${MAGMA_DIR}/lib -L ${MAGMA_DIR}/lib -lmagma -Wl,-rpath,${ICL_CUDA_ROOT}/lib64 -L${ICL_CUDA_ROOT}/lib64 -lcublas -lcusparse -lcudart
sh methane> ./main
 done
sh methane> cat main.f90
program main
    !! lda >= n, ldb >= n
    integer, parameter:: n = 10, lda = n, ldb = n, nrhs = 5
    integer:: info
    integer:: IPIVs( n )
    double precision:: A( lda, n ), B( ldb, nrhs )
    call magmaf_init()
    ! call DGESV( n, nrhs, A, lda, IPIVs, B, ldb, info )
    ! call magma_dgesv( n, nrhs, A, lda, IPIVs, B, ldb, info )  !! can't call C API
    call magmaf_dgesv( n, nrhs, A, lda, IPIVs, B, ldb, info )  !! call Fortran API
    call magmaf_finalize()
    print *, "done"
end program

$MAGMA_DIR and $ICL_CUDA_ROOT are paths to wherever those libraries are installed.

I'm confused by your code that has rows for the n parameter (that's fine) but cols for the lda and ldb parameters. The lda and ldb are really the number of rows in the A and B arrays, not the number of columns. lda and ldb >= rows. Of course, A is square, n x n, but B is presumably tall, n x nrhs.

Mark