MAGMA BLAS and Auxiliary, which header to use?

[Tahir Malas]

Hi,

When I try to use low level functions like magmablas_stranspose and add #include "magma_lapack.h" for that purpose, I got name errors like

magma/include/magma_mangling.h:32:10: error: #error "One of ADD_, NOCHANGE, or UPCASE must be defined to set how Fortran functions are name mangled. For example, in MAGMA, add -DADD_ to CFLAGS, FFLAGS, etc. in make.inc. If using CMake, it defines MAGMA_GLOBAL instead." 

I did use the make.inc file for MKL gcc version, and -DADD_ is added to C flags. Does anyone have idea on how to handle this problem?

Thanks... 

[Mark Gates]

Hi Tahir,

1) You don't need to include magma_lapack.h to call magmablas_stranspose. Just include magma_v2.h should define magmablas_stranspose. Including magma_lapack.h is needed only if you are calling lapack functions using MAGMA's prototypes, e.g., lapackf77_zgetrf or blasf77_zgemm.

2) If you really do want to include magma_lapack.h, you need to define one of ADD_, NOCHANGE, or UPCASE before including it. That is, not just in make.inc when MAGMA is compiled, but also when you include that header in your application.

Mark

[Tahir Malas]

Hi Mark,

I did not add magma_lapack.h but I got a link time error (undefined symbol: magmablas_stranspose). Then when I add it I get this error. Do you have an idea why I may be getting this error then? I have been using dense magma routines without any problems so far. When I tried to use magma_s_spmv and related routines to set the matrix and vectors, I started to get those errors.

Thanks,

Tahir

[Mark Gates]

It seems odd that you would get that undefined symbol error. Can you give the exact command that is generating that link error?

You can check whether the symbol exists in the library:

[mgates@saturn magma]$ nm lib/libmagma.so | grep magmablas_stranspose 

00000000004d1910 T magmablas_stranspose                  # << this one

00000000004d1b60 T magmablas_stranspose_batched

00000000004d1dc0 T magmablas_stranspose_batched_stride

00000000004d3780 T magmablas_stranspose_inplace

00000000004ff050 T magmablas_stranspose_inplace_v1

00000000004ff000 T magmablas_stranspose_v1

000000000067d8d0 r _ZZ20magmablas_stransposeE12__FUNCTION__

000000000067d8b0 r _ZZ28magmablas_stranspose_batchedE12__FUNCTION__

000000000067dc70 r _ZZ28magmablas_stranspose_inplaceE12__FUNCTION__

000000000067d880 r _ZZ35magmablas_stranspose_batched_strideE12__FUNCTION__

magma-sparse does indeed call magmablas_stranspose (below), and looking at the magma_s_spmv, that function indirectly calls the stranspose.

[mgates@saturn magma]$ nm lib/libmagma_sparse.so | grep magmablas_stranspose 

                 U magmablas_stranspose

[mgates@saturn magma]$ nm lib/libmagma_sparse.so | grep magma_s_spmv

0000000000148210 T magma_s_spmv

00000000001494c0 T magma_s_spmv_shift

Mark

[Tahir Malas]

Yes, it is odd indeed. The exact command is a part of complex makefile system used to install our software. make does not complain but make install does give that undefided symbol error. Here is the install command:

install: $(SUBDIRS:%=%_install) | install_start

$(MAKE) -C . strip_install

$(MAKE) -C . check_install

$(MAKE) -C . generate_manifest

@echo "-install $(CONFIG) ended successfully `date`" >> $(OUTROOT)/successBuild

May not mean much though. I checked the symbols in libmagma.so and libmagma_sparse.so; there is no problem.

I also tested magma sparse with its own tests. Other than a failure for CSR5 storage, they run OK:

% #    usage: ./run_sspmv [ --blocksize 4 --alignment 4 (for SELLP) ] matrices

% # matrix info: 2209-by-2209 with 10857 nonzeros

% > MKL  : 3.04e-03 seconds 7.15e-03 GFLOP/s    (CSR).
% > MAGMA: 6.16e-06 seconds 3.53e+00 GFLOP/s    (standard CSR).
% > MAGMA: 5.12e-06 seconds 4.24e+00 GFLOP/s    (standard ELL).
% |x-y|_F/|y| = 0.00e+00.  Tester spmv ELL:  ok
% > MAGMA: 5.42e-06 seconds 4.01e+00 GFLOP/s    (SELLP).
% |x-y|_F/|y| = 0.00e+00 Tester spmv SELL-P:  ok
% > MAGMA: 1.91e-05 seconds 0.00e+00 GFLOP/s    (CSR5).
% |x-y|_F/|y| = 1.33e-01 Tester spmv CSR5:  failed
% > cuSPARSE: 4.47e-06 seconds 4.86e+00 GFLOP/s    (CSR).
% |x-y|_F/|y| = 0.00e+00 Tester spmv cuSPARSE CSR:  ok
% > cuSPARSE: 1.76e-05 seconds 1.24e+00 GFLOP/s    (HYB).
% |x-y|_F/|y| = 0.00e+00 Tester spmv cuSPARSE HYB:  ok
 3.04e-03 7.15e-03 4.47e-06 4.86e+00 1.76e-05 1.24e+00 5.12e-06 4.24e+00 5.42e-06 4.01e+00 nan nan

So, do you have any other suggestions?

Tahir

[Mark Gates]

Can you run that `make install` (using -j1, not parallel make), and show the command that failed and its output? One possibility is the order of libraries is wrong. They should be: -lmagma_sparse -lmagma, with sparse first, since it depends on magma and the linker processes left-to-right. E.g.,

[mgates@b01 magma]$ make sparse/testing/testing_zspmv

g++ -fPIC -fopenmp -Wl,-rpath,/home/mgates/magma/lib \

    -o sparse/testing/testing_zspmv sparse/testing/testing_zspmv.o \

    -L./testing -ltest \

    -L./lib -lmagma_sparse -lmagma \

    -L/sw/cuda/10.1/lib64 -L/sw/intel/mkl/lib/intel64 -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core \

    -lpthread -lstdc++ -lm -lgfortran -lcublas -lcusparse -lcudart -lcudadevrt -lnvToolsExt

Mark

[Tahir Malas]

Yes, it was the order of libraries, when I changed the order as you mentioned, it worked.

Thanks! That was a big help.

Tahir