Problem With SWAP After Compiling With 64-bit integers

[Xiniz Ronda]

Hi,

As we talked last to be able to have large matrix dimension I changed the makefile like this to get the benefit of 64-bit integers:

cp ../make.inc-examples/make.inc.mkl-gcc-ilp64 ../make.inc

And make.inc look like bellow:

GPU_TARGET ?= Ampere Volta


# --------------------
# programs

CC        = gcc
CXX       = g++
NVCC      = nvcc
FORT      = gfortran

ARCH      = ar
ARCHFLAGS = cr
RANLIB    = ranlib


# --------------------
# flags

# Use -fPIC to make shared (.so) and static (.a) library;
# can be commented out if making only static library.
FPIC      = -fPIC

CFLAGS    = -O3 $(FPIC) -fopenmp -DNDEBUG -DADD_ -Wall -Wshadow -DMAGMA_WITH_MKL
FFLAGS    = -O3 $(FPIC)          -DNDEBUG -DADD_ -Wall -Wno-unused-dummy-argument
F90FLAGS  = -O3 $(FPIC)          -DNDEBUG -DADD_ -Wall -Wno-unused-dummy-argument -x f95-cpp-input
NVCCFLAGS = -O3                  -DNDEBUG -DADD_ -Xcompiler "$(FPIC) -Wall -Wno-unused-function" -std=c++11
LDFLAGS   =     $(FPIC) -fopenmp

# Defining MAGMA_ILP64 or MKL_ILP64 changes magma_int_t to int64_t in include/magma_types.h
CFLAGS    += -DMKL_ILP64
FFLAGS    += -fdefault-integer-8
F90FLAGS  += -fdefault-integer-8
NVCCFLAGS += -DMKL_ILP64

# Options to do extra checks for non-standard things like variable length arrays;
# it is safe to disable all these
CFLAGS   += -pedantic -Wno-long-long
#CFLAGS   += -Werror  # uncomment to ensure all warnings are dealt with

# C++11 (gcc >= 4.7) is not required, but has benefits like atomic operations
CXXFLAGS := $(CFLAGS) -std=c++11
CFLAGS   += -std=c99

# --------------------
# libraries

# IMPORTANT: these link lines are for 64-bit int !!!!
# For regular 64-bit builds using 64-bit pointers and 32-bit int,
# use the lp64 library, not the ilp64 library. See make.inc.mkl-gcc or make.inc.mkl-icc.

# see MKL Link Advisor at http://software.intel.com/sites/products/mkl/
# gcc/gfortran with MKL 10.3, GNU OpenMP threads, 64-bit int (use -fopenmp in CFLAGS, LDFLAGS)
# note -DMAGMA_ILP64 or -DMKL_ILP64, and -fdefault-integer-8 in FLAGS above
LIB       = -lmkl_gf_ilp64 -lmkl_gnu_thread -lmkl_core -lpthread -lstdc++ -lm -lgfortran

# Supposedly, gcc can use Intel threads (libiomp5) instead, but be careful that
# libiomp5 and libgomp are NOT BOTH linked. Above, we use gnu threads as a safer option.
# gcc/gfortran with MKL 10.3, Intel OpenMP threads, 64-bit int (remove -fopenmp from LDFLAGS above)
#LIB       = -lmkl_gf_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lstdc++ -lm -lgfortran

LIB      += -lcublas -lcusparse -lcudart -lcudadevrt


# --------------------
# directories

# define library directories preferably in your environment, or here.
# for MKL run, e.g.: source /opt/intel/composerxe/mkl/bin/mklvars.sh intel64
#MKLROOT ?= /opt/intel/composerxe/mkl
#CUDADIR ?= /usr/local/cuda
-include make.check-mkl
-include make.check-cuda

LIBDIR    = -L$(CUDADIR)/lib64 \
            -L$(MKLROOT)/lib/intel64

INC       = -I$(CUDADIR)/include \
            -I$(MKLROOT)/include

After make clean, make and make install I started a test with dgetrf.

./testing_dgetrf_gpu -n 33  --version 3 --dev 1 -c

But I have an error:

Intel MKL ERROR: Parameter 6 was incorrect on entry to DLASWP.

I printed the ipiv vector like this( %lld is correct?):

       for (int i=0; i<N; i++)
               printf("ipiv[%d]=%lld\n",i+1, ipiv[i]);

ipiv[1]=30064771079
ipiv[2]=107374182426
ipiv[3]=42949672979
ipiv[4]=38654705676
ipiv[5]=85899345933
ipiv[6]=85899345947
ipiv[7]=137438953505
ipiv[8]=81604378651
ipiv[9]=141733920797
ipiv[10]=85899345949
ipiv[11]=141733920791
ipiv[12]=103079215134
ipiv[13]=128849018906
ipiv[14]=128849018910
ipiv[15]=137438953505
ipiv[16]=137438953505
ipiv[17]=5643126232787714081
ipiv[18]=6879089492090773332
ipiv[19]=7309475735947207785
ipiv[20]=3616776654095798341
ipiv[21]=6879358834543718243
ipiv[22]=6002823612316285300
ipiv[23]=8104636957431586612
ipiv[24]=3917907733236434277
ipiv[25]=7585580681123880802
ipiv[26]=6873397238825176656
ipiv[27]=7588681118894815604
ipiv[28]=7298982808718102121
ipiv[29]=7378425899888504434
ipiv[30]=4062022921311843174
ipiv[31]=7309475735945831265
ipiv[32]=7595718147995682117
ipiv[33]=3347432800179322985

It is strange to me, any help are welcome.

Cheers

[Ahmad Abdelfattah]

Can you please share the exact output of the MAGMA tester? Did you install a release of MAGMA? Or a fresh clone from its repo?

I have MAGMA installed from the repo for 64bit integers. Your test for the 33x33 matrix works fine on my side. I also tested for bigger matrices up to 10k, all seem to pass. 

If you are going to install MAGMA from the repo, please run “make lib -j <N>” from MAGMA’s root directory, and then “make -j <N>” inside the “/testing" subdirectory. This is to avoid building the sparse component, which currently conflicts with some of the interfaces in CUDA-11.x

Thanks,

Ahmad

--
You received this message because you are subscribed to the Google Groups "MAGMA User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to magma-user+...@icl.utk.edu.
To view this discussion on the web visit https://groups.google.com/a/icl.utk.edu/d/msgid/magma-user/CA%2Bzr%3DE0rhKUkbJqRZ12dKKeOiv3taYD%3DFar6C1W4Cg_M7D39FQ%40mail.gmail.com.

[Mark Gates]

Note you can do just `make -j N testing` in magma's root directory, or equivalently `make -j N` in the testing directory, where N is the number of cores for make to use. No need to `make lib` first.

# Use 20 cores

make -j 20 testing

Mark