Cross compiling OpenBLAS for ARM

[Dave Mazzoni]

Hi all -

  You've been so helpful in the past I thought I'd try a post here with my problem.

  I'm trying to cross compile OpenBLAS for the CORTEXA9 (an Arm processor). I've installed the latest stable gcc 5.4.0 which also includes the gfortran compiler. When I try this make from the OpenBLAS directory:

make CC=/usr/local/bin/gcc FC=/usr/local/bin/gfortran TARGET=CORTEXA9

I get many lines of activity, but it fails at this:

make[1]: Entering directory `/home/dmazzoni/Downloads/OpenBLAS/exports'

make[1]: warning: -jN forced in submake: disabling jobserver mode.

/usr/local/bin/gcc -O2 -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=64 -DASMNAME= -DASMFNAME=_ -DNAME=_ -DCNAME= -DCHAR_NAME=\"_\" -DCHAR_CNAME=\"\" -DNO_AFFINITY -I..  -shared -o ../libopenblas_cortexa9p-r0.2.18.dev.so \

-Wl,--whole-archive ../libopenblas_cortexa9p-r0.2.18.dev.a -Wl,--no-whole-archive \

-Wl,-soname,libopenblas.so.0 -lm -lpthread -lgfortran -lm -lpthread -lgfortran

/usr/local/bin/gcc -O2 -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=64 -DASMNAME= -DASMFNAME=_ -DNAME=_ -DCNAME= -DCHAR_NAME=\"_\" -DCHAR_CNAME=\"\" -DNO_AFFINITY -I..  -w -o linktest linktest.c ../libopenblas_cortexa9p-r0.2.18.dev.so -L/usr/local/lib/gcc/x86_64-unknown-linux-gnu/5.4.0 -L/usr/local/lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/usr/local/lib/gcc/x86_64-unknown-linux-gnu/5.4.0/../../..  -lgfortran -lm -lquadmath -lm -lc   && echo OK.

/tmp/ccuBn8Oq.o: In function `main':

linktest.c:(.text.startup+0x620): undefined reference to `spotri_'

linktest.c:(.text.startup+0x627): undefined reference to `dpotri_'

...

Any suggestions?

Thanks in advance!

Dave

[Dave Mazzoni]

OK, needed to do  make clean and then:

sudo make BINARY=32 CC=/usr/local/bin/gcc FC=/usr/local/bin/gfortran TARGET=CORTEXA9

but that err's at 

sgemm_kernel.o: no such file or directory.

Any hints?

[Vladimír Chalupecký]

I have never cross-compiled anything and I don't have a solution, just a couple of hints that might help.

I think that the core of the problem is that you are not using a cross-compiler for arm, judging from error messages above.

Check the contents of Makefile.conf. I think that you will find ARCH=x86. That causes that the makefiles will not find the sgemm kernel for your CPU defined in kernel/arm/KERNEL.ARMV7 as sgemm_kernel_4x4_vfpv3.S. That file lives in kernel/arm while in your case the makefiles most likely search for it in kernel/x86.

As for the cross-compiler, for example I am on Fedora that has gcc-arm-linux-gnu. If I use

make BINARY=32 TARGET=CORTEXA9 CC=arm-linux-gnu-gcc HOSTCC=gcc

then Makefile.conf contains correct ARCH=arm.

Unfortunately, Fedora's gcc-arm-linux-gnu does not support for cross-building user space programs so the compilation fails with 'stdio.h: No such file or directory' (or at least that's what I think is the reason, I might just as well be doing something completely wrong). Moreover, I would probably also need a cross-compiler for fortran anyway.

[Dave Mazzoni]

Thanks for the suggestions. In my case, with gcc 5.4 it supports cross compiling to many targets (including Arm). The same is the case for gfortran (part of gcc). I built it from source. My make command is:

sudo make BINARY=32 CC=/usr/local/bin/gcc FC=/usr/local/bin/gfortran HOSTCC=gcc TARGET=CORTEXA9

the TARGET *should* tell the compiler to cross-compile if it is not the same as the Host, right? The README suggest that is the way to do it, or I'm missing something.

[Dave Mazzoni]

OK, I installed gcc-arm-linux-gnueabi and gfortran-arm-linux-gnueabi, deleted and unzip'd the OpenBLAS directory and tried this:

make CC=arm-linux-gnueabi-gcc FC=arm-linux-gnueabi-gfortran HOSTCC=gcc TARGET=CORTEXA9

Everything looked great until it screeched to a stop with this:

...

/gcc-usr/lib/cross/gcc-cross/armarm-linux--linuxgnueabi/-gnueabi4.7/../../../..//4.7arm-linux/../../../../arm--gnueabi/binlinux/ld: error-gnueabi/: secondtst.obin/ld: uses failed to VFP  mergeregister arguments, testsecond does not

/usr/lib/gcc- target cross/arm-specific linux-gnueabi/4.7data/../../../../arm-linux-gnueabi/bin/ld: failed to merge target specific data of file secondtst.o

 of file dsecnd_INT_ETIME.o

/usrcollect2: error: ld returned 1 exit status

/lib/gcc-cross/arm-linux-gnueabi/4.7/../../../../arm-linux-gnueabi/bin/ld: error: dsecndtst.o uses VFP register arguments, testdsecnd does not

/usr/lib/gcc-cross/arm-linux-gnueabi/4.7/../../../../arm-linux-gnueabi/bin/ld: make[2]: failed to merge*** [testsecond] Error 1

 target specific data of file dsecndtst.o

collect2: error: ld returned 1 exit status

make[2]: *** [testdsecnd] Error 1

make[2]: Leaving directory `/home/dmazzoni/Downloads/OpenBLAS/OpenBLAS-develop/lapack-netlib/INSTALL'

make[1]: *** [lapack_install] Error 2

make[1]: Leaving directory `/home/dmazzoni/Downloads/OpenBLAS/OpenBLAS-develop/lapack-netlib'

make: *** [netlib] Error 2

So... Progress, but now the dreaded link errors.

Any suggestions?

[Vladimír Chalupecký]

/lib/gcc-cross/arm-linux-gnueabi/4.7/../../../../arm-linux-gnueabi/bin/ld: error: dsecndtst.o uses VFP register arguments, testdsecnd does not

This line suggests that the two files were not compiled with the same set of options.

From lapack-netlib/INSTALL/Makefile,

testdsecnd uses '$(LOADER) $(LOADOPTS) -o testdsecnd dsecnd_$(TIMER).o dsecndtst.o'

while dsecndtst.o uses '$(FORTRAN) $(OPTS) -c $< -o $@'

Please check what lapack-netlib/make.inc contains but it seems that your $(LOADOPTS) does not contain -mfpu=vfpv3.

If you don't need LAPACK, you can try adding NO_LAPACK=1 to your make command. Otherwise you can hack lapack-netlib/Makefile to not descend into the INSTALL directory (assuming it's not needed for lapack to compile), or try asking OpenBLAS, maybe it's a bug in their makefile system (in how they populate lapack-netlib/make.inc).

[Dave Mazzoni]

I've got the OpenBLAS make to cross-compile the library by doing this:

• Find all installed packages for the ARM compiler: sudo aptitude search arm-linux-gnueabi
• Purge ALL gnueabi versions: sudo apt-get purge --auto-remove gfortran-arm-linux-gnueabi, etc for any "installed"
• Install only gnueabihf versions: sudo aptitude install gfortran-arm-linux-gnueabihf (this also installs gnueabihf gcc)
• From the OpenBLAS directory: make CC=arm-linux-gnueabihf-gcc FC=arm-linux-gnueabihf-gfortran HOSTCC=gcc TARGET=CORTEXA9
• Install the ARM libs: sudo make PREFIX=/usr/local install

The make runs to completion and I can install the ARM OpenBLAS library. I followed similar steps to get FFTW to cross-compile if anyone is interested.

Next, installing go for cross-compiling I did this:

•  export CGO_ENABLED=1
•  export CC_FOR_TARGET=arm-linux-gnueabihf-gcc
•  export GOGCCFLAGS="-fPIC -pthread -mfpu=vfp -mfloat-abi=hard --static"
•  export GOOS=linux
•  export GOARCH=arm
•  export GOARM=7
•  bash ./all.bash
  
• 
  
• go env shows:
• GOARCH="arm"
• GOBIN=""
• GOEXE=""
• GOHOSTARCH="amd64"
• GOHOSTOS="linux"
• GOOS="linux"
• GOPATH="/opt/Projects/Golang"
• GORACE=""
• GOROOT="/usr/local/go"
• GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
• CC="arm-linux-gnueabihf-gcc"
• GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build149148529=/tmp/go-build -gno-record-gcc-switches"
• CXX="g++"
• CGO_ENABLED="1"

Notice the GOGCCFLAGS get mangled (see golang go issues 8161). I can work around that by specifying GOGCCFLAGS on the command line:

GOOS=linux GOARCH=arm GOARM=7 GOGCCFLAGS="-fPIC -pthread -mfpu=vfp -mfloat-abi=hard  --static" CC=arm-linux-gnueabihf-gcc CGO_LDFLAGS="-L/usr/local/lib -lopenblas " go build -x rand_zgeev.go

However, when I copy this to the arm system, it does not run:

 ./rand_zgeev 

-sh: ./rand_zgeev: not found

Any ideas why this won't run?