Skip to first unread message
Jim Galarowicz
May 15, 2019, 2:47:18 PM5/15/19
to OpenBLAS-users
Hi all,
I am working on the ECP project (https://www.exascaleproject.org/) assisting project teams with getting their projects/codes running on different platforms, ultimately using gitlab-ci.
One of the teams is trying to compile a set of proxy applications on IBM Power 9 systems. We were lots of errors when using openblas with the xl compilers to compile the proxy applications.
The errors were similar to what I'm seeing below when trying to compile openblas with xl compilers on the same IBM Power 9 system at ORNL.
$ uname -a
Linux login1 4.14.0-49.18.1.el7a.ppc64le #1 SMP Thu Nov 29 03:27:24 EST 2018 ppc64le ppc64le ppc64le GNU/Linux
Linux login1 4.14.0-49.18.1.el7a.ppc64le #1 SMP Thu Nov 29 03:27:24 EST 2018 ppc64le ppc64le ppc64le GNU/Linux
$ tail of /proc/cpuinfo
processor : 127
cpu : POWER9, altivec supported
clock : 3800.000000MHz
revision : 2.1 (pvr 004e 1201)
cpu : POWER9, altivec supported
clock : 3800.000000MHz
revision : 2.1 (pvr 004e 1201)
We are using spack to compile/build.
I tried building openblas with xl compilers and I'm getting these errors.
We may be on the wrong track, so thought I would ask the experts about our approach.
I tried 0.3.5 and 0.3.6 (shown below)
Thanks,
Jim G
Here is part of the output:
==> Installing openblas
==> Searching for binary cache of openblas
==> Warning: No Spack mirrors are currently configured
==> No binary for openblas found: installing from source
==> Using cached archive: /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/var/spack/cache/openblas/openblas-0.3.5.tar.gz
==> Staging archive: /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/var/spack/stage/openblas-0.3.5-sk6pbiernzr32cv64ooppcidpzgkjpre/v0.3.5.tar.gz
==> Created stage in /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/var/spack/stage/openblas-0.3.5-sk6pbiernzr32cv64ooppcidpzgkjpre
==> No patches needed for openblas
==> Building openblas [MakefilePackage]
==> Executing phase: 'edit'
==> Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
'make' '-j128' 'CC=/autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/lib/spack/env/xl/xlc' 'FC=/autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/lib/spack/env/xl/xlf90' 'MAKE_NB_JOBS=0' 'USE_OPENMP=0' 'USE_THREAD=0' 'libs' 'netlib' 'shared'
59 errors found in build log:
5893 3 warnings generated.
5894 3 warnings generated.
5895 7 warnings generated.
5896 3 warnings generated.
5897 ar -ru ../../libopenblas_power8-r0.3.5.a memory.o xerbla.o c_abs.o z_abs.o openblas_set_num_threads.o openblas_get_num_threads.o openblas_get_num_procs.o openblas_get_config
.o openblas_get_parallel.o openblas_error_handle.o openblas_env.o parameter.o
5898 make[1]: Leaving directory `/tmp/jgalarowicz/spack-stage/spack-stage-uFICs9/OpenBLAS-0.3.5/driver/others'
>> 5899 /sw/ascent/xl/16.1.1-1/xlC/16.1.1/bin/.orig/xlc: error: 1501-294 No input file specified. Please use -qhelp for more information.
5900 expr: syntax error
>> 5901 /sw/ascent/xl/16.1.1-1/xlC/16.1.1/bin/.orig/xlc: error: 1501-294 No input file specified. Please use -qhelp for more information.
5902 expr: syntax error
5903 make[1]: Entering directory `/tmp/jgalarowicz/spack-stage/spack-stage-uFICs9/OpenBLAS-0.3.5/kernel'
5904 /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/lib/spack/env/xl/xlc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -fno-fast-math -DMAX_STACK_ALLOC=
2048 -Wall -m64 -DF_INTERFACE_IBM -fPIC -DNO_WARMUP -DMAX_CPU_NUMBER=128 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.5\" -DASMNAME=samax_k -DASMFNAME=samax_k -DNAME=samax_k -DCNA
ME=samax_k -DCHAR_NAME=\"samax_k\" -DCHAR_CNAME=\"samax_k\" -DNO_AFFINITY -I.. -UDOUBLE -UCOMPLEX -UCOMPLEX -UDOUBLE -DUSE_ABS -UUSE_MIN ../kernel/power/amax.S -o samax_k.o
5905 /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/lib/spack/env/xl/xlc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -fno-fast-math -DMAX_STACK_ALLOC=
2048 -Wall -m64 -DF_INTERFACE_IBM -fPIC -DNO_WARMUP -DMAX_CPU_NUMBER=128 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.5\" -DASMNAME=samin_k -DASMFNAME=samin_k -DNAME=samin_k -DCNA
ME=samin_k -DCHAR_NAME=\"samin_k\" -DCHAR_CNAME=\"samin_k\" -DNO_AFFINITY -I.. -UDOUBLE -UCOMPLEX -UCOMPLEX -UDOUBLE -DUSE_ABS -DUSE_MIN ../kernel/power/amin.S -o samin_k.o
5906 /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/lib/spack/env/xl/xlc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -fno-fast-math -DMAX_STACK_ALLOC=
2048 -Wall -m64 -DF_INTERFACE_IBM -fPIC -DNO_WARMUP -DMAX_CPU_NUMBER=128 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.5\" -DASMNAME=smax_k -DASMFNAME=smax_k -DNAME=smax_k -DCNAME=
smax_k -DCHAR_NAME=\"smax_k\" -DCHAR_CNAME=\"smax_k\" -DNO_AFFINITY -I.. -UDOUBLE -UCOMPLEX -UCOMPLEX -UDOUBLE -UUSE_ABS -UUSE_MIN ../kernel/power/max.S -o smax_k.o
5907 /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/lib/spack/env/xl/xlc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -fno-fast-math -DMAX_STACK_ALLOC=
2048 -Wall -m64 -DF_INTERFACE_IBM -fPIC -DNO_WARMUP -DMAX_CPU_NUMBER=128 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.5\" -DASMNAME=smin_k -DASMFNAME=smin_k -DNAME=smin_k -DCNAME=
smin_k -DCHAR_NAME=\"smin_k\" -DCHAR_CNAME=\"smin_k\" -DNO_AFFINITY -I.. -UDOUBLE -UCOMPLEX -UCOMPLEX -UDOUBLE -UUSE_ABS -DUSE_MIN ../kernel/power/min.S -o smin_k.o
...
6547 #define STACKSIZE 320
6548 ^
6549 warning: 1540-5200 The option "-mvsx" is not supported.
6550 warning: 1540-5200 The option "-malign-power" is not supported.
6551 warning: 1540-5200 The option "-fno-fast-math" is not supported.
6552 In file included from ../kernel/power/sasum.c:50:
>> 6553 ../kernel/power/sasum_microk_power8.c:172:8: error: unknown register name 'vs32' in asm
6554 "vs32","vs33","vs34","vs35","vs36","vs37","vs38","vs39",
6555 ^
6556 3 warnings and 1 error generated.
>> 6557 Error while processing ../kernel/power/sasum.c.
6558 warning: 1540-5200 The option "-mvsx" is not supported.
6559 warning: 1540-5200 The option "-malign-power" is not supported.
6560 warning: 1540-5200 The option "-fno-fast-math" is not supported.
>> 6561 make[1]: *** [sasum_k.o] Error 1
6562 make[1]: *** Waiting for unfinished jobs....
6563 In file included from ../kernel/power/sscal.c:39:
>> 6564 ../kernel/power/sscal_microk_power8.c:136:8: error: unknown register name 'vs32' in asm
6565 "vs32","vs33","vs34","vs35","vs36","vs37","vs38","vs39",
6566 ^
6567 3 warnings and 1 error generated.
>> 6568 Error while processing ../kernel/power/sscal.c.
>> 6569 make[1]: *** [sscal_k.o] Error 1
6570 warning: 1540-5200 The option "-mvsx" is not supported.
6571 warning: 1540-5200 The option "-malign-power" is not supported.
6572 warning: 1540-5200 The option "-fno-fast-math" is not supported.
6573 3 warnings generated.
6574 warning: 1540-5200 The option "-mvsx" is not supported.
6575 warning: 1540-5200 The option "-malign-power" is not supported.
...
6590 warning: 1540-5200 The option "-malign-power" is not supported.
6591 warning: 1540-5200 The option "-fno-fast-math" is not supported.
6592 warning: 1540-5200 The option "-mvsx" is not supported.
6593 warning: 1540-5200 The option "-malign-power" is not supported.
6594 warning: 1540-5200 The option "-fno-fast-math" is not supported.
6595 In file included from ../kernel/power/sdot.c:39:
>> 6596 ../kernel/power/sdot_microk_power8.c:169:8: error: unknown register name 'vs32' in asm
6597 "vs32","vs33","vs34","vs35","vs36","vs37","vs38","vs39",
6598 ^
6599 3 warnings and 1 error generated.
>> 6600 Error while processing ../kernel/power/sdot.c.
6601 In file included from ../kernel/power/sdot.c:39:
>> 6602 ../kernel/power/sdot_microk_power8.c:169:8: error: unknown register name 'vs32' in asm
6603 "vs32","vs33","vs34","vs35","vs36","vs37","vs38","vs39",
6604 ^
>> 6605 make[1]: *** [dsdot_k.o] Error 1
6606 3 warnings and 1 error generated.
>> 6607 Error while processing ../kernel/power/sdot.c.
6608 In file included from ../kernel/power/sdot.c:39:
>> 6609 ../kernel/power/sdot_microk_power8.c:169:8: error: unknown register name 'vs32' in asm
6610 "vs32","vs33","vs34","vs35","vs36","vs37","vs38","vs39",
6611 ^
==> Searching for binary cache of openblas
==> Warning: No Spack mirrors are currently configured
==> No binary for openblas found: installing from source
==> Using cached archive: /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/var/spack/cache/openblas/openblas-0.3.5.tar.gz
==> Staging archive: /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/var/spack/stage/openblas-0.3.5-sk6pbiernzr32cv64ooppcidpzgkjpre/v0.3.5.tar.gz
==> Created stage in /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/var/spack/stage/openblas-0.3.5-sk6pbiernzr32cv64ooppcidpzgkjpre
==> No patches needed for openblas
==> Building openblas [MakefilePackage]
==> Executing phase: 'edit'
==> Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
'make' '-j128' 'CC=/autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/lib/spack/env/xl/xlc' 'FC=/autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/lib/spack/env/xl/xlf90' 'MAKE_NB_JOBS=0' 'USE_OPENMP=0' 'USE_THREAD=0' 'libs' 'netlib' 'shared'
59 errors found in build log:
5893 3 warnings generated.
5894 3 warnings generated.
5895 7 warnings generated.
5896 3 warnings generated.
5897 ar -ru ../../libopenblas_power8-r0.3.5.a memory.o xerbla.o c_abs.o z_abs.o openblas_set_num_threads.o openblas_get_num_threads.o openblas_get_num_procs.o openblas_get_config
.o openblas_get_parallel.o openblas_error_handle.o openblas_env.o parameter.o
5898 make[1]: Leaving directory `/tmp/jgalarowicz/spack-stage/spack-stage-uFICs9/OpenBLAS-0.3.5/driver/others'
>> 5899 /sw/ascent/xl/16.1.1-1/xlC/16.1.1/bin/.orig/xlc: error: 1501-294 No input file specified. Please use -qhelp for more information.
5900 expr: syntax error
>> 5901 /sw/ascent/xl/16.1.1-1/xlC/16.1.1/bin/.orig/xlc: error: 1501-294 No input file specified. Please use -qhelp for more information.
5902 expr: syntax error
5903 make[1]: Entering directory `/tmp/jgalarowicz/spack-stage/spack-stage-uFICs9/OpenBLAS-0.3.5/kernel'
5904 /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/lib/spack/env/xl/xlc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -fno-fast-math -DMAX_STACK_ALLOC=
2048 -Wall -m64 -DF_INTERFACE_IBM -fPIC -DNO_WARMUP -DMAX_CPU_NUMBER=128 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.5\" -DASMNAME=samax_k -DASMFNAME=samax_k -DNAME=samax_k -DCNA
ME=samax_k -DCHAR_NAME=\"samax_k\" -DCHAR_CNAME=\"samax_k\" -DNO_AFFINITY -I.. -UDOUBLE -UCOMPLEX -UCOMPLEX -UDOUBLE -DUSE_ABS -UUSE_MIN ../kernel/power/amax.S -o samax_k.o
5905 /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/lib/spack/env/xl/xlc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -fno-fast-math -DMAX_STACK_ALLOC=
2048 -Wall -m64 -DF_INTERFACE_IBM -fPIC -DNO_WARMUP -DMAX_CPU_NUMBER=128 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.5\" -DASMNAME=samin_k -DASMFNAME=samin_k -DNAME=samin_k -DCNA
ME=samin_k -DCHAR_NAME=\"samin_k\" -DCHAR_CNAME=\"samin_k\" -DNO_AFFINITY -I.. -UDOUBLE -UCOMPLEX -UCOMPLEX -UDOUBLE -DUSE_ABS -DUSE_MIN ../kernel/power/amin.S -o samin_k.o
5906 /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/lib/spack/env/xl/xlc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -fno-fast-math -DMAX_STACK_ALLOC=
2048 -Wall -m64 -DF_INTERFACE_IBM -fPIC -DNO_WARMUP -DMAX_CPU_NUMBER=128 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.5\" -DASMNAME=smax_k -DASMFNAME=smax_k -DNAME=smax_k -DCNAME=
smax_k -DCHAR_NAME=\"smax_k\" -DCHAR_CNAME=\"smax_k\" -DNO_AFFINITY -I.. -UDOUBLE -UCOMPLEX -UCOMPLEX -UDOUBLE -UUSE_ABS -UUSE_MIN ../kernel/power/max.S -o smax_k.o
5907 /autofs/nccsopen-svm1_home/jgalarowicz/ciminiqmc-project/spack/lib/spack/env/xl/xlc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -fno-fast-math -DMAX_STACK_ALLOC=
2048 -Wall -m64 -DF_INTERFACE_IBM -fPIC -DNO_WARMUP -DMAX_CPU_NUMBER=128 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.5\" -DASMNAME=smin_k -DASMFNAME=smin_k -DNAME=smin_k -DCNAME=
smin_k -DCHAR_NAME=\"smin_k\" -DCHAR_CNAME=\"smin_k\" -DNO_AFFINITY -I.. -UDOUBLE -UCOMPLEX -UCOMPLEX -UDOUBLE -UUSE_ABS -DUSE_MIN ../kernel/power/min.S -o smin_k.o
...
6547 #define STACKSIZE 320
6548 ^
6549 warning: 1540-5200 The option "-mvsx" is not supported.
6550 warning: 1540-5200 The option "-malign-power" is not supported.
6551 warning: 1540-5200 The option "-fno-fast-math" is not supported.
6552 In file included from ../kernel/power/sasum.c:50:
>> 6553 ../kernel/power/sasum_microk_power8.c:172:8: error: unknown register name 'vs32' in asm
6554 "vs32","vs33","vs34","vs35","vs36","vs37","vs38","vs39",
6555 ^
6556 3 warnings and 1 error generated.
>> 6557 Error while processing ../kernel/power/sasum.c.
6558 warning: 1540-5200 The option "-mvsx" is not supported.
6559 warning: 1540-5200 The option "-malign-power" is not supported.
6560 warning: 1540-5200 The option "-fno-fast-math" is not supported.
>> 6561 make[1]: *** [sasum_k.o] Error 1
6562 make[1]: *** Waiting for unfinished jobs....
6563 In file included from ../kernel/power/sscal.c:39:
>> 6564 ../kernel/power/sscal_microk_power8.c:136:8: error: unknown register name 'vs32' in asm
6565 "vs32","vs33","vs34","vs35","vs36","vs37","vs38","vs39",
6566 ^
6567 3 warnings and 1 error generated.
>> 6568 Error while processing ../kernel/power/sscal.c.
>> 6569 make[1]: *** [sscal_k.o] Error 1
6570 warning: 1540-5200 The option "-mvsx" is not supported.
6571 warning: 1540-5200 The option "-malign-power" is not supported.
6572 warning: 1540-5200 The option "-fno-fast-math" is not supported.
6573 3 warnings generated.
6574 warning: 1540-5200 The option "-mvsx" is not supported.
6575 warning: 1540-5200 The option "-malign-power" is not supported.
...
6590 warning: 1540-5200 The option "-malign-power" is not supported.
6591 warning: 1540-5200 The option "-fno-fast-math" is not supported.
6592 warning: 1540-5200 The option "-mvsx" is not supported.
6593 warning: 1540-5200 The option "-malign-power" is not supported.
6594 warning: 1540-5200 The option "-fno-fast-math" is not supported.
6595 In file included from ../kernel/power/sdot.c:39:
>> 6596 ../kernel/power/sdot_microk_power8.c:169:8: error: unknown register name 'vs32' in asm
6597 "vs32","vs33","vs34","vs35","vs36","vs37","vs38","vs39",
6598 ^
6599 3 warnings and 1 error generated.
>> 6600 Error while processing ../kernel/power/sdot.c.
6601 In file included from ../kernel/power/sdot.c:39:
>> 6602 ../kernel/power/sdot_microk_power8.c:169:8: error: unknown register name 'vs32' in asm
6603 "vs32","vs33","vs34","vs35","vs36","vs37","vs38","vs39",
6604 ^
>> 6605 make[1]: *** [dsdot_k.o] Error 1
6606 3 warnings and 1 error generated.
>> 6607 Error while processing ../kernel/power/sdot.c.
6608 In file included from ../kernel/power/sdot.c:39:
>> 6609 ../kernel/power/sdot_microk_power8.c:169:8: error: unknown register name 'vs32' in asm
6610 "vs32","vs33","vs34","vs35","vs36","vs37","vs38","vs39",
6611 ^
martin-frbg
May 20, 2019, 10:34:07 AM5/20/19
to OpenBLAS-users
ppc64le is only known to compile with gcc right now. The xlc compiler - apparently based on clang (?) does not seem to support all register names and constraints that gcc/gasm does. See issue 1699 in the project's issue tracker on github https://github.com/xianyi/OpenBLAS/issues/1699 and related links therein, in particular the reaction of IBM's David Edelsohn in https://github.com/xianyi/OpenBLAS/issues/1078#issuecomment-332984211 Renaming from "vs" to the alternate "v" names only brings up the next incompatibility.
Jim Galarowicz
May 20, 2019, 3:56:10 PM5/20/19
to OpenBLAS-users
Hi - Thank you for the reply! I appreciate your response. We will proceed with gcc. Since I posted this I've found out that there are some people working on trying to get a xl compiled version, but haven't had total success so far.
Jim G
Reply all
Reply to author
Forward
0 new messages