9.1.beta0: build problem using Homebrew's openblas

[Markus Wageringel]

On macOS 10.13.6, trying to make use of #27870 using Homebrew's openblas, the build fails for me at cvxopt:

    gcc -bundle -undefined dynamic_lookup -L/Applications/SageMath/local/lib -Wl,-rpath,/Applications/SageMath/local/lib -L/usr/local/opt/openssl/lib -L. -L/Applications/SageMath/local/lib -Wl,-rpath,/Applications/SageMath/local/lib -L/usr/local/opt/openssl/lib -L/Applications/SageMath/local/lib -Wl,-rpath,/Applications/SageMath/local/lib build/temp.macosx-10.9-x86_64-3.7/src/C/gsl.o -L/Applications/SageMath/local -L/Applications/SageMath/local -L/Applications/SageMath/local/lib -lm -lgsl -lopenblas -o build/lib.macosx-10.9-x86_64-3.7/cvxopt/gsl.cpython-37m-darwin.so

    ld: library not found for -lopenblas

    clang: error: linker command failed with exit code 1 (use -v to see invocation)

    error: command 'gcc' failed with exit status 1

    Running setup.py install for cvxopt: finished with status 'error'

As openblas and readline are keg-only, I ran configure as

    ./configure \
      LDFLAGS="-L/usr/local/opt/readline/lib -L/usr/local/opt/openblas/lib -L/usr/local/opt/mpfr/lib" \
      CPPFLAGS="-I/usr/local/opt/readline/include -I/usr/local/opt/openblas/include" \
      PKG_CONFIG_PATH="/usr/local/opt/readline/lib/pkgconfig:/usr/local/opt/openblas/lib/pkgconfig"

but cvxopt does not seem to use this. Other packages depending on BLAS like fflas_ffpack and suitsparse build successfully though, correctly using -L/usr/local/opt/openblas/lib.

On the other hand, even though numpy builds successfully, the logs look like it does not find openblas either, but it picks the macOS blas from the Accelerate framework instead.

[Dima Pasechnik]

On Sun, Jan 12, 2020 at 12:10 AM Markus Wageringel
<markus.w...@gmail.com> wrote:
>
> On macOS 10.13.6, trying to make use of #27870 using Homebrew's openblas, the build fails for me at cvxopt:
>
> gcc -bundle -undefined dynamic_lookup -L/Applications/SageMath/local/lib -Wl,-rpath,/Applications/SageMath/local/lib -L/usr/local/opt/openssl/lib -L. -L/Applications/SageMath/local/lib -Wl,-rpath,/Applications/SageMath/local/lib -L/usr/local/opt/openssl/lib -L/Applications/SageMath/local/lib -Wl,-rpath,/Applications/SageMath/local/lib build/temp.macosx-10.9-x86_64-3.7/src/C/gsl.o -L/Applications/SageMath/local -L/Applications/SageMath/local -L/Applications/SageMath/local/lib -lm -lgsl -lopenblas -o build/lib.macosx-10.9-x86_64-3.7/cvxopt/gsl.cpython-37m-darwin.so
> ld: library not found for -lopenblas
> clang: error: linker command failed with exit code 1 (use -v to see invocation)
> error: command 'gcc' failed with exit status 1
> Running setup.py install for cvxopt: finished with status 'error'
>
> As openblas and readline are keg-only, I ran configure as
>
> ./configure \
> LDFLAGS="-L/usr/local/opt/readline/lib -L/usr/local/opt/openblas/lib -L/usr/local/opt/mpfr/lib" \
> CPPFLAGS="-I/usr/local/opt/readline/include -I/usr/local/opt/openblas/include" \
> PKG_CONFIG_PATH="/usr/local/opt/readline/lib/pkgconfig:/usr/local/opt/openblas/lib/pkgconfig"
>
> but cvxopt does not seem to use this.

cvxopt really needs env.vars CVXOPT_BLAS_LIB_DIR, CVXOPT_BLAS_LIB,
CVXOPT_LAPACK_LIB set, it does not use pkgconf.
(but it should).
We can propose an upstream fix for this, and meanwhile set these vars
in spkg-install

> Other packages depending on BLAS like fflas_ffpack and suitsparse build successfully though, correctly using -L/usr/local/opt/openblas/lib.
>
> On the other hand, even though numpy builds successfully, the logs look like it does not find openblas either, but it picks the macOS blas from the Accelerate framework instead.

numpy uses a file called site.cfg for these sorts of config settings,
which in turn is created by
build/pkgs/numpy/lapack_conf.py

So one can hack the latter.

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/038a20b3-7486-41bb-b2a5-c4ee189c6739%40googlegroups.com.

[Dima Pasechnik]

one obvious thing to try is to get rid of

if not (os.environ['UNAME'] == 'Darwin'):

in build/pkgs/numpy/lapack_conf.py

[Dima Pasechnik]

On Sun, Jan 12, 2020 at 9:58 AM Dima Pasechnik <dim...@gmail.com> wrote:
>

> On Sun, Jan 12, 2020 at 12:10 AM Markus Wageringel
> <markus.w...@gmail.com> wrote:
> >
> > On macOS 10.13.6, trying to make use of #27870 using Homebrew's openblas, the build fails for me at cvxopt:
> >
> > gcc -bundle -undefined dynamic_lookup -L/Applications/SageMath/local/lib -Wl,-rpath,/Applications/SageMath/local/lib -L/usr/local/opt/openssl/lib -L. -L/Applications/SageMath/local/lib -Wl,-rpath,/Applications/SageMath/local/lib -L/usr/local/opt/openssl/lib -L/Applications/SageMath/local/lib -Wl,-rpath,/Applications/SageMath/local/lib build/temp.macosx-10.9-x86_64-3.7/src/C/gsl.o -L/Applications/SageMath/local -L/Applications/SageMath/local -L/Applications/SageMath/local/lib -lm -lgsl -lopenblas -o build/lib.macosx-10.9-x86_64-3.7/cvxopt/gsl.cpython-37m-darwin.so
> > ld: library not found for -lopenblas
> > clang: error: linker command failed with exit code 1 (use -v to see invocation)
> > error: command 'gcc' failed with exit status 1
> > Running setup.py install for cvxopt: finished with status 'error'
> >
> > As openblas and readline are keg-only, I ran configure as
> >
> > ./configure \
> > LDFLAGS="-L/usr/local/opt/readline/lib -L/usr/local/opt/openblas/lib -L/usr/local/opt/mpfr/lib" \
> > CPPFLAGS="-I/usr/local/opt/readline/include -I/usr/local/opt/openblas/include" \
> > PKG_CONFIG_PATH="/usr/local/opt/readline/lib/pkgconfig:/usr/local/opt/openblas/lib/pkgconfig"
> >
> > but cvxopt does not seem to use this.
>
> cvxopt really needs env.vars CVXOPT_BLAS_LIB_DIR, CVXOPT_BLAS_LIB,
> CVXOPT_LAPACK_LIB set, it does not use pkgconf.
> (but it should).
> We can propose an upstream fix for this, and meanwhile set these vars
> in spkg-install

please see https://github.com/cvxopt/cvxopt/issues/156

[Markus Wageringel]

Thanks for the quick response, Dima. With the change to lapack_conf.py, the file site.cfg looks correct:

[ALL]

library_dirs = /Applications/SageMath/local/lib

include_dirs = /Applications/SageMath/local/include

[blas]

include_dirs = /usr/local/Cellar/openblas/0.3.7/include

library_dirs = /usr/local/Cellar/openblas/0.3.7/lib

blas_libs    = openblas

[lapack]

library_dirs = /usr/local/Cellar/openblas/0.3.7/lib

lapack_libs  = openblas

I ran `make numpy-clean && make numpy`, but numpy still does not find openblas (logs attached).

[Markus Wageringel]

I manually set those CVXOPT_... variables in the spkg-install file for now and building cvxopt then succeeded.

Building Sage now almost completes, but the file src/sage/libs/readline.pyx fails to compile with these errors:

    [1/1] gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I./sage/cpython -Isage/cpython -I/Applications/SageMath/local/include -I/Applications/SageMath/src -I/Applications/SageMath/src/sage/ext -I/Applications/SageMath/local/include/python3.7m -I/Applications/SageMath/local/lib/python3.7/site-packages/numpy/core/include -Ibuild/cythonized -I/Applications/SageMath/local/include/python3.7m -c build/cythonized/sage/libs/readline.c -o build/temp.macosx-10.9-x86_64-3.7/build/cythonized/sage/libs/readline.o -fno-strict-aliasing -DCYTHON_CLINE_IN_TRACEBACK=1 -std=c99

    build/cythonized/sage/libs/readline.c:1446:36: error: use of undeclared identifier 'rl_catch_signals'

      __pyx_t_1 = __Pyx_PyInt_From_int(rl_catch_signals); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 87, __pyx_L1_error)

                                       ^

    build/cythonized/sage/libs/readline.c:1468:36: error: use of undeclared identifier 'rl_catch_sigwinch'

      __pyx_t_1 = __Pyx_PyInt_From_int(rl_catch_sigwinch); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 88, __pyx_L1_error)

                                       ^

    build/cythonized/sage/libs/readline.c:1542:36: warning: implicit declaration of function 'rl_set_signals' is invalid in C99 [-Wimplicit-function-declaration]

      __pyx_t_1 = __Pyx_PyInt_From_int(rl_set_signals()); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 104, __pyx_L1_error)

                                       ^

    build/cythonized/sage/libs/readline.c:1604:36: warning: implicit declaration of function 'rl_clear_signals' is invalid in C99 [-Wimplicit-function-declaration]

      __pyx_t_1 = __Pyx_PyInt_From_int(rl_clear_signals()); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 120, __pyx_L1_error)

                                       ^

    build/cythonized/sage/libs/readline.c:1849:36: error: passing 'void' to parameter of incompatible type 'int'

      __pyx_t_1 = __Pyx_PyInt_From_int(rl_forced_update_display()); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 190, __pyx_L1_error)

                                       ^~~~~~~~~~~~~~~~~~~~~~~~~~

    ...

The Homebrew readline is version 8.0.1. In particular, the header file `/usr/local/opt/readline/include/readline/readline.h` does contain `rl_catch_signals`, `rl_catch_sigwinch` and `rl_clear_signals`.

The system readline in `/usr/include/readline/readline.h` does not contain these though, so it looks like that is used instead.

[Dima Pasechnik]

On Sun, Jan 12, 2020 at 1:37 PM Markus Wageringel
<markus.w...@gmail.com> wrote:
>
> I manually set those CVXOPT_... variables in the spkg-install file for now and building cvxopt then succeeded.
>
> Building Sage now almost completes, but the file src/sage/libs/readline.pyx fails to compile with these errors:
>
>
> [1/1] gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I./sage/cpython -Isage/cpython -I/Applications/SageMath/local/include -I/Applications/SageMath/src -I/Applications/SageMath/src/sage/ext -I/Applications/SageMath/local/include/python3.7m -I/Applications/SageMath/local/lib/python3.7/site-packages/numpy/core/include -Ibuild/cythonized -I/Applications/SageMath/local/include/python3.7m -c build/cythonized/sage/libs/readline.c -o build/temp.macosx-10.9-x86_64-3.7/build/cythonized/sage/libs/readline.o -fno-strict-aliasing -DCYTHON_CLINE_IN_TRACEBACK=1 -std=c99
> build/cythonized/sage/libs/readline.c:1446:36: error: use of undeclared identifier 'rl_catch_signals'
> __pyx_t_1 = __Pyx_PyInt_From_int(rl_catch_signals); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 87, __pyx_L1_error)
> ^
> build/cythonized/sage/libs/readline.c:1468:36: error: use of undeclared identifier 'rl_catch_sigwinch'
> __pyx_t_1 = __Pyx_PyInt_From_int(rl_catch_sigwinch); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 88, __pyx_L1_error)
> ^
> build/cythonized/sage/libs/readline.c:1542:36: warning: implicit declaration of function 'rl_set_signals' is invalid in C99 [-Wimplicit-function-declaration]
> __pyx_t_1 = __Pyx_PyInt_From_int(rl_set_signals()); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 104, __pyx_L1_error)
> ^
> build/cythonized/sage/libs/readline.c:1604:36: warning: implicit declaration of function 'rl_clear_signals' is invalid in C99 [-Wimplicit-function-declaration]
> __pyx_t_1 = __Pyx_PyInt_From_int(rl_clear_signals()); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 120, __pyx_L1_error)
> ^
> build/cythonized/sage/libs/readline.c:1849:36: error: passing 'void' to parameter of incompatible type 'int'
> __pyx_t_1 = __Pyx_PyInt_From_int(rl_forced_update_display()); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 190, __pyx_L1_error)
> ^~~~~~~~~~~~~~~~~~~~~~~~~~
> ...
>
>
> The Homebrew readline is version 8.0.1. In particular, the header file `/usr/local/opt/readline/include/readline/readline.h` does contain `rl_catch_signals`, `rl_catch_sigwinch` and `rl_clear_signals`.
>
> The system readline in `/usr/include/readline/readline.h` does not contain these though, so it looks like that is used instead.

One may overrule Homebrew to actually force it to install readline in /usr/local
Otherwise, it's indeed a problem.
I've opened https://trac.sagemath.org/ticket/29000 to deal with this.

>
>
> Am 12.01.2020 um 10:58 schrieb Dima Pasechnik <dim...@gmail.com>:
>
> On Sun, Jan 12, 2020 at 12:10 AM Markus Wageringel
> <markus.w...@gmail.com> wrote:
>
>
> On macOS 10.13.6, trying to make use of #27870 using Homebrew's openblas, the build fails for me at cvxopt:
>
> gcc -bundle -undefined dynamic_lookup -L/Applications/SageMath/local/lib -Wl,-rpath,/Applications/SageMath/local/lib -L/usr/local/opt/openssl/lib -L. -L/Applications/SageMath/local/lib -Wl,-rpath,/Applications/SageMath/local/lib -L/usr/local/opt/openssl/lib -L/Applications/SageMath/local/lib -Wl,-rpath,/Applications/SageMath/local/lib build/temp.macosx-10.9-x86_64-3.7/src/C/gsl.o -L/Applications/SageMath/local -L/Applications/SageMath/local -L/Applications/SageMath/local/lib -lm -lgsl -lopenblas -o build/lib.macosx-10.9-x86_64-3.7/cvxopt/gsl.cpython-37m-darwin.so
> ld: library not found for -lopenblas
> clang: error: linker command failed with exit code 1 (use -v to see invocation)
> error: command 'gcc' failed with exit status 1
> Running setup.py install for cvxopt: finished with status 'error'
>
> As openblas and readline are keg-only, I ran configure as
>
> ./configure \
> LDFLAGS="-L/usr/local/opt/readline/lib -L/usr/local/opt/openblas/lib -L/usr/local/opt/mpfr/lib" \
> CPPFLAGS="-I/usr/local/opt/readline/include -I/usr/local/opt/openblas/include" \
> PKG_CONFIG_PATH="/usr/local/opt/readline/lib/pkgconfig:/usr/local/opt/openblas/lib/pkgconfig"
>
> but cvxopt does not seem to use this.
>
>
> cvxopt really needs env.vars CVXOPT_BLAS_LIB_DIR, CVXOPT_BLAS_LIB,
> CVXOPT_LAPACK_LIB set, it does not use pkgconf.
> (but it should).
> We can propose an upstream fix for this, and meanwhile set these vars
> in spkg-install
>
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/BBB346E3-DBD1-44B4-AD7A-5D4B0B9D1BF5%40gmail.com.