On OSX BigSur 11.4 with 9.4.beta0, ./configure picks up python3 from system while import readline is broken

[Sébastien Labbé]

Bonjour,

A weird thing happens during the ./configure of sage 9.4.beta0 on the computer of a colleague of mine running MacOS BigSur 11.4. It says the system python3 is good, but it seems it should not as `import readline` is broken.

More precisely, after git cloning 9.4.beta0,  we did make configure. Then ./configure. Then we brew install the list of suggested packages including freetype, fplll, python3, readline, etc. We got problem with not allowed permissions for few of the packages including freetype.

In particular, a problem occurs with readline:

$ brew install readline

Warning: readline 8.1 is already installed, it's just not linked.

To link this version, run:

  brew link readline

$ brew link readline

Warning: Refusing to link macOS provided/shadowed software: readline

For compilers to find readline you may need to set:

  export LDFLAGS="-L/usr/local/opt/readline/lib"

  export CPPFLAGS="-I/usr/local/opt/readline/include"

For pkg-config to find readline you may need to set:

  export PKG_CONFIG_PATH="/usr/local/opt/readline/lib/pkgconfig"

Then, running ./configure again says that python3 is ok while having some problem with it:

----------------------------------------------------------------

Checking whether SageMath should install SPKG python3...

checking whether any of bzip2 xz libffi is installed as or will be installed as SPKG... no

checking for python3 >= 3.7.0, < 3.10.0 with modules sqlite3, ctypes, math, hashlib, crypt, readline, socket, zlib, distutils.core... 

checking ... whether /usr/local/bin/python3 is good... Traceback (most recent call last):

  File "<string>", line 1, in <module>

ImportError: dlopen(/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/readline.cpython-39-darwin.so, 2): Library not loaded: /usr/local/opt/readline/lib/libreadline.8.dylib

  Referenced from: /usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/readline.cpython-39-darwin.so

  Reason: image not found

no, the version is in the supported range but cannot import one of the required modules: sqlite3, ctypes, math, hashlib, crypt, readline, socket, zlib, distutils.core

checking ... whether /usr/bin/python3 is good... yes

checking for python3 >= 3.7.0, < 3.10.0 with modules sqlite3, ctypes, math, hashlib, crypt, readline, socket, zlib, distutils.core... /usr/bin/python3

configure: will use system package and not install SPKG python3

checking whether /usr/bin/python3 is configured to build multiarch extensions... no

checking whether "-march=native" works with the C/C++ compilers configured for building extensions for /usr/bin/python3... yes

--------------------------------------------------------------------------------

Indeed the python3 provided by the system does not allow to import readline:

$ python3

Python 3.9.5 (default, May  4 2021, 03:36:27) 

[Clang 12.0.0 (clang-1200.0.32.29)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import sqlite3, ctypes, math, hashlib, crypt, readline, socket, zlib, distutils.core

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

ImportError: dlopen(/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/readline.cpython-39-darwin.so, 2): Library not loaded: /usr/local/opt/readline/lib/libreadline.8.dylib

  Referenced from: /usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/readline.cpython-39-darwin.so

  Reason: image not found

Then, we ran make and we are still waiting for the end of that (still compiling gfortran).

Is it okay that ./configure picks up python3 from the system provided import readline is broken?

Thank you,

Sébastien

[Sébastien Labbé]

Also, is it normal that

$ cat build/pkgs/gfortran/distros/homebrew.txt
gcc

? homebrew package gcc includes gfortran ?

[Dima Pasechnik]

yes
>
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/235f0411-9dff-42be-8083-2ea3fab24e3cn%40googlegroups.com.

[Dima Pasechnik]

On Thu, Jun 3, 2021 at 1:18 PM Sébastien Labbé <sla...@gmail.com> wrote:
>
> Bonjour,
>
> A weird thing happens during the ./configure of sage 9.4.beta0 on the computer of a colleague of mine running MacOS BigSur 11.4. It says the system python3 is good, but it seems it should not as `import readline` is broken.

I presume you mean Homebrew's python3 rather than system's.

% /usr/bin/python3
Python 3.8.2 (default, Dec 21 2020, 15:06:04)

[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

did he do

source .homebrew-build-env

?

This must be done in any Homebrew-based Sage build or use session.

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/89bfdb5c-3de5-495f-bf73-3e3337aad5b0n%40googlegroups.com.

[Sébastien Labbé]

did he do

source .homebrew-build-env

?

This must be done in any Homebrew-based Sage build or use session.

 Good point. We did it, but maybe not before the second ./configure. So, we redid from scratch:

$ source .homebrew-build-env

source .homebrew-build-env

Doing this does not change the python3 to /usr/bin/python3

$ which python3

/usr/local/bin/python3

$ python3

Python 3.9.5 (default, May  4 2021, 03:36:27) 

[Clang 12.0.0 (clang-1200.0.32.29)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import readline

$Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

ImportError: dlopen(/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/readline.cpython-39-darwin.so, 2): Library not loaded: /usr/local/opt/readline/lib/libreadline.8.dylib

  Referenced from: /usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/readline.cpython-39-darwin.so

  Reason: image not found

Although /usr/bin/python is really python3.7 and it can import readline correctly (I forgot to copy paste that from his computer).

Then, the following run of ./configure is weird for two reasons:

- it says that /usr/bin/python3 is okay, while an error is raised

- it says that gcc is installed on the system but gfortran is not

$ ./configure

checking for a BSD-compatible install... /usr/bin/install -c

checking whether build environment is sane... yes

checking for a thread-safe mkdir -p... config/install-sh -c -d

checking for gawk... no

checking for mawk... no

checking for nawk... no

checking for awk... awk

checking whether make sets $(MAKE)... yes

checking whether make supports nested variables... yes

checking whether to enable maintainer-specific portions of Makefiles... yes

checking for root user... no

checking build system type... x86_64-apple-darwin20.5.0

checking host system type... x86_64-apple-darwin20.5.0

checking for a sed that does not truncate output... /usr/bin/sed

checking for ar... yes

checking for m4... yes

checking for pkg-config... /usr/local/bin/pkg-config

checking pkg-config is at least version 0.29... yes

checking for ranlib... yes

checking for strip... yes

checking for GNU or BSD tar... /usr/bin/tar

checking for GNU make... /usr/bin/make

checking for Python... /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

checking for latex... yes

checking for perl... /usr/bin/perl

checking for Perl version 5.8.0 or later... yes

checking whether a conda environment is active... no

checking for gcc... gcc

checking whether the C compiler works... yes

checking for C compiler default output file name... a.out

checking for suffix of executables... 

checking whether we are cross compiling... no

checking for suffix of object files... o

checking whether we are using the GNU C compiler... yes

checking whether gcc accepts -g... yes

checking for gcc option to accept ISO C89... none needed

checking whether gcc understands -c and -o together... yes

checking whether make supports the include directive... yes (GNU style)

checking dependency style of gcc... none

checking how to run the C preprocessor... gcc -E

checking for g++... g++

checking whether we are using the GNU C++ compiler... yes

checking whether g++ accepts -g... yes

checking dependency style of g++... none

checking for gfortran... gfortran

checking whether we are using the GNU Fortran compiler... no

checking whether gfortran accepts -g... no

checking for gcc... gcc

checking whether we are using the GNU Objective C compiler... yes

checking whether gcc accepts -g... yes

checking dependency style of gcc... none

checking for g++... g++

checking whether we are using the GNU Objective C++ compiler... yes

checking whether g++ accepts -g... yes

checking dependency style of g++... none

checking how to run the C++ preprocessor... g++ -E

checking for grep that handles long lines and -e... /usr/bin/grep

checking for egrep... /usr/bin/grep -E

checking for ANSI C header files... yes

checking for sys/types.h... yes

checking for sys/stat.h... yes

checking for stdlib.h... yes

checking for string.h... yes

checking for memory.h... yes

checking for strings.h... yes

checking for inttypes.h... yes

checking for stdint.h... yes

checking for unistd.h... yes

checking complex.h usability... yes

checking complex.h presence... yes

checking for complex.h... yes

checking for sqrt in -lm... yes

You are using OS X Lion (or later).

You are strongly advised to install Apple's latest Xcode

unless you already have it. You can install this using

the App Store. Also, make sure you install Xcode's

Command Line Tools -- see Sage's README.txt.

-----------------------------------------------------------------------------

Checking whether SageMath should install SPKG gcc...

checking for C compiler vendor... clang

checking whether g++ supports C++11 features with -std=gnu++11... yes

checking for gcc option to accept ISO C99... none needed

checking if gcc accepts -dumpversion option... yes

checking gcc version... 4.2.1

checking if g++ accepts -dumpversion option... yes

checking g++ version... 4.2.1

checking whether C++ compiler accepts "-march=native"... yes

checking for OpenMP flag of C compiler... unknown

checking for OpenMP flag of C++ compiler... unknown

configure: will use system package and not install SPKG gcc

checking for the location of crti.o... 

-----------------------------------------------------------------------------

..........

..........

-----------------------------------------------------------------------------

Checking whether SageMath should install SPKG arb...

checking installing flint? ... checking arb.h usability... yes

checking arb.h presence... yes

checking for arb.h... yes

checking for acb_mat_eig_simple in -larb... yes

configure: will use system package and not install SPKG arb

checking for Fortran flag needed to accept free-form source... unknown

configure: Your Fortran compiler does not accept free-format source code

configure: which means the compiler is either seriously broken, or

configure: is too old to build Sage.

-----------------------------------------------------------------------------

Checking whether SageMath should install SPKG gfortran...

checking the version of gfortran... 11.1.0

configure: no suitable system package found for SPKG gfortran

-----------------------------------------------------------------------------

[...]

-----------------------------------------------------------------------------

Checking whether SageMath should install SPKG readline...

configure: readline has already been installed by SageMath

-----------------------------------------------------------------------------

Checking whether SageMath should install SPKG fplll...

configure: fplll has already been installed by SageMath

-----------------------------------------------------------------------------

Checking whether SageMath should install SPKG freetype...

checking whether any of gcc libpng is installed as or will be installed as SPKG... no

checking for FREETYPE... yes

configure: will use system package and not install SPKG freetype

-----------------------------------------------------------------------------

[...]

-----------------------------------------------------------------------------

Checking whether SageMath should install SPKG python3...

checking whether any of bzip2 xz libffi is installed as or will be installed as SPKG... no

checking for python3 >= 3.7.0, < 3.10.0 with modules sqlite3, ctypes, math, hashlib, crypt, readline, socket, zlib, distutils.core... 

checking ... whether /usr/local/bin/python3 is good... Traceback (most recent call last):

  File "<string>", line 1, in <module>

ImportError: dlopen(/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/readline.cpython-39-darwin.so, 2): Library not loaded: /usr/local/opt/readline/lib/libreadline.8.dylib

  Referenced from: /usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/readline.cpython-39-darwin.so

  Reason: image not found

no, the version is in the supported range but cannot import one of the required modules: sqlite3, ctypes, math, hashlib, crypt, readline, socket, zlib, distutils.core

checking ... whether /usr/bin/python3 is good... yes

checking for python3 >= 3.7.0, < 3.10.0 with modules sqlite3, ctypes, math, hashlib, crypt, readline, socket, zlib, distutils.core... /usr/bin/python3

configure: will use system package and not install SPKG python3

checking whether /usr/bin/python3 is configured to build multiarch extensions... no

checking whether "-march=native" works with the C/C++ compilers configured for building extensions for /usr/bin/python3... yes

[...]

-----------------------------------------------------------------------------

configure:

    notice: the following SPKGs did not find equivalent system packages:

        eclib fflas_ffpack gfortran giac gsl iml lcalc openblas r suitesparse sympow   _recommended cbc cmake coxeter3 gp2c igraph libsemigroups lrslib ninja_build pandoc pari_elldata pari_galpol pari_nftables pari_seadata perl_cpan_polymake_prereq perl_term_readline_gnu tox

        

checking for the package system in use... homebrew

configure:

    hint: installing the following system packages, if not

    already present, is recommended and may avoid having to

    build them (though some may have to be built anyway):

      $ brew install gcc gsl openblas r suite-sparse

    

    Homebrew can issue suggestions regarding keg-only packages.

    The following command is to automatically apply these suggestions

    for packages relevant for Sage to make them available for the build.

    Run it once to apply the suggestions for the current session.

    Add it to your shell profile to apply them for all future sessions.

    

      $ source /Users/pascalweil/GitBox/sage/.homebrew-build-env

configure:

    hint: installing the following system packages, if not

    already present, may provide additional optional features:

      $ brew install ffmpeg imagemagick texinfo cbc cmake igraph ninja pandoc tox

    

    Homebrew can issue suggestions regarding keg-only packages.

    The following command is to automatically apply these suggestions

    for packages relevant for Sage to make them available for the build.

    Run it once to apply the suggestions for the current session.

    Add it to your shell profile to apply them for all future sessions.

    

      $ source /Users/pascalweil/GitBox/sage/.homebrew-build-env

configure:

    hint: After installation, re-run configure using:

      $ ./config.status --recheck && ./config.status

               

As we can see above, `gfortran` was not found on the system, but gcc was installed by brew. We double check that gcc is installed:

$ brew install gcc

Updating Homebrew...

==> Auto-updated Homebrew!

Updated 3 taps (homebrew/core, homebrew/cask and caskroom/cask).

==> New Formulae

osinfo-db                                          osinfo-db-tools

==> Updated Formulae

Updated 5 formulae.

==> Updated Casks

Updated 6 casks.

Warning: gcc 11.1.0_1 is already installed and up-to-date.

To reinstall 11.1.0_1, run:

  brew reinstall gcc

Questions:

- How should he correctly install gfortran with homebrew?

- Why does source .homebrew-build-env does not fix the PATH of the correct python3 : /usr/bin/python3 instead of /usr/local/bin/python3 ?

Thank you again,

Sébastien

[Dima Pasechnik]

On Thu, Jun 3, 2021 at 2:07 PM Sébastien Labbé <sla...@gmail.com> wrote:
>
>
>
>> did he do
>>
>> source .homebrew-build-env
>>
>> ?
>>
>> This must be done in any Homebrew-based Sage build or use session.
>
>
> Good point. We did it, but maybe not before the second ./configure. So, we redid from scratch:
>
>
> $ source .homebrew-build-env
> source .homebrew-build-env
>
> Doing this does not change the python3 to /usr/bin/python3
>
> $ which python3
> /usr/local/bin/python3
> $ python3
> Python 3.9.5 (default, May 4 2021, 03:36:27)
> [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import readline
> $Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: dlopen(/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/readline.cpython-39-darwin.so, 2): Library not loaded: /usr/local/opt/readline/lib/libreadline.8.dylib
> Referenced from: /usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/readline.cpython-39-darwin.so
> Reason: image not found

works for me:

% python3

Python 3.9.5 (default, May 4 2021, 03:36:27)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
>>>

% brew install readline
Warning: readline 8.1 is already installed and up-to-date.
To reinstall 8.1, run:
brew reinstall readline

% ls -l /usr/local/opt/readline/lib/libreadline.8.dylib
lrwxr-xr-x 1 dima staff 21 18 Nov 2020
/usr/local/opt/readline/lib/libreadline.8.dylib ->
libreadline.8.1.dylib


I think your Homebrew install is shot. Please fix it, it has nothing
to do with Sage at this point.
Maybe,

brew reinstall readline
brew reinstall python3

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/9d1dc149-5ac1-4c41-8d8c-b314aec7b687n%40googlegroups.com.

[Sébastien Labbé]

I think your Homebrew install is shot. Please fix it, it has nothing
to do with Sage at this point.
Maybe,

brew reinstall readline
brew reinstall python3

ok thank you, we will check this out.

Sébastien