OS X Big Sur progress report

[John H Palmieri]

On the latest "release candidate" for OS X Big Sur + corresponding Xcode and command line tools, Sage now builds for me. With various homebrew packages installed, I can run "./configure" and "make", and the build succeeds. There are a number of doctest failures with the message

    ld: warning: dylib (/Users/palmieri/Sage/sage-9.3.beta0/local/lib/libec.dylib) was built for newer macOS version (11.0) than being linked (10.15)

Once I suppress those, I only get a handful of doctest errors, at least some of which are because of Python 3.9 issues. So there is significant progress here, compared to a few weeks ago.

--

John

[Andrew]

Thanks for this John. Did you do anything special to compile sage with big sur? I just tried:

make distclean && source .homebrew-build-env && ./configure --with-system-gfortran=no --with-system-openblas=no --with-system-gmp=no && make

but the build fails with:

Error building Sage.

The following package(s) may have failed to build (not necessarily

during this run of 'make all-start'):

* package:         givaro-4.1.1

  last build time: 16 Nov 14:19

  log file:        /usr/local/src/sage/logs/pkgs/givaro-4.1.1.log

  build directory: /usr/local/src/sage/local/var/tmp/sage/build/givaro-4.1.1

This is probably not related to 9.3.beta1 as sage has been failing to build, with givaro giving an error, on this laptop since 9.1. If anyone has any ideas as to how I might fix this I'd be grateful. I have tried deleting all of the homebrew packages and reinstalling. completely removing xcode and reinstalling, playing with the compilation switches, ... 

In case anyone has time to look at this, here is my install.log.gz file . Here is my list of brew packages:

adns eigen gmp libdvdcss libvpx openexr r tkdiff

alluxio ffmpeg gnu-getopt libevent libyaml openjpeg rav1e unbound

aom fish      gnu-sed libffi little-cms2 openssl@1.1 readline unrar

arb        flac gnupg libgcrypt lua opus rsync vtk

aspell flint gnutls libgpg-error lz4 p11-kit rtmpdump webkit2png

autoconf fontconfig gobject-introspection libheif lzo p7zip rubberband webp

automake freetype gpatch libidn2 macvim pandoc ruby wget

bash frei0r graphite2 libksba make pari sass x264

bash-completion fribidi gsl liblqr metis pcre sdl2 x265

bdw-gc gawk guile libmpc mpfi pcre2 shared-mime-info xmlto

boost gcc harfbuzz libogg mpfr perl sip xvid

cairo gd hdf5 libomp netcdf pinentry snappy xxhash

ceres-solver gdbm icu4c libpng nettle pixman speex xz

cimg gettext ilmbase libsamplerate ninja pkg-config sqlite yarn

cmake gflags imagemagick libsndfile node poppler srt yasm

coreutils ghostscript isl libsoxr npth popt suite-sparse zeromq

cscope giflib jpeg libtasn1 nspr ppl swig zlib

dart git lame libtiff nss protobuf szip zstd

dav1d git-extras leptonica libtool ntl pyqt tbb

djvu2pdf git-lfs libass libunistring numpy python@3.8 tesseract

djvulibre glib libassuan libusb openblas python@3.9 texinfo

docbook glog libbluray libvidstab opencore-amr qpdf the_silver_searcher

docbook-xsl glpk libde265 libvorbis opencv qt theora

font-firacode-nerd-font                             gimp                                                gitup                                               pdf-images

Andrew

[Dima Pasechnik]

On Mon, Nov 16, 2020 at 5:23 AM Andrew <andrew...@gmail.com> wrote:
>
> Thanks for this John. Did you do anything special to compile sage with big sur? I just tried:
>
> make distclean && source .homebrew-build-env && ./configure --with-system-gfortran=no --with-system-openblas=no --with-system-gmp=no && make
>

this forces Sage to build mpir from source (mpir is basically
unmaintained, as we have a ticket to replace the default MP package on
Sage with
GMP, but it's not merged yet), as well as many more packages in Homebrew.

With homebrew present, there is always a chance there will be a
conflict with Sage libraries you build and the ones
in Homebrew.

Besides, platform-specific tools such as compilers are not guaranteed
to work correctly with OSs they were not tested on, so it's
naive to expect that e.g. gfortran 9.2 will work on macOS 11.

Any reasons for you not to follow the recommendation to install
Homebrew packages we recommend?
(see the output of ./configure)

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/8c646c22-8ace-4dcc-9358-677d80757fc5n%40googlegroups.com.

[John H Palmieri]

In my experience, homebrew's "ntl" package can lead to problems when building Sage, so if possible, try deleting it. I have these packages:

% brew list

autoconf    ghc        jansson        mpc        ppl
automake    ghostscript    jpeg        mpfi        python@3.8
bdw-gc        git        libatomic_ops    mpfr        python@3.9
bison        glib        libevent    mpir        r
boost        glpk        libffi        nauty        re2c
bzip2        gmp        libidn2        ncurses        readline
cmake        gnutls        libmpc        nettle        sphinx-doc
curl        gpatch        libmpdclient    ninja        sqlite
doxygen        gsl        libpng        openblas    suite-sparse
fontconfig    guile        libtasn1    openexr        tbb
freetype    haskell-stack    libtiff        openssl@1.1    texinfo
gcc        icu4c        libtool        p11-kit        unbound
gcc@9        ilmbase        libunistring    pandoc        webp
gd        intltool    libxml2        pcre        xz
gdbm        isl        meson        pcre2        yasm
gettext        ispell        metis        pkg-config    zeromq

Then I really just did "source .homebrew-build-env", "./configure", and "make". I have to admit this is with beta releases of Xcode, but I think the previous beta release should match the official version. (I've done this with beta releases of Xcode 12.2 + corresponding command line tools, and also now with a beta release of Xcode 12.3 + command line tools.) I haven't actually tried with the officially released Xcode 12.2 on Big Sur, but I don't think that would change anything for me.

--

John

[Dima Pasechnik]

If you have a random subset of partially outdated Homebrew packages,
spiced up with manually installed random crap in /usrr/local/, there
is no chance of getting things right on Homebrew.

On Mon, Nov 16, 2020 at 6:19 PM John H Palmieri <jhpalm...@gmail.com> wrote:
>
> In my experience, homebrew's "ntl" package can lead to problems when building Sage, so if possible, try deleting it.

Why?! It all works, if you install the recommended by ./configure
list. And if not, open a ticket, stop spreading FUD here please.

We either adapt a systematic approach, with putting more and more
packages into Homebrew, or we will keep dragging our feet here.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/f1f2f472-7c26-48bd-81c7-cb68f2a8a03bn%40googlegroups.com.

[John H Palmieri]

On Monday, November 16, 2020 at 10:59:32 AM UTC-8 dim...@gmail.com wrote:

If you have a random subset of partially outdated Homebrew packages,
spiced up with manually installed random crap in /usrr/local/, there
is no chance of getting things right on Homebrew.

On Mon, Nov 16, 2020 at 6:19 PM John H Palmieri <jhpalm...@gmail.com> wrote:
>
> In my experience, homebrew's "ntl" package can lead to problems when building Sage, so if possible, try deleting it.

Why?! It all works, if you install the recommended by ./configure
list. And if not, open a ticket, stop spreading FUD here please.

The point is that "ntl" is *not* in the recommended list, and its presence, at least in the past, interfered with building Sage. Is Sage guaranteed to build if you install not only the recommended homebrew packages but lots of other things? Have we really tested that none of those other packages interfere with building Sage?

Two tickets are already open, and you've in fact commented on them: https://trac.sagemath.org/ticket/29339, https://trac.sagemath.org/ticket/30745. FUD? Please. Try to be a little polite.

[Dima Pasechnik]

On Mon, Nov 16, 2020 at 10:21 PM John H Palmieri <jhpalm...@gmail.com> wrote:
>
>
>
> On Monday, November 16, 2020 at 10:59:32 AM UTC-8 dim...@gmail.com wrote:
>>
>> If you have a random subset of partially outdated Homebrew packages,
>> spiced up with manually installed random crap in /usrr/local/, there
>> is no chance of getting things right on Homebrew.
>>
>> On Mon, Nov 16, 2020 at 6:19 PM John H Palmieri <jhpalm...@gmail.com> wrote:
>> >
>> > In my experience, homebrew's "ntl" package can lead to problems when building Sage, so if possible, try deleting it.
>>
>> Why?! It all works, if you install the recommended by ./configure
>> list. And if not, open a ticket, stop spreading FUD here please.
>
>
> The point is that "ntl" is *not* in the recommended list, and its presence, at least in the past, interfered with building Sage. Is Sage guaranteed to build if you install not only the recommended homebrew packages but lots of other things? Have we really tested that none of those other packages interfere with building Sage?
>
> Two tickets are already open, and you've in fact commented on them: https://trac.sagemath.org/ticket/29339, https://trac.sagemath.org/ticket/30745. FUD? Please. Try to be a little polite.

Apologies, I'm sorry. I forgot NTL is still "banned" (and lack of
sleep...). I've just put the branch up on
https://trac.sagemath.org/ticket/30745 to advertise them on Homebrew.
Please test (this works on macOS 10.15.7 for me with xcode 12).

>
>
>>
>>
>> We either adapt a systematic approach, with putting more and more
>> packages into Homebrew, or we will keep dragging our feet here.
>
>
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/193512b6-249c-4a38-be40-5a5843f51328n%40googlegroups.com.

[Andrew]

Dear Dima and John,

Thanks for your hints. I have tried deleting ntl, flint and arb but I am still getting stuck on givaro. I'll try removing some more brew packages and see if this helps and report back. The list brew packages recommended by configure is:

    pandoc ffmpeg imagemagick texinfo ppl

All of these already appear in my list of packages above so, yes, all of these are already installed.

Andrew

[Matthias Koeppe]

According to your log, you seem to have /usr/local/include/stdlib.h and /usr/local/include/math.h -- which seem to be breaking your giac build. This seems nonstandard.

[Andrew]

Thanks very much for this Matthias. This is almost certainly my problem -- and I have to confess to user error as this was a hack I put in at one point to get around problems with an xcode update. I have deleted a bunch of spurious links like these and am now recompiling. It has already gotten much further than before. Ih particular, givaro now compiles. So I am optimistic that it will finish now.

Thank you again -- and DIma and John -- for your help!

Andrew

[Andrew]

Just to confirm, I was able to compile 9.3.beta1 on macosx 10.0.1 running xcode 12.2 with homebrew.

The following doctests failed with `make ptestlong`:

sage -t --long --random-seed=0 src/sage/misc/sagedoc.py  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/tests/cmdline.py  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/structure/element.pyx  # 8 doctests failed

sage -t --long --random-seed=0 src/sage/misc/sageinspect.py  # 11 doctests failed

sage -t --long --random-seed=0 src/sage/functions/other.py  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/misc/cachefunc.pyx  # 6 doctests failed

sage -t --long --random-seed=0 src/sage/misc/cython.py  # 10 doctests failed

sage -t --long --random-seed=0 src/sage/graphs/connectivity.pyx  # 2 doctests failed

sage -t --long --random-seed=0 src/sage/repl/ipython_extension.py  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/parallel/decorate.py  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/ext/memory_allocator.pxd  # 3 doctests failed

sage -t --long --random-seed=0 src/sage/ext/memory_allocator.pyx  # 3 doctests failed

sage -t --long --random-seed=0 src/sage/arith/long.pxd  # 2 doctests failed

sage -t --long --random-seed=0 src/sage/cpython/cython_metaclass.pyx  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/misc/superseded.py  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/cpython/getattr.pyx  # 2 doctests failed

sage -t --long --random-seed=0 src/sage/structure/factory.pyx  # 2 doctests failed

sage -t --long --random-seed=0 src/sage/misc/lazy_attribute.pyx  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/misc/inherit_comparison.pyx  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/rings/integer_fake.pxd  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/misc/session.pyx  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/docs/instancedoc.pyx  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/repl/preparse.py  # 12 doctests failed

sage -t --long --random-seed=0 src/sage/cpython/wrapperdescr.pxd  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/misc/nested_class.pyx  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/cpython/string.pyx  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/misc/gperftools.py  # 1 doctest failed

sage -t --long --random-seed=0 src/sage/rings/tate_algebra_ideal.pyx  # 2 doctests failed

but the most amusing failure was at the end:

Doctests interrupted: 2791/4216 files tested

----------------------------------------------------------------------

Total time for all tests: 4482.7 seconds

    cpu time: 10717.5 seconds

    cumulative wall time: 13159.6 seconds

Traceback (most recent call last):

...

OSError: [Errno 24] Too many open files

make: *** [ptestlong] Error 1

That sage is compiling with the just released version of macosx and the latest xcode I think that this is a win.

[John H Palmieri]

You can probably eliminate some of the doctest errors with the following hack, which may not be the right way to proceed, but at least for me it allows doctesting to complete. Some of your other errors may be caused by using Python 3.9, and people are working on fixing those.

diff --git a/src/sage/doctest/parsing.py b/src/sage/doctest/parsing.py
index 4e1a6b32e4..bfb7f36940 100644
--- a/src/sage/doctest/parsing.py
+++ b/src/sage/doctest/parsing.py
@@ -40,6 +40,7 @@ optional_regex = re.compile(r'(arb216|arb218|py2|py3|long time|not implemented|n
 # which has not been patched, we need to ignore that message.
 # See :trac:`29317`.
 glpk_simplex_warning_regex = re.compile(r'(Long-step dual simplex will be used)')
+big_sur_warning_regex = re.compile(r'ld: warning: dylib.*was built for newer macOS version .* than being linked.*')
 find_sage_prompt = re.compile(r"^(\s*)sage: ", re.M)
 find_sage_continuation = re.compile(r"^(\s*)\.\.\.\.:", re.M)
 find_python_continuation = re.compile(r"^(\s*)\.\.\.([^\.])", re.M)
@@ -1087,6 +1088,7 @@ class SageOutputChecker(doctest.OutputChecker):
         """
         got = self.human_readable_escape_sequences(got)
         got = glpk_simplex_warning_regex.sub('', got)
+        got = big_sur_warning_regex.sub('', got)
         if isinstance(want, MarkedOutput):
             if want.random:
                 return True