Newest Xcode is causing doctest failures

[John H Palmieri]

On an OS X machine, I just did a minor upgrade on the operating system (to 10.13.4) and also Xcode and the command line tools. Now I am getting doctest failures:

sage -t --long src/sage/rings/polynomial/multi_polynomial_ideal.py  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/groebner_fan.py  # 70 doctests failed
sage -t --long src/doc/fr/tutorial/tour_advanced.rst  # 2 doctests failed
sage -t --long src/doc/en/tutorial/tour_advanced.rst  # 2 doctests failed
sage -t --long src/doc/ja/tutorial/tour_advanced.rst  # 2 doctests failed
sage -t --long src/doc/pt/tutorial/tour_advanced.rst  # 2 doctests failed
sage -t --long src/doc/de/tutorial/tour_advanced.rst  # 2 doctests failed
sage -t --long src/doc/ru/tutorial/tour_advanced.rst  # 2 doctests failed

Here is a typical example:

sage -t --long src/sage/rings/polynomial/multi_polynomial_ideal.py
**********************************************************************
File "src/sage/rings/polynomial/multi_polynomial_ideal.py", line 3304, in sage.rings.polynomial.multi_polynomial_ideal.NCPolynomialIdeal.groebner_fan
Failed example:
    g.reduced_groebner_bases()
Exception raised:
    Traceback (most recent call last):
      File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/VANILLA/sage-8.2.rc0/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 551, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/VANILLA/sage-8.2.rc0/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 961, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.rings.polynomial.multi_polynomial_ideal.NCPolynomialIdeal.groebner_fan[3]>", line 1, in <module>
        g.reduced_groebner_bases()
      File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/VANILLA/sage-8.2.rc0/local/lib/python2.7/site-packages/sage/rings/polynomial/groebner_fan.py", line 1064, in reduced_groebner_bases
        X = [ReducedGroebnerBasis(self, [S(f) for f in G[i].split(',')], G[i]) for i in range(len(G))]
      File "sage/structure/parent.pyx", line 920, in sage.structure.parent.Parent.__call__ (build/cythonized/sage/structure/parent.c:9734)
        return mor._call_(x)
      File "sage/structure/coerce_maps.pyx", line 145, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (build/cythonized/sage/structure/coerce_maps.c:4555)
        raise
      File "sage/structure/coerce_maps.pyx", line 140, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (build/cythonized/sage/structure/coerce_maps.c:4417)
        return C._element_constructor(x)
      File "sage/rings/polynomial/multi_polynomial_libsingular.pyx", line 982, in sage.rings.polynomial.multi_polynomial_libsingular.MPolynomialRing_libsingular._element_constructor_ (build/cythonized/sage/rings/polynomial/multi_polynomial_libsingular.cpp:12001)
        raise TypeError("Could not find a mapping of the passed element to this ring.")
    TypeError: Could not find a mapping of the passed element to this ring.
**********************************************************************

Any suggestions?

--
John

[François Bissey]

Hum… I’d say a change in the host libc++. That’s annoying, I’d try rebuilding
brial and singular and then the bits of sage depending on them.
But we should get to the bottom of this, it may mean something is using something
it shouldn’t.

François

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

[Dima Pasechnik]

make distclean 

and

make ?

[François Bissey]

We definitely want to know if that works!

[Dima Pasechnik]

---------- Forwarded message ----------
From: John H Palmieri <jhpalm...@gmail.com>
Date: Sun, Apr 1, 2018 at 5:18 AM
Subject: Private message regarding: Newest Xcode is causing doctest failures
To: dim...@gmail.com


Hi Dima,

My messages to sage-devel are getting blocked, I think. Two comments:

- when I do an incremental upgrade, I do not see these failures.
- when I do 'make distclean; make ptestlong', I see them (with
8.2.beta8 or later; I haven't checked earlier versions).

So the problem is in building some package which hasn't been updated
in a while (that is, one that doesn't get rebuilt when I pull the most
recent devel branch and then build).

John

[Dima Pasechnik]

John emailed to me, saying:

My messages to sage-devel are getting blocked, I think. Two comments:

- when I do an incremental upgrade, I do not see these failures.
- when I do 'make distclean; make ptestlong', I see them (with 8.2.beta8 or later; I haven't checked earlier versions).

So the problem is in building some package which hasn't been updated in a while (that is, one that doesn't get rebuilt when I pull the most recent devel branch and then build).

  John

[Volker Braun]

On Sunday, April 1, 2018 at 9:01:17 AM UTC+2, Dima Pasechnik wrote:

- when I do an incremental upgrade, I do not see these failures.
- when I do 'make distclean; make ptestlong', I see them (with
8.2.beta8 or later; I haven't checked earlier versions).

 Is it the other way round? Surely it should work with  "make distclean; make ptestlong"?

[John H Palmieri]

It is not the other way around: failures when building from scratch, no failures when upgrading.

--
John

 

[François Bissey]

I (painfully) checked on my old laptop and I get this stuff when
building from scratch too. So we have some kind of new mis-compilation
with clang.

François

[Volker Braun]

Aaaand Apple force-pushed the borked Xcode out as a practical April Fool's joke, so now everyone has it. Including our buildbot.

[François Bissey]

That’s bad. Blocker class bad just at the time of rc.
The obvious suspect is singular.

François

> On 1/04/2018, at 16:17, John H Palmieri <jhpalm...@gmail.com> wrote:
>
> Hi François,

>
> My messages to sage-devel are getting blocked, I think. Two comments:
>

> - when I do an incremental upgrade, I do not see these failures.
> - when I do 'make distclean; make ptestlong', I see them (with 8.2.beta8 or later; I haven't checked earlier versions).
>

> So the problem is in building some package which hasn't been updated in a while (that is, one that doesn't get rebuilt when I pull the most recent devel branch and then build).
>
> John
>
>
>

[François Bissey]

> On 2/04/2018, at 16:27, John H Palmieri <jhpalm...@gmail.com> wrote:

> You made a guess in email that it was Singular. Suggestions for how to test this?
>

Based on the libraries apparently involved. But looking at the chain of
calls, i would be more inlined to suspect something in sage itself. Either
in the glue to singular or in the coercion framework.
But that’s speculative, I have no real ideas. Does anyone know if that breaks
8.1 as well? If not, we may be able to bisect something.

François

[John H Palmieri]

I just tried with 8.1, and all tests pass. That builds with gcc, not clang, so it's a big difference.

--
John
 

[François Bissey]

Forgot about that! So the earliest we can test is 8.2.beta5.

François

[John H Palmieri]

8.2.beta5 doesn't build without #24704 ("ppl build fails with clang-5.0"), in case that provides any clues. WIth #24704, I get the same doctest errors.

--
John

[Dima Pasechnik]

Oh, clang 5.0 is tricky. I did have trouble with it on #22679 (porting to FreeBSD), this is by the way how the patch from #24704 arose.

 

--
John

[Dima Pasechnik]

On the latest OSX/Xcode, on the 8.2.rc1 branch with SAGE_CHECK=yes, I get an error while testing cython:

[cython-0.28.1] ERROR: runTest (__main__.CythonRunTestCase)
[cython-0.28.1] compiling (cpp) and running libcpp_algo
[cython-0.28.1] ----------------------------------------------------------------------
[cython-0.28.1] Traceback (most recent call last):
[cython-0.28.1]   File "runtests.py", line 1114, in run
[cython-0.28.1]     ext_so_path = self.runCompileTest()
[cython-0.28.1]   File "runtests.py", line 803, in runCompileTest
[cython-0.28.1]     self.test_directory, self.expect_errors, self.expect_warnings, self.annotate)
[cython-0.28.1]   File "runtests.py", line 1050, in compile
[cython-0.28.1]     so_path = self.run_distutils(test_directory, module, workdir, incdir)
[cython-0.28.1]   File "runtests.py", line 973, in run_distutils
[cython-0.28.1]     build_extension.run()
[cython-0.28.1]   File "/Users/dimpase/Sage/sagetrac-mirror/local/lib/python2.7/distutils/command/build_ext.py", line 340, in run
[cython-0.28.1]     self.build_extensions()
[cython-0.28.1]   File "/Users/dimpase/Sage/sagetrac-mirror/local/lib/python2.7/distutils/command/build_ext.py", line 449, in build_extensions
[cython-0.28.1]     self.build_extension(ext)
[cython-0.28.1]   File "runtests.py", line 486, in build_extension
[cython-0.28.1]     _build_ext.build_extension(self, ext)
[cython-0.28.1]   File "/Users/dimpase/Sage/sagetrac-mirror/local/lib/python2.7/distutils/command/build_ext.py", line 499, in build_extension
[cython-0.28.1]     depends=ext.depends)
[cython-0.28.1]   File "/Users/dimpase/Sage/sagetrac-mirror/local/lib/python2.7/site-packages/numpy/distutils/ccompiler.py", line 343, in CCompiler_compile
[cython-0.28.1]     single_compile(o)
[cython-0.28.1]   File "/Users/dimpase/Sage/sagetrac-mirror/local/lib/python2.7/site-packages/numpy/distutils/ccompiler.py", line 303, in single_compile
[cython-0.28.1]     self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
[cython-0.28.1]   File "/Users/dimpase/Sage/sagetrac-mirror/local/lib/python2.7/site-packages/numpy/distutils/unixccompiler.py", line 61, in UnixCCompiler__compile
[cython-0.28.1]     raise CompileError(msg)
[cython-0.28.1] CompileError: Command "cc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/dimpase/Sage/sagetrac-mirror/local/var/tmp/sage/build/cython-0.28.1/src/tests/run -I/Users/dimpase/Sage/sagetrac-mirror/local/include/python2.7 -c libcpp_algo.cpp -o /Users/dimpase/Sage/sagetrac-mirror/local/var/tmp/sage/build/cython-0.28.1/src/TEST_TMP/run/cpp/libcpp_algo/libcpp_algo.o -DCYTHON_REFNANNY=1" failed with exit status 1
[cython-0.28.1]
[cython-0.28.1] ----------------------------------------------------------------------
[cython-0.28.1] Ran 10737 tests in 3506.735s

On Wednesday, April 4, 2018 at 5:28:25 AM UTC+2, John H Palmieri wrote:

[Dima Pasechnik]

I would appreciate tips on how to at least get a detailed log here.

[Volker Braun]

I've opened a blocker ticket #25113 

[Dima Pasechnik]

Hi John,

haven't you forgotten the update of command-line tools?

could you post the outputs of your softwareupdate and sysctl -n machdep.cpu.brand_string

(see example on https://trac.sagemath.org/ticket/25113)

I cannot reproduce these errors.

Dima