Numpy crashing on sage 9.2.RC0

[Zachary Scherr]

Hi All,

   I was doing some bug hunting and noticed that scipy's pytests crashed pretty early.  I tracked down exactly what was causing the problem and noticed it can be reproduced by the following sequence of commands:

sage: import numpy as np

sage: x = np.random.randn(1000000,2)

sage: y = np.random.randn(2,2)

sage: x.dot(y)

produces:

------------------------------------------------------------------------

0   signals.cpython-38-darwin.so        0x00000001088bc62a print_backtrace + 58

1   signals.cpython-38-darwin.so        0x00000001088c0277 sigdie + 39

2   signals.cpython-38-darwin.so        0x00000001088c0212 sigdie_for_sig + 290

3   libsystem_platform.dylib            0x00007fff6c88e5fd _sigtramp + 29

------------------------------------------------------------------------

Unhandled SIGBUS: A bus error occurred.

This probably occurred because a *compiled* module has a bug

in it and is not properly wrapped with sig_on(), sig_off().

Python will now terminate.

------------------------------------------------------------------------

/Users/zscherr/sage/src/bin/sage-python: line 2: 78812 Bus error: 10           sage -python "$@"

I haven't tested on any other platforms, but for what it's worth I have Catalina 10.15.7 with Xcode 12.

Thanks,

Zach

[Dima Pasechnik]

On Sun, Oct 11, 2020 at 8:14 PM Zachary Scherr <zsc...@gmail.com> wrote:
>
> Hi All,
>
> I was doing some bug hunting and noticed that scipy's pytests crashed pretty early. I tracked down exactly what was causing the problem and noticed it can be reproduced by the following sequence of commands:
>
> sage: import numpy as np
> sage: x = np.random.randn(1000000,2)
> sage: y = np.random.randn(2,2)
> sage: x.dot(y)

please test this in "plain" Sage python, i.e. does

./sage --python

>>> import numpy as np
>>> x = np.random.randn(1000000,2)
>>> y = np.random.randn(2,2)
>>> x.dot(y)

reproduce the crash?

>
> produces:
>
> ------------------------------------------------------------------------
> 0 signals.cpython-38-darwin.so 0x00000001088bc62a print_backtrace + 58
> 1 signals.cpython-38-darwin.so 0x00000001088c0277 sigdie + 39
> 2 signals.cpython-38-darwin.so 0x00000001088c0212 sigdie_for_sig + 290
> 3 libsystem_platform.dylib 0x00007fff6c88e5fd _sigtramp + 29
> ------------------------------------------------------------------------
> Unhandled SIGBUS: A bus error occurred.
> This probably occurred because a *compiled* module has a bug
> in it and is not properly wrapped with sig_on(), sig_off().
> Python will now terminate.
> ------------------------------------------------------------------------
> /Users/zscherr/sage/src/bin/sage-python: line 2: 78812 Bus error: 10 sage -python "$@"
>
> I haven't tested on any other platforms, but for what it's worth I have Catalina 10.15.7 with Xcode 12.
>
> Thanks,
> Zach
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/28f58136-d3c7-40b7-9a46-36fc347acc2en%40googlegroups.com.

[Zachary Scherr]

That also crashes with: 86622 segmentation fault  ./sage --python

I tested this both with building sage against homebrew python 3.8.6 and against sage's default python 3.8.5 and the same error occurred in both cases.  I don't have this problem using my system-wide numpy so it's possible that because sage numpy is built from source that maybe one of my homebrew libraries is causing issues.  I found https://github.com/numpy/numpy/issues/11551 which could be relevant and might indicate that it's not a sage problem.

I should also add that if I lower the dimension of x to 100000 then it works without problem.

[Dima Pasechnik]

On Mon, Oct 12, 2020 at 1:00 AM Zachary Scherr <zsc...@gmail.com> wrote:
>
> That also crashes with: 86622 segmentation fault ./sage --python
>
> I tested this both with building sage against homebrew python 3.8.6 and against sage's default python 3.8.5 and the same error occurred in both cases. I don't have this problem using my system-wide numpy so it's possible that because sage numpy is built from source that maybe one of my homebrew libraries is causing issues. I found https://github.com/numpy/numpy/issues/11551 which could be relevant and might indicate that it's not a sage problem.

If you are talking about Homebrew's numpy, it should be more or less
identical with Sage's numpy (assuming you use Homebrew's openblas in
Sage)
cf. https://github.com/Homebrew/homebrew-core/blob/master/Formula/numpy.rb

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/a67ca4b5-526b-40d7-a476-520d49890207n%40googlegroups.com.

[Zachary Scherr]

Hi Dima,

   My system-wide numpy is installed via pip.  I was just remarking that I didn't build my system-wide numpy from source so it's not a fair comparison.  It's possible that the issues aren't sage related but are related to building sage's numpy against some homebrew libraries.

[Zachary Scherr]

In case anyone is curious, I've tracked the problem down to homebrew's openBLAS and so it has nothing to do with sage.  If I build numpy from source through homebrew then I get exactly the same error using my system python3.  I'm wondering if maybe this has something to do with the fact that openBLAS is bottled with a different processor type than the one that is on my system.  In any case I'll ask over there and see if I can get some help.

[Isuru Fernando]

I looked at the openblas formula in homebrew and they are not passing the TARGET option. When using DYNAMIC_ARCH=1, a target should be specified.

Isuru

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/7ba7ef07-8460-4180-8670-397e1fb6df93n%40googlegroups.com.

[Zachary Scherr]

Aha.  In fact, if I do

brew install openblas --build-from-source

then it picks up the TARGET automatically and I don't have any problems with numpy whatsoever.  I'll still ask over there, but I'm guessing this is the problem with trying to use bottled versions of the libraries.

[Dima Pasechnik]

On Mon, Oct 12, 2020 at 6:17 PM Isuru Fernando <isu...@gmail.com> wrote:
>
> I looked at the openblas formula in homebrew and they are not passing the TARGET option. When using DYNAMIC_ARCH=1, a target should be specified.

care to open an issue with them on https://github.com/Homebrew/homebrew-core ?

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CA%2B01voNTGybhpxULUXSuBxFFc2fAXkwyer%3DQ0pSAPpP3Am8uDQ%40mail.gmail.com.

[Zachary Scherr]

I posted about it on their discussion page, but I'm not sure there will be anything they can do.  They build their libraries on several different versions of MacOS, but I'm not sure they build with several different TARGETS.  I can try to pursue this, but the easy fix for now is that if people want to install openblas via homebrew they should install from source.

[Isuru Fernando]

You don't need different TARGETs. `DYNAMIC_ARCH=1` means that it will detect at runtime what the CPU is and will use the kernel matching the CPU at runtime. `TARGET` gives the oldest CPU that this code will run on so that the common code is compiled to target this.

Isuru

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/e0f2c485-cb29-4db3-85bc-3c2632835c90n%40googlegroups.com.

[Zachary Scherr]

Ah, thank you.  I'll post the issue and see if I can get homebrew to resolve it!

[Zachary Scherr]

Hi Isuru,

   I've been trying to fix homebrew's openblas, and I'm running into an issue that I was hoping you might know something about.  If I build openBLAS with TARGET=HASWELL then I don't have any issues with numpy.  However, if I build it with TARGET=NEHALEM, which is what they want for homebrew, then I still run into the same numpy error.  I see that Conda-forge uses TARGET=PRESCOTT, but if I try that I also run into the same numpy problem.  Do you have any insight into why this might be the case?

Thanks,

Zach

[Isuru Fernando]

Hi Zachary,

You might be hitting https://github.com/xianyi/OpenBLAS/issues/2728 which is fixed in https://github.com/xianyi/OpenBLAS/pull/2729 but not in 0.3.10.

Isuru

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/37a70675-95bb-419d-ab3c-4baa10eb632an%40googlegroups.com.

[Zachary Scherr]

Hi Isuru,

   Thank you so much.  I missed your message yesterday, but I also asked on their forum and they made the same suggestion and said it would be fixed in 3.11. I still asked the homebrew people to add a TARGET to their builds so hopefully that will get straightened out as well.