Threading and cypari

[David Roe]

Upgrading Sage from 9.2 to 9.3 breaks the LMFDB (segfaults on startup).  We believe that the problem arises from an interaction between Pari and threads, and were able to duplicate it in Sage with the following:

from concurrent.futures import ThreadPoolExecutor
sage: with ThreadPoolExecutor() as e:
....:     j = e.submit(is_squarefree, 15)
....: j.result()

...

~/sage/sage-9.4.beta0/local/lib/python3.9/site-packages/sage/rings/integer.pyx in sage.rings.integer.Integer.is_squarefree (build/cythonized/sage/rings/integer.c:37634)()
   6007             False
   6008         """
-> 6009         return self.__pari__().issquarefree()
   6010
   6011     cpdef __pari__(self):

~/sage/sage-9.4.beta0/local/lib/python3.9/site-packages/sage/rings/integer.pyx in sage.rings.integer.Integer.__pari__ (build/cythonized/sage/rings/integer.c:37754)()
   6029
   6030         """
-> 6031         return new_gen_from_mpz_t(self.value)
   6032
   6033     def _interface_init_(self, I=None):

~/sage/sage-9.4.beta0/local/lib/python3.9/site-packages/sage/libs/pari/convert_gmp.pyx in sage.libs.pari.convert_gmp.new_gen_from_mpz_t (build/cythonized/sage/libs/pari/convert_gmp.c:1298)()
     50         True
     51     """
---> 52     sig_on()
     53     return new_gen(_new_GEN_from_mpz_t(value))
     54

SignalError: Segmentation fault

Searching for cypari issues, #94 suggests that there may be a problem with "relatively recent decision of maintaining pointers on the PARI stack."

At this point, we're a little lost.  We understand that sage overall is not threadsafe.  But we hope that cypari (and the parts of Sage that we use in the LMFDB) can be, since it was in 9.2.  Any tips on where to look next (or fixes to suggest!) would be welcome.

David

[Vincent Delecroix]

Thread support in cypari2 is definitely a reasonable requirement.
I don't think this is tested.

It would be nice to

1) reproduce the segfault outside of sage only using python and
cypari2

2) open an issue on cypari2 development website at
https://github.com/sagemath/cypari2.

Best
Vincent

Le 04/06/2021 à 20:14, David Roe a écrit :
> Upgrading Sage from 9.2 to 9.3 breaks the LMFDB <http://www.lmfdb.org>

> <https://github.com/sagemath/cypari2/issues/94> suggests that there may be

[David Roe]

On Fri, Jun 4, 2021 at 3:24 PM Vincent Delecroix <20100.d...@gmail.com> wrote:

Thread support in cypari2 is definitely a reasonable requirement.
I don't think this is tested.

It would be nice to

1) reproduce the segfault outside of sage only using python and
    cypari2

2) open an issue on cypari2 development website at
    https://github.com/sagemath/cypari2.

Thanks.  I've made https://github.com/sagemath/cypari2/issues/107.

David

[David Roe]

This bug is continuing to seriously impact the LMFDB.  We haven't had a functional debugger for 3 years.  It's preventing us from serving large downloads, limiting users' ability to download large search results.  Today, it's interfering in efforts to add the Stein-Watkins and BHKSSW elliptic curve databases to the LMFDB, since the files they contain are too large to host with a non-threaded 30 second timeout.

Help resolving the bug or suggesting workarounds is welcome.

David

[Vincent Delecroix]

Hi David,

I think that a reasonable move is to revert the use of pointers on the
PARI stack. It created a lot of trouble with memory management.

Best
Vincent

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/sage-devel/CAChs6_m6a1akrUuX2s7dObcDJQ-jpH3WMZQauD8b3s00Xg0%3Dtw%40mail.gmail.com.

[David Roe]

Thanks for the suggestion.  Is there a PR when cypari2 started using pointers on the PARI stack?  I'm just trying to figure out how to make such a change.

David

To view this discussion visit https://groups.google.com/d/msgid/sage-devel/CAGEwAAk_sjON0uYO3gAmt3VsbaY69qMvFqVZnm_PtGPuEgKD-g%40mail.gmail.com.

[Vincent Delecroix]

This is commit 551a5277456a3170dd690773e22f319cc1ad795a from
https://github.com/sagemath/cypari2/pull/59.

> To view this discussion visit https://groups.google.com/d/msgid/sage-devel/CAChs6_mLznvwxKea5RRAQVNWMiZREyGd3UU3TR2ybVTv_-u-fw%40mail.gmail.com.

[Marc Culler]

The problematic bullet point in that PR is:

* Don't copy everything from the PARI stack. During the 2018 PARI/GP atelier, I talked to Karim Belabas about a strategy to achieve this. The basic idea is that we need to have a linked list containing the Gens which live on the PARI stack. When the time comes that we need to garbage collect the PARI stack, we traverse that list and copy the GEN objects from the PARI stack to the PARI heap.

That has been reverted in CyPari, which you could look at as a guide to what needs to be done.

The linked list design is not implemented correctly, and it is not clear if there is a reasonable way to fix it.  The Cypari2#112 issue contains pretty concrete examples of how it prevents Gen objects from being deallocated.

- Marc