Python PEP 703: no more Interpreter lock

[Thierry Dumont]

With PEP 703, Cpython will move (slowly) to a non GIL implementation,
allowing true multi-threading. It will take time (-> 3.15) and will be
optional at the beginning. See:
https://peps.python.org/pep-0703/
Which consequences for the Python part of Sage ? Are there many parts
which could be multi-threaded ?

t.d.

[Dima Pasechnik]

there are many parts, certainly - and it's a huge project, involving doing novel research in algorithms, to use this.

One of the difficulties is that without the help from the language, parallelism is a hard work to make good use of.

E.g. Pari/GP multithreading can't be described as huge success, as in some cases it slows code down, rather than speeding it up.

GAP's multi-year attempts at HPC GAP cannot be described as success. It's stalled at alpha-stage.

Dima

t.d.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/7ea5fa49-2438-4f0b-b350-cbbb0852a6bb%40math.univ-lyon1.fr.

[Nils Bruin]

For each library used, we'll need to assess if it can operate in a thread-safe way and if it currently is used in a thread-safe way. 

There are some parts that will be fundamentally very hard to make multi-threaded: maxima through the ECL binary interface, for instance, will probably fundamentally be problematic. The design of maxima is rather fundamentally not multi-threaded, but even if you lock maxima to only run in one thread, there will be problems to let it operate in a multi-threaded process: ECL itself does support multi-threaded operation, but requires signal handling control (*particularly* when it's multi-threaded, because signals are used to coordinate critical sections garbage collection), so currently upon entry of ECL, it swaps the signal handlers and upon exit swaps them back. As far as I know, signal handlers are on process level; not thread-local. So I don't think we'll be able to allow multiple threads safely whenever ECL is entered. Perhaps the ECL people are willing to consider developing other designs that are more appropriate for supporting ECL as a library (it is "embedded common lisp" after all) in a multi-threaded environment or perhaps we can integrate ECL's signal handling requirements more closely with those of sagemath so that switching the handlers is no longer required.

Note that in sage it is very hard to predict which libraries get invoked when running high-level code, because operations may get dispatched depending on implementation details of objects (which, for instance, for finite fields, may depend on the field size!)