New Flint Thread Pool

[Bill Hart]

Hi all,

Recently Dan Schultz added a new thread pool to Flint for the fast, threaded multivariate polynomial arithmetic in Flint.

I have now rolled this out to all of Flint. We've also extended the interface slightly.

Old Flint Threading Function

======================

We still have the following functions as before:

flint_set_num_threads(n) : set up a thread pool of n - 1 worker threads (in addition to the master thread) and set the maximum number of worker threads the master thread can get from the thread pool at any time to n - 1.

flint_get_num_threads() : returns the maximum number of threads, n, that are available to the current thread (including the current thread itself), i.e. the current thread can start an additional n - 1 worker threads.

New Flint Threading Functionality

==========================

It is also now possible to temporarily restrict the number of worker threads that can be started by the current thread, e.g. when calling a threaded function from within that thread. This is done using the function:

n = flint_set_num_workers(m) : set the number of worker threads the current thread can start (this cannot exceed the maximum number the thread is allowed to start). The old number of workers is returned by the function.

The number of workers can then be restored to the original value using:

flint_reset_num_workers(n)

In addition, when a thread from the thread pool is woken, there is now an additional parameter to the thread_pool_wake function which specifies the maximum number of additional workers that thread is allowed to start.

Which Functions in Flint are Threaded?

==============================

We've now used this thread pool arrangement to thread the following in Flint:

* multivariate multiplication, division and gcd

* polynomial taylor shift

* the Flint FFT

* matrix multiplication over Z/nZ

* polynomial factorisation over Z/pZ

* modular composition over Z/pZ

* generating powers of a polynomial over Z/pZ modulo another

* the quadratic sieve integer factoring function

Implementing Threaded Functions in Flint

================================ 

If you want to see a simple example of how to implement threaded Flint functions, refer to the first 100 or so lines of:

https://github.com/wbhart/flint2/blob/trunk/fmpz_poly/taylor_shift_multi_mod_threaded.c 

A more sophisticated (recursive) divide and conquer example is given in:

https://github.com/wbhart/flint2/blob/trunk/fmpz_poly/taylor_shift_divconquer.c 

There are more advanced implementations in:

https://github.com/wbhart/flint2/blob/trunk/nmod_poly/compose_mod_brent_kung_vec_preinv_threaded.c

and

https://github.com/wbhart/flint2/blob/trunk/nmod_poly_factor/factor_distinct_deg_threaded.c

which implement some kind of cooperation so that threads don't sit around idle waiting for others to finish

The qsieve module also makes sophisticated use of the threadpool and can be referred to as an example.

Using Threaded Flint

================

As for using threads in Flint, this is very easy. Only the function flint_set_num_threads(n) needs to be used. It sets the number of threads that Flint will use, and many threaded functions will automatically use them.

For example, polynomial multiplication over Z uses the FFT, which is now threaded. Simply calling flint_set_num_threads(4) before calling such a function in Flint will cause the FFT to use 4 threads, which will give some speedup.

Much more work is needed to thread additional functions in Flint.

New Flint Release?
===============

I am currently working on a new Flint release, which we hope to have available in a little over a month. The functionality above is currently in trunk, but not in an official release.

I will call for volunteers to help close some of the 80 odd remaining easy tickets in about two weeks, after I have closed all the really big tickets, the ones that only I can close and the really trivial ones. Volunteers would be really welcome at that point so that we can keep the release roughly on schedule for the end of March.

A Flint release has been held up for roughly 5 years due to a massive quantity of code that needed cleaning up, rewriting and properly testing, that had been contributed by various people. Many serious bugs have been fixed and others are still outstanding.

Bill.

[Fredrik Johansson]

On Thu, Feb 27, 2020 at 3:59 PM 'Bill Hart' via flint-devel <flint...@googlegroups.com> wrote:

Hi all,

New Flint Release?
===============

I am currently working on a new Flint release, which we hope to have available in a little over a month. The functionality above is currently in trunk, but not in an official release.

I will call for volunteers to help close some of the 80 odd remaining easy tickets in about two weeks, after I have closed all the really big tickets, the ones that only I can close and the really trivial ones. Volunteers would be really welcome at that point so that we can keep the release roughly on schedule for the end of March.

A Flint release has been held up for roughly 5 years due to a massive quantity of code that needed cleaning up, rewriting and properly testing, that had been contributed by various people. Many serious bugs have been fixed and others are still outstanding.

Here are the tickets: https://github.com/wbhart/flint2/milestone/2

Besides working on the open tickets, it would also be helpful if some volunteers could help edit the new documentation (currently at http://flintlib.org/sphinx/). It needs proofreading and light editing to fix broken markup, and the docs for the new modules should be checked for clarity/correctness.

Fredrik