prange and thread affinity

[Rémi Lehe]

I was wondering whether there is a way to set the thread affinity (i.e. how threads bind to physical cores) when using Numba and prange.

For instance, let us assume that I am running an application which is parallelized with MPI, and then with prange for each MPI rank, and that I have access to, say, a compute node with 2 NUMA domains and 12 cores in each NUMA domains.

In that case, one reasonable choice would be to have 2 MPI ranks running 12 threads each, whereby each MPI rank runs on a different NUMA domain (I believe there are options in mpirun to do this), and whereby each thread runs on the same NUMA domain as the MPI rank that created them.

In OpenMP, setting this type of thread affinity is possible through environment variables:
export OMP_NUM_THREADS=12
export OMP_PLACES=cores
export OMP_PROC_BIND=close
(see e.g. http://pages.tacc.utexas.edu/~eijkhout/pcse/html/omp-affinity.html)

Are there similar similar options in Numba, when using prange?
(I know only about `export NUMBA_NUM_THREADS=12`, but that does not control thread affinity.)

Thanks!

[Jim Pivarski]

I was also wondering to what extent prange is like OpenMP. For instance, does it do something like static scheduling? Dynamic? Does it have a chunk size?

Is there a conversation chart somewhere? I understand that prange is not as fully featured as OpenMP's parallel-for, but it would be good to have a translation dictionary from one language to another and the limitations explicitly spelled out.

Such as, for instance, Rémi's question about pinning threads.

Jim

--
You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numba-users+unsubscribe@continuum.io.
To post to this group, send email to numba...@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/579bd178-685d-4599-87c5-6f4389aca713%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

[stuart]

<I've stripped email addresses>

Thanks for the questions. Answering them in chronological in order.

The short answer to setting thread affinity and thread placement options is that there is no way to do this yet from Numba, and off the top of my head I couldn't think of a combination of `taskset`, `numactl` and `cgroups` to achieve this either. I've added an issue here: https://github.com/numba/numba/issues/2496. Perhaps someone else might know of a way with external tools?

As to the actual implementation of prange and how much it is like OpenMP, there is a large section of documentation on implementation details in PR state here: https://github.com/numba/numba/pull/2487/files. There is no conversion chart from OpenMP at present but I think as prange becomes more feature rich and moves out of being experimental that might be a good thing to have, thanks for the suggestion.

Thanks,

--
stuart

[rl...@lbl.gov]

Great, thanks for the information, Stuart.