Threadpool in Cython

[Robin Quessard]

Hello,

I am parallelizing some Cython code however I have 3 nested loops and can only parallelize the third one.

Basically I have a first loop where each iteration depends on the previous one. Inside this loop I have another loop where once again each iteration depends on the previous one. Inside this second loop I have a third loop which is parallelizable. 

My code looks like

for x in range(~200):

    ....

    for y in range(~10000):

        ....

        for z in prange(~200, nogil=True):

I have put the order of magnitude of the length of each loop. 

Anyway, I currently have some strange results where the parallelized code sometimes lead to a ~20% time reduction and sometimes lead to a x1000 time augmentation !

My guess is that I'm creating and deleting threads 10000 * 200 = 2M times which can create a huge slowdown.

My question is therefore : do you know if there is a way to recycle threads and declare a threadpool in Cython in order to parallelize my code without creating and deleting threads non-stop ? Is it maybe through the schedule and chunksize parameters that I can optimize my code better ?

Thank you for your help !

 

[Ashutosh Varma]

I don't think your problem is with creating and deleting threads. Cython use OpenMP as parallel backend which manages the threadpool.

For the first time when your prange loop rutn, ~200 threads are created and after the loop is done they goes to sleep. And when once again prange loop runs they are reused.

If you are sure that tasks take equal amount of time may be experimenting with chunksize with schedule=static could help.

----------------------------------------------------------------------

Ce message est confidentiel. Son contenu ne représente en aucun cas un

engagement de la part d' Akur8 sous réserve de tout accord conclu

par écrit entre vous et Akur8. Toute publication, utilisation

ou diffusion, même partielle, doit être autorisée préalablement. Si

vous n'êtes pas destinataire de ce message, merci d'en avertir

immédiatement l'expéditeur.

This message is confidential. Its contents do not constitute a

commitment by Akur8 except where provided for in a written

agreement between you and Akur8. Any unauthorised disclosure, use

or dissemination, either whole or partial, is prohibited. If you are

not the intended recipient of the message, please notify the sender

immediately.

----------------------------------------------------------------------

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/a850ecfe-60a3-42b7-b5b8-a3347c007bdcn%40googlegroups.com.