Parallel Computing using Cython

[consuli]

I have figured out, that parallel computing using CPython Interpreter is pretty unattractive, because of the GIL. Compare http://python-forum.io/Thread-When-will-the-GIL-be-removed?pid=5778#pid5778

So my question are:
1. Can I use C/C++ parallel computing interfaces like MPI in Cython to bypass the Python GIL?
2. Are there more comfortable (high level) parallel computing interfaces in C/C++ than MPI?
3. Is it possible to make use of them with nearly no C/C++ knowledge?

Regards
Consuli

[Hai Nguyen]

You can check mpi4py for MPI stuff (although I do not answer your questions).

Hai 

[Peter]

I don't know if this directly answers your question, but in that thread you call the python "multiprocessing" package a "dirty GIL workaround". Well dirty it might be, but it works. 

I modified your code in that thread to use multiprocessing imap(), and got a 3.4 speedup on a 4 processor machine. Then I cythonised it (a couple of simple modifications to your function is_prime) and got a further 9 to 10x speedup. So yes, I regularly do parallel computing in Cython.

[Meagan Lang]

1. You can use MPI at the C/C++ level and then wrap it in Cython. This is my favorite method as it gives the greatest speedup and gives you the most control. This bypasses the Python GIL because execution of the calling python script will be done by launching a new python interpreter on each process.

2. I have never used/needed anything at a higher level than MPI in C/C++. OpenMP threaded parallelism may be considered to be at a higher level by some and could be useful in some cases, though I have never tried to wrap an OpenMP parallelized code in Cython.

3. Use of both MPI and OpenMP will require some knowledge of C/C++. If you are looking for something fully in Python, try mpi4py or multiprocessing as previously suggested (though it seems you have already tried multiprocessing). These can both provide some speedup, although I have found both to be rather clumsy when compared with just C/C++ MPI and are more prone to miss communications, hang on locked up queues/pipes, and/or ignore barriers.

 - Meagan

[Chris Barker]

Another option is to use one of the higher level libs that let you do out-of-gil multi-threading/.processing with python:

Theano: made for GPUs, but I think it will do multicore CPUs too)

numexpr: only simple expressions, but maybe that's enough?

numba: JIT compiler for LLVM

dask: parallel programing and task management

These all let you write Python (numpy) code, and run it a multiprocessor environment at a function by function or expression by expression level.

-CHB

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov