passing an MPI communicator from mpi4py to C via ctypes

[Steve Plimpton]

Hi - I have several parallel codes/libraries with C-style interfaces
that we wrap with Python via ctypes.

The thing I don't know how to do in mpi4py (or any other Python/MPI
package) is pass an MPI communicator from Python to our C function
that effectively instantiates the library. I see that this is
possible with mpi4py via SWIG, as there is an example in the manual.

However, I'd really rather do it via ctypes (i.e. all in Python),
since it makes everything else so easy. Ideally it would then
portably work with any MPI that both mpi4py and my lib was built with,
even though different MPIs store communicators differently.

Does anyone know if this is possible, or how to do it?

The key SWIG lines (from the manual example) are these:
%include mpi4py/mpi4py.i
%mpi4py_typemap(Comm, MPI_Comm);
but I don't know if/how that can be translated to ctypes lingo.

It seems like it should be possible, since ctypes supports a variety
of pointer data types, and I presume that under-the-hood, mpi4py is
storing pointers to the native MPI communicators that it passes to MPI
directly. If I could pass that pointer to my lib, it would be easy
to de-reference.

Thanks,
Steve

[Aron Ahmadia]

You can always use a void pointer if you don't want to use mpi4py's Cython bindings. 

--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpi4py+un...@googlegroups.com.
To post to this group, send email to mpi...@googlegroups.com.
Visit this group at http://groups.google.com/group/mpi4py.
To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/201407152351.s6FNprTA009465%40singsing.sandia.gov.
For more options, visit https://groups.google.com/d/optout.

[Lisandro Dalcin]

On 16 July 2014 10:35, Aron Ahmadia <ar...@ahmadia.net> wrote:
> You can always use a void pointer if you don't want to use mpi4py's Cython
> bindings.
>

The thing is that currently mpi4py has no way to expose the underlying
raw MPI handle to pure Python code. Additionally, to expose the handle
we should deal with the differences of MPI implementations (i.e., in
MPICH handles are "int", but in Open MPI they are "void*").

I would like to add support in mpi4py to use ctypes or cffi (which
IMHO is better than ctypes), but I'm failing to devise a Python-level
API letting users pass handles around in Python code.

One possible option would be to provide methods c2py() and py2c()
(following the Fortran-specific conversion functions). These functions
would return an integer that users could employ to create ctypes or
cffi instances. What do you think?




--
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169

[Steve Plimpton]

> I would like to add support in mpi4py to use ctypes or cffi (which
> IMHO is better than ctypes), but I'm failing to devise a Python-level
> API letting users pass handles around in Python code.
>
> One possible option would be to provide methods c2py() and py2c()
> (following the Fortran-specific conversion functions). These functions
> would return an integer that users could employ to create ctypes or
> cffi instances. What do you think?

Thanks for the quick response ...

I don't fully understand the issue and option you mention, but what I
want to be able to do in a Python/mpi4py program is the following. I
think "handle" just needs to be returned as a pointer to the MPI_Comm
"object" that a particular MPI uses, e.g. an MPICH int or maybe the
OpenMPI void ptr itself:

comm = mpi4py.MPI.COMM_WORLD # or any other comm that mpi4py stores
handle = mpi4py.magic(comm) # the magic: a ptr to the MPI_Comm "object"
mylib_create(handle) # this is a C func in my lib,
# via ctypes I will pass "handle"
# as a void ptr
mylib_method() # off to the races ...

My C funcs are like this, just a thin wrapper on a C++ lib, that takes
an MPI_Comm in its constructor:

MyLib *lib;
void mylib_create(void *ptr) {
MPI_Comm comm = *((MPI_Comm *) ptr);
lib = new MyLib(comm);
}
void mylib_method() {
lib->method();
}

Note that if this line can be made to work:
handle = mpi4py.magic(comm)
then I'm done.

I don't need to write any additional C code, use SWIG, compile an
extension to Python, install anything in Python, etc. My lib is
just built as a shared lib and loaded at run-time by ctypes. The lib
needs to know nothing about Python or mpi4py.

Steve

------------------------

[Lisandro Dalcin]

On 17 July 2014 18:46, Steve Plimpton <sjp...@sandia.gov> wrote:
> Note that if this line can be made to work:
> handle = mpi4py.magic(comm)
> then I'm done.
>
> I don't need to write any additional C code, use SWIG, compile an
> extension to Python, install anything in Python, etc. My lib is
> just built as a shared lib and loaded at run-time by ctypes. The lib
> needs to know nothing about Python or mpi4py.

Here you have. Take a look at the example in demo/

https://bitbucket.org/mpi4py/mpi4py/commits/95760010758b8cee15a8e005896f571a3781138b

PS: I still do not like this approach. The thing is that all your
library functions need to written to accept a pointer to MPI_Comm.
What if the library you are wrapping is not your own code and you have
to pass a plain MPI_Comm instead of a pointer? I'm still thinking on
something better, but anyway the _addressof() function will stay there
and will be available in the next release.

[Steve Plimpton]

Thanks Lisandro! and that was fast! This looks like just what I need.

>PS: I still do not like this approach. The thing is that all your
>library functions need to written to accept a pointer to MPI_Comm.
>What if the library you are wrapping is not your own code and you have
>to pass a plain MPI_Comm instead of a pointer?

My libs only take an MPI_Comm (not a pointer), for a single method,
the constructor. I think that is pretty common for scientific libs
(e.g. an FFT lib) that is designed to be instantiated from another
parallel code (Python or otherwise). An instance of the lib just
needs to know (one time) what comm it is running on, so its internal
MPI calls can be consistent with the caller. If the caller wants to
setup different MPI communicators, e.g. on subsets of procs, then it
just instantiates the lib once per communicator.

Any lib like that just needs to provide an alternate constructor
that accepts MPI_Comm *, and converts it to an MPI_Comm. That's
what I'll now add to my libs.

>I'm still thinking on
>something better, but anyway the _addressof() function will stay there
>and will be available in the next release.

I am a newbie to mpi4py, so not clear which of these this means:

a) there is now a development version of mpi4py I could
grab that would have this feature

b) I need to do something special to my version of mpi4py
to add this feature, thus only I could use it

c) it will be available at some point soon in the main
version of mpi4py (when would that be?)

I ask, b/c I plan to now include this logic in the Python wrappers for
several libs that we distribute open-source, and advise our users to
use mpi4py if they want to use the libs from a parallel Python script.

So I'm also asking when it will be available for general
users of mpi4py.

Thanks again,
Steve

[Lisandro Dalcin]

On 18 July 2014 19:46, Steve Plimpton <sjp...@sandia.gov> wrote:
>
> Any lib like that just needs to provide an alternate constructor
> that accepts MPI_Comm *, and converts it to an MPI_Comm. That's
> what I'll now add to my libs.
>

Yes, of course, but I insist: What if the library does not provide a
constructor for MPI_Comm*, but just MPI_Comm? I would like to support
such use case. For example, you mentioned FFTW, however look at the
API http://www.fftw.org/doc/MPI-Data-Distribution-Functions.html ,
these routines accept "MPI_Comm" handles, but not pointers to
MPI_Comm. Then you need to write some C code to wrap these calls to
accept a pointer. I would rather provide some mechanism in mpi4py such
that you can use ctypes with the stock libfftw, no need to write C
wrapper code. Am I being clear enough?

>
> a) there is now a development version of mpi4py I could
> grab that would have this feature
>
> b) I need to do something special to my version of mpi4py
> to add this feature, thus only I could use it
>

For now, you have to git clone the repository from Bitbucket, install
Cython (required to work with the in-development mpi4py), and then
re-install mpi4py.

> c) it will be available at some point soon in the main
> version of mpi4py (when would that be?)
>

In the near future, I'll make a new release mpi4py 2.0 that will
include this feature.

> I ask, b/c I plan to now include this logic in the Python wrappers for
> several libs that we distribute open-source, and advise our users to
> use mpi4py if they want to use the libs from a parallel Python script.
>
> So I'm also asking when it will be available for general
> users of mpi4py.
>

I cannot tell you a definitive release date, but hopefully it should
happens in a couple of months.

[Lisandro Dalcin]

On 18 July 2014 19:46, Steve Plimpton <sjp...@sandia.gov> wrote:
> Any lib like that just needs to provide an alternate constructor
> that accepts MPI_Comm *, and converts it to an MPI_Comm. That's
> what I'll now add to my libs.

I've added MPI._sizeof() and updated the ctypes wrapping example:
https://bitbucket.org/mpi4py/mpi4py/src/master/demo/wrap-ctypes/helloworld.py

Now there is no need to implement alternate constructors accepting
"void*" or "MPI_Comm *", a plain "MPI_Comm" would do as long as you
are careful to use MPI._sizeof() to check if the MPI handle fits in
"int" to use ctypes.c_int or otherwise use ctypes.c_void_p.

PS: Your former approach still works. Just wanted to let you know
about this new approach that IMHO is less cryptic.

[Steve Plimpton]

sorry for the delayed reply - for some reason your subsequent
messages didn't get CCd to my email ...

>Yes, of course, but I insist: What if the library does not provide a
>constructor for MPI_Comm*, but just MPI_Comm? I would like to support

>such use case ... I would rather provide some mechanism in mpi4py such

>that you can use ctypes with the stock libfftw, no need to write C
>wrapper code. Am I being clear enough?

yes, I agree that would be ideal. In my case, I'm only wrapping
my own libs, so it's not an issue to add a constructor
what accepts MPI_Comm *, instead of just MPI_Comm.

Your subsequent post that handles this on the Python side:

if MPI._sizeof(MPI.Comm) == ctypes.sizeof(ctypes.c_int):
MPI_Comm = ctypes.c_int
else:
MPI_Comm = ctypes.c_void_p

seems like a clever solution. I suppose there could be some alternate
MPI that doesn't fall in those 2 categories, but I don't know of any
MPI where MPI_Comm isn't either an int or a pointer.

I'll look forward to seeing these add-ons in mpi4py 2.0.

Thanks again for being responsive,
Steve

[noam.be...@gmail.com]

On Friday, July 18, 2014 at 8:14:51 PM UTC-4, Lisandro Dalcin wrote:

I cannot tell you a definitive release date, but hopefully it should
happens in a couple of months.

Hi - I'm just chiming in to say that it would be really useful for me if this feature you were discussing with Steve Plimpton were released. I'm interested in using his library with python, and it's been great working with mpi4py so far, but this additional capability would be a great help.

Thanks for all the good work with mpi4py.

         Noam 

[Lisandro Dalcin]

On 30 June 2015 at 15:45, <noam.be...@gmail.com> wrote:
> Hi - I'm just chiming in to say that it would be really useful for me if
> this feature you were discussing with Steve Plimpton were released. I'm
> interested in using his library with python, and it's been great working
> with mpi4py so far, but this additional capability would be a great help.

Well, the feature is certainly available in the master branch. About
making an official release, I've been extremely busy with other
matters. Besides, I'm waiting for Open MPI folks to fix bugs they
introduced in 1.8.6, so I can get my tests run clean.


--
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

[georg...@gmail.com]

Hi,

I have the opposite request: I would like to pass an MPI communicator from C to python. Is there a way to initialize mpi4py communicator using the address of the communicator created in C?

Thanks,

George.

On Wednesday, July 1, 2015 at 4:47:06 AM UTC-4, Lisandro Dalcin wrote:

[Lisandro Dalcin]

On Wed, 10 Apr 2019 at 21:56, <georg...@gmail.com> wrote:

Hi,

I have the opposite request: I would like to pass an MPI communicator from C to python. Is there a way to initialize mpi4py communicator using the address of the communicator created in C?

Look at test/test_ctypes.py

--

Lisandro Dalcin
============
Research Scientist

Extreme Computing Research Center (ECRC)

King Abdullah University of Science and Technology (KAUST)

http://ecrc.kaust.edu.sa/

[George Zaki]

Thanks Lisandro. That was very helpful.

Best regards,

George.

--
You received this message because you are subscribed to a topic in the Google Groups "mpi4py" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mpi4py/jPqNrr_8UWY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mpi4py+un...@googlegroups.com.

To post to this group, send email to mpi...@googlegroups.com.

Visit this group at https://groups.google.com/group/mpi4py.
To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/CAEcYPwBG_-63Vg61hO6tn0%3DopH2reZY4pktgsD%3DzHQzvCGzZ7A%40mail.gmail.com.