numpy.dtype to mpi4py datatype

[ble...@karlin.mff.cuni.cz]

Hi,

is there a simple way to get mpi4py.MPI.Datatype object from numpy.dtype?

Jan

[Lisandro Dalcin]

On 14 January 2015 at 13:55, <ble...@karlin.mff.cuni.cz> wrote:
> Hi,
>
> is there a simple way to get mpi4py.MPI.Datatype object from numpy.dtype?
>

Well, it is not "simple" in the sense that there is no public API for it.

In [1]: import numpy as np

In [2]: from mpi4py import MPI

In [3]: nt = np.dtype('float32')

In [4]: datatype = MPI._typedict[nt.char]

In [5]: datatype.Get_name()
Out[5]: 'MPI_FLOAT'


--
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

[Jan Blechta]

Dne středa 14. ledna 2015 13:14:49 UTC+1 Lisandro Dalcin napsal(a):

On 14 January 2015 at 13:55,  <ble...@karlin.mff.cuni.cz> wrote:
> Hi,
>
> is there a simple way to get mpi4py.MPI.Datatype object from numpy.dtype?
>

Well, it is not "simple" in the sense that there is no public API for it.

It's simple enough, thanks!
 

In [1]: import numpy as np

In [2]: from mpi4py import MPI

In [3]: nt = np.dtype('float32')

In [4]: datatype = MPI._typedict[nt.char]

I can see MPI.__TypeDict__ rather then MPI._typedict in mpi4py 1.3.1.
 

[Lisandro Dalcin]

On 14 January 2015 at 17:27, Jan Blechta <ble...@karlin.mff.cuni.cz> wrote:
> I can see MPI.__TypeDict__ rather then MPI._typedict in mpi4py 1.3.1.

Oh, yes, sorry, at some point I renamed it.

[Jan Blechta]

On Thu, 15 Jan 2015 10:45:24 +0300
Lisandro Dalcin <dal...@gmail.com> wrote:

> On 14 January 2015 at 17:27, Jan Blechta <ble...@karlin.mff.cuni.cz>
> wrote:
> > I can see MPI.__TypeDict__ rather then MPI._typedict in mpi4py
> > 1.3.1.
>
> Oh, yes, sorry, at some point I renamed it.

So MPI.__TypeDict__ is current and future name, right?

>
>

[Lisandro Dalcin]

In 1.3.1, it is __TypeDict__, in the next release, it will be
_typedict (unless I find a better, nicer, public API for this).

[Jan Blechta]

On Thu, 15 Jan 2015 12:18:44 +0300

Lisandro Dalcin <dal...@gmail.com> wrote:

> On 15 January 2015 at 11:23, Jan Blechta <ble...@karlin.mff.cuni.cz>
> wrote:
> > On Thu, 15 Jan 2015 10:45:24 +0300
> > Lisandro Dalcin <dal...@gmail.com> wrote:
> >
> >> On 14 January 2015 at 17:27, Jan Blechta
> >> <ble...@karlin.mff.cuni.cz> wrote:
> >> > I can see MPI.__TypeDict__ rather then MPI._typedict in mpi4py
> >> > 1.3.1.
> >>
> >> Oh, yes, sorry, at some point I renamed it.
> >
> > So MPI.__TypeDict__ is current and future name, right?
> >
>
> In 1.3.1, it is __TypeDict__, in the next release, it will be
> _typedict (unless I find a better, nicer, public API for this).

What about allowing construction of MPI.Datatype from np.dtype and
python types?

Jan

>
>

[Lisandro Dalcin]

On 15 January 2015 at 12:32, Jan Blechta <ble...@karlin.mff.cuni.cz> wrote:
> What about allowing construction of MPI.Datatype from np.dtype and
> python types?

Seems like the obvious solution. I have to think a little about how to
handle np.dtype, as mpi4py does not depend on numpy.

[talebi....@gmail.com]

Hi,

I am also looking for a solution for this problem to create a MPI datatype from numpy data type. There is a discussion in: https://stackoverflow.com/questions/36915351/how-to-create-struct-for-passing-between-nodes-using-mpi4py .  Is this still the solution or there is a more standard way that is implemented in mpi4py? Thank you.

[Lisandro Dalcin]

On Wed, 9 May 2018 at 14:45, <talebi....@gmail.com> wrote:

> I am also looking for a solution for this problem to create a MPI
datatype from numpy data type. There is a discussion in:
https://stackoverflow.com/questions/36915351/how-to-create-struct-for-passing-between-nodes-using-mpi4py
. Is this still the solution or there is a more standard way that is
implemented in mpi4py? Thank you.

There is nothing automatic yet. You have to do it by hand using
Datatype.Create_struct().

PS: Proposals very welcome, but remember that mpi4py currently DOES NOT
have numpy as a dependency.

--
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)

Extreme Computing Research Center (ECRC)

King Abdullah University of Science and Technology (KAUST)

http://ecrc.kaust.edu.sa/

4700 King Abdullah University of Science and Technology

al-Khawarizmi Bldg (Bldg 1), Office # 0109

[talebi....@gmail.com]

I see. I am working on that. To avoid adding the numpy dependency, we could have a "utility" module to add such functionality.

[talebi....@gmail.com]

I have come up with the function below that creates a MPI datatype based on some complex numpy structure array. Though, it cannot handle a complex structure that is inside the array. The assumption is that all types inside the structure can be found in the default MPI typedict using "MPI._typedict[dtype.char]".

def define_mpi_type(stc_array):
    field_dtypes = []
    field_shapes = []
    for field in stc_array.dtype.fields:
        myarfield = np.ascontiguousarray(stc_array[field])
        field_dtypes.append(myarfield.dtype)
        field_shapes.append(myarfield.shape)

    blens = ()
    disps = (0, )
    types=()
    i=-1
    for dtype in field_dtypes:
        i=i+1

        shape = field_shapes[i]
        blens = blens + (shape[1], )

        ar_datatype = MPI._typedict[dtype.char]

        disps = disps +(dtype.itemsize*shape[1],)

        types = types + (ar_datatype,)

    disps1 = (0,)
    for d in range (1,len(disps)-1):
        disps1 = disps1 + (disps1[d-1] + disps[d],)


    type1 = MPI.Datatype.Create_struct(blens, disps1, types)
    type1.Commit()

    return type1

[talebi....@gmail.com]

Hi,

I kind of implemented my own mpi datatype creation from nump dtype (similar to the last post). Now I am getting a problem with a new test case, see below. The output "type1" has the following:
type1.ub = 111464
type1.extent = 111464
type1.size = 111460

The problem is once I try to use type1 to transfer some numpy array, it says: "message: cannot infer count, buffer length 499898100 is not a multiple of datatype extent 111464 (lb:0, ub:111464) " apparently because the size is different to the extent. Any idea what is the problem?

def define_mpi_type2_test():
    dtype_char = ['B', 'B', 'I', 'd', 'd', 'd', 'd', 'd', 'd', 'd', 'd', 'f']
    blens = (16, 16, 1, 1, 1, 1, 1, 1, 1, 1, 1, 27840)
    disps1 = (0, 16, 32, 36, 44, 52, 60, 68, 76, 84, 92, 100)
    types=()
    for i in range (len(dtype_char)):
        ar_datatype = MPI._typedict[dtype_char[i]]

        types = types + (ar_datatype,)

    type1 = MPI.Datatype.Create_struct(blens,disps1,types)
    type1.Commit()
    return type1

[Lisandro Dalcin]

On Fri, 5 Apr 2019 at 12:25, <talebi....@gmail.com> wrote:

Hi,

I kind of implemented my own mpi datatype creation from nump dtype (similar to the last post). Now I am getting a problem with a new test case, see below. The output "type1" has the following:
type1.ub = 111464
type1.extent = 111464
type1.size = 111460

The problem is once I try to use type1 to transfer some numpy array, it says: "message: cannot infer count, buffer length 499898100 is not a multiple of datatype extent 111464 (lb:0, ub:111464) " apparently because the size is different to the extent. Any idea what is the problem?

By default, numpy dtypes are "packed" (no padding for alignment), while MPI datatypes are not (to match default compiler behavior).

My recommendation would be to create the numpy dtype with align=True, https://docs.scipy.org/doc/numpy/reference/generated/numpy.dtype.html#numpy.dtype

--

Lisandro Dalcin
============
Research Scientist

[Lisandro Dalcin]

Please note that the code I've attached in the previous email is WIP, there are quite a few of cases where things will not work, the full MPI type system is not so trivial...

[talebi....@gmail.com]

This indeed worked for my case. 

As always, very helpful. Thanks very much!