How to offset MPI_Type_vector?

[George Way]

In C you can send columns of a matrix

- first column

MPI_Send( mat, 1, MPI_column, ... );

- second column

MPI_Send( &(mat[0][1]), 1, MPI_column, ... );

datatype: MPI_Type_vector(count, blocklength, stride, MPI_DOUBLE, &MPI_column);

Can a vector datatype be used to send another column than the first one?

With mpi4py I tried to offset the beginning of the datatype by using 'Create_resized' to set 'lower bound' different from 0. Sadly this makes no difference.

MPI_column = MPI.DOUBLE.Create_vector(count, blocklength, stride)

MPI_column = MPI_column.Create_resized(lowerbound, mat.nbytes)

How to offset MPI_Type_vector?

[Lisandro Dalcin]

You are confusing concepts. Look again at your code... are you offsetting MPI_Type vector? No! You are offsetting the buffer, not the MPI datatype.

So the solution is to figure out how to offset the buffer in a language like Python with NumPy arrays and a different memory model, where you don't just get raw memory addresses with the C "&" operator like in "&mat[0][1]".

Additionally, you also be very specific about what you are trying to communicate, otherwise mpi4py may error out of overzealous error checking. Long story short, here you have the code:

from mpi4py import MPI

import numpy as np

comm = MPI.COMM_WORLD
rank = comm.Get_rank()

n = 5
mat = np.arange(n*n, dtype='i').reshape(n, n).astype('d')

mpi_column = MPI.DOUBLE.Create_vector(
    count=n,
    blocklength=1,
    stride=n
).Commit()

idx = 1 # column index in range(0, n)
buf = (mat[0, idx:idx+1], 1, mpi_column)  # (buf, count, datatype)
col = np.zeros(n)
MPI.COMM_WORLD.Sendrecv(
    sendbuf=buf, dest=rank,
    recvbuf=col, source=rank,
)

print(mat)
print(col)

--

Lisandro Dalcin
============
Senior Research Scientist
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/

[George Way]

Now I get it. Thanks.

[George Way]

I ended up using Create_subarray.

Consider buf = (X, 1, mpi_column)  # (buf, count, datatype)

Apparently X has to be a np.array of length 1 containing the first element.

Is there a way to make X independent of the number of dimensions? I only need the arrays first element.

Current solution:

for 2D: X = mat[0:1, 0:1]

for 3D: X = mat[0:1, 0:1, 0:1]

When copying X = np.array(mat[0]) i get segfault Address not mapped.

[Lisandro Dalcin]

On Wed, 17 Feb 2021 at 12:44, George Way <demonhunte...@gmail.com> wrote:

I ended up using Create_subarray.

Consider buf = (X, 1, mpi_column)  # (buf, count, datatype)

Apparently X has to be a np.array of length 1 containing the first element.

Actually, X can be any length, and that length will be ignored because you pass count=1 (second tuple item).

But X has to be (C or Fortran) contiguous.

 

Is there a way to make X independent of the number of dimensions? I only need the arrays first element.

If you use Create_subarray() properly, then you just pass (X, 1, subarray_datatype). X can be any dimension, and subarray_datatype is in charge of encoding the slice you want to communicate.

For example, suppose X is a C-contiguous 4D array of shape (N0,N1,N2,N3) and you want to communicate the slice X[:, a:b, :, :], then you create the subarray datatype corresponding to the slice this way:

subarray_datatype = MPI.DOUBLE.Create_subarray(

    sizes=(N0, N1, N2, N3),

    subsizes=(N0, b-a, N2, N3),

    substarts=(0, a, 0, 0),

    order=MPI.ORDER_C, # ir your array has Fortran order, use MPI.ORDER_FORTRAN

)

--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpi4py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/fbdc2d16-a748-4452-8284-b7c90d0c94cdn%40googlegroups.com.

[George Way]

Thank you for the helpful Create_subarray explanation.

Sorry, the point is that X is not contiguous in my case.

I have to get array(first_element of X).

Actually this is a numpy related question.

[Lisandro Dalcin]

On Wed, 17 Feb 2021 at 14:42, George Way <demonhunte...@gmail.com> wrote:

Thank you for the helpful Create_subarray explanation.

Sorry, the point is that X is not contiguous in my case.

I have to get array(first_element of X).

OK, but then... if your X is non-contiguous, then it is most likely a slice of another parent contiguous array. It is respect to that parent contiguous array that you have to create your subarray datatype.

Do you have control about the non-contiguous X array? perhaps you should avoid taking slices, and rather construct your subarray datatype from the base array ?