How to offset MPI_Type_vector?

12 views
Skip to first unread message

George Way

unread,
Dec 9, 2020, 11:19:14 AM12/9/20
to mpi4py
In C you can send columns of a matrix
- first column
MPI_Send( mat, 1, MPI_column, ... );
- second column
MPI_Send( &(mat[0][1]), 1, MPI_column, ... );

datatype: MPI_Type_vector(count, blocklength, stride, MPI_DOUBLE, &MPI_column);

Can a vector datatype be used to send another column than the first one?
With mpi4py I tried to offset the beginning of the datatype by using 'Create_resized' to set 'lower bound' different from 0. Sadly this makes no difference.

MPI_column = MPI.DOUBLE.Create_vector(count, blocklength, stride)
MPI_column = MPI_column.Create_resized(lowerbound, mat.nbytes)

How to offset MPI_Type_vector?

Lisandro Dalcin

unread,
Dec 10, 2020, 1:46:13 AM12/10/20
to mpi...@googlegroups.com
You are confusing concepts. Look again at your code... are you offsetting MPI_Type vector? No! You are offsetting the buffer, not the MPI datatype.
So the solution is to figure out how to offset the buffer in a language like Python with NumPy arrays and a different memory model, where you don't just get raw memory addresses with the C "&" operator like in "&mat[0][1]".
Additionally, you also be very specific about what you are trying to communicate, otherwise mpi4py may error out of overzealous error checking. Long story short, here you have the code:

from mpi4py import MPI
import numpy as np

comm = MPI.COMM_WORLD
rank = comm.Get_rank()

n = 5
mat = np.arange(n*n, dtype='i').reshape(n, n).astype('d')

mpi_column = MPI.DOUBLE.Create_vector(
    count=n,
    blocklength=1,
    stride=n
).Commit()

idx = 1 # column index in range(0, n)
buf = (mat[0, idx:idx+1], 1, mpi_column)  # (buf, count, datatype)
col = np.zeros(n)
MPI.COMM_WORLD.Sendrecv(
    sendbuf=buf, dest=rank,
    recvbuf=col, source=rank,
)

print(mat)
print(col)


--
Lisandro Dalcin
============
Senior Research Scientist
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/

George Way

unread,
Dec 10, 2020, 4:57:43 AM12/10/20
to mpi4py
Now I get it. Thanks.

George Way

unread,
Feb 17, 2021, 4:44:26 AMFeb 17
to mpi4py
I ended up using Create_subarray.
Consider buf = (X, 1, mpi_column)  # (buf, count, datatype)
Apparently X has to be a np.array of length 1 containing the first element.
Is there a way to make X independent of the number of dimensions? I only need the arrays first element.
Current solution:
for 2D: X = mat[0:1, 0:1]
for 3D: X = mat[0:1, 0:1, 0:1]
When copying X = np.array(mat[0]) i get segfault Address not mapped.

Lisandro Dalcin

unread,
Feb 17, 2021, 6:09:47 AMFeb 17
to mpi...@googlegroups.com
On Wed, 17 Feb 2021 at 12:44, George Way <demonhunte...@gmail.com> wrote:
I ended up using Create_subarray.
Consider buf = (X, 1, mpi_column)  # (buf, count, datatype)
Apparently X has to be a np.array of length 1 containing the first element.

Actually, X can be any length, and that length will be ignored because you pass count=1 (second tuple item).
But X has to be (C or Fortran) contiguous.
 
Is there a way to make X independent of the number of dimensions? I only need the arrays first element.

If you use Create_subarray() properly, then you just pass (X, 1, subarray_datatype). X can be any dimension, and subarray_datatype is in charge of encoding the slice you want to communicate.

For example, suppose X is a C-contiguous 4D array of shape (N0,N1,N2,N3) and you want to communicate the slice X[:, a:b, :, :], then you create the subarray datatype corresponding to the slice this way:

subarray_datatype = MPI.DOUBLE.Create_subarray(
    sizes=(N0, N1, N2, N3),
    subsizes=(N0, b-a, N2, N3),
    substarts=(0, a, 0, 0),
    order=MPI.ORDER_C, # ir your array has Fortran order, use MPI.ORDER_FORTRAN
)
--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpi4py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/fbdc2d16-a748-4452-8284-b7c90d0c94cdn%40googlegroups.com.

George Way

unread,
Feb 17, 2021, 6:42:38 AMFeb 17
to mpi4py
Thank you for the helpful Create_subarray explanation.
Sorry, the point is that X is not contiguous in my case.
I have to get array(first_element of X).
Actually this is a numpy related question.

Lisandro Dalcin

unread,
Feb 17, 2021, 7:11:05 AMFeb 17
to mpi...@googlegroups.com
On Wed, 17 Feb 2021 at 14:42, George Way <demonhunte...@gmail.com> wrote:
Thank you for the helpful Create_subarray explanation.
Sorry, the point is that X is not contiguous in my case.
I have to get array(first_element of X).

OK, but then... if your X is non-contiguous, then it is most likely a slice of another parent contiguous array. It is respect to that parent contiguous array that you have to create your subarray datatype.
Do you have control about the non-contiguous X array? perhaps you should avoid taking slices, and rather construct your subarray datatype from the base array ?


Reply all
Reply to author
Forward
Message has been deleted
Message has been deleted
0 new messages