Max message size??? (bytes received but buffer size is 32768)

[a.ge...@gmail.com]

Hi guys,

I am using mpi4py to implement a agent-based model in python. Everthing was running quite nice, but now I ran into a problem.

I am using MPI to transfer many properties integers, floats, and list or tuples of then. All put in one long list (of lists...) I use the pickle-version of isend and recv

and for small data everthing works. But for more data to transfer (seem over 32768 Bytes) I get the following message:

""""

dataDict[nodeType, mpiPeer] = requestDict[nodeType, mpiPeer].wait()

  File "MPI/Request.pyx", line 234, in mpi4py.MPI.Request.wait (src/mpi4py.MPI.c:75805)

  File "MPI/msgpickle.pxi", line 397, in mpi4py.MPI.PyMPI_wait (src/mpi4py.MPI.c:44201)

mpi4py.MPI.Exception: Message truncated, error stack:

PMPI_Wait(183)....................: MPI_Wait(request=0x7f180b7447e0, status=0x7ffd1f1d9b30) failed

MPIR_Wait_impl(77)................: 

MPIDI_CH3U_Receive_data_found(129): Message from rank 0 and tag 3 truncated; 53012 bytes received but buffer size is 32768

"""""

Is there a maximum message size for pickled mpi messages? I found no other problem, since the rest of the setup ist the same, only the amount of data increases.

I am using mpi4py version 2.0.0 on ubuntu 12.04. 

Thanks for the help!!

Best,
Andreas

[Lisandro Dalcin]

On 14 July 2017 at 10:39, <a.ge...@gmail.com> wrote:
>
> Is there a maximum message size for pickled mpi messages? I found no other
> problem, since the rest of the setup ist the same, only the amount of data
> increases.
>

You are likely using "irecv()". Unfortunately, there is no way to
issue an "irecv()" call and determine the size of the incoming
message. This is an MPI limitation, but also a feature, as you can
post an "irecv()" at a point in time were the matching send operation
has not yet being issued. In an attempt to support "irecv()" for
pickled streams, mpi4py allocates a buffer of 32768 bytes to receive
the message. This is obviously not enough for your messages, so the
default buf size leads to truncated messages in your case. There are
three ways to "fix" this problem:

1) Avoid using "irecv()", maybe you can refactor your code to use
"iprobe()" and "recv()", though that is not thread-safe.
2) Issue plain "recv()" in a separate thread, mpi4py releases the GIL
and things will just work. Of course, you need an MPI implementation
with proper support for MPI_THREAD_MULTIPLE.
3) Use a larger buffer size by passing an integer as first argument,
i.e, use "request = comm.irecv(131072, source, tag)". This is likely
the easiest approach, but with a few downsides: you allocate too many
memory pages if the message size is small, or the buffer may still not
be enough if your messages ever grow larger than your guess.

--
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 0109
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

[Lisandro Dalcin]

On 14 July 2017 at 10:39, <a.ge...@gmail.com> wrote:

> Hi guys,
>
> I am using mpi4py to implement a agent-based model in python. Everthing was
> running quite nice, but now I ran into a problem.

BTW, I would like to bring to your attention this new thing I'm
implementing: https://mpi4py.readthedocs.io/en/latest/mpi4py.futures.html,
you need to install mpi4py from the master branch of the git repo. Not
sure if it is actually useful for your application, but just in
case...

[Devansh Tripathi]

 I am the same issue but in julia. I think language won't make a difference in my question.

In my case, I am `u[2]` which is of size 8 bytes (used sizeof() to check this) but MPI error message of showing 17 bytes as message size. Here is the error message:

`MPIDI_CH3U_Receive_data_found(120): Message from rank 0 and tag 0 truncated; 17 bytes received but buffer size is 8`

`sizeof(right_bc) = 8      # right_bc = buffer for receiving data `

`sizeof(u[2]) = 8              # u[2] = sending message `

So why am I receiving this error of 17 bytes? How 17 bytes come into picture. If you want I can attach the related code snippet too.

--

Devansh,

School of Mathematics,

IISER TVM, Kerala, 

India

[Lisandro Dalcin]

On Sun, 2 Jun 2024 at 10:21, Devansh Tripathi <devanshtrip...@gmail.com> wrote:

 I am the same issue but in julia. I think language won't make a difference in my question.

What you think may not hold true. Please ask MPI.jl developers about your issue.

--

Lisandro Dalcin
============
Senior Research Scientist