Gathering dictionaries

[Jacob Wren]

Hello,

I am gathering a list of nested dictionaries. This worked as expected via:

   comm.gather(nested_dicts, root=0) (*)

But, as I have scaled, I encountered the error:

   SystemError: Negative size passed to PyBytes_FromStringAndSize

Thus, I instead tried to use comm.Gather(sendbuf, recvbuf, root=0). Now I am receiving the error:

   TypeError: a bytes-like object is required, not 'dict'

Where I replaced (*) with:

   recvbuf = None
   if rank == 0:
       recvbuf = np.empty([size, len(nested_dicts)])
   comm.Gather(nested_dicts, recvbuf, root=0)

I would appreciate any help.

Many thanks,

Jake

[Lisandro Dalcin]

On Sun, 6 Nov 2022 at 04:34, Jacob Wren <jacob...@gmail.com> wrote:

 

I am gathering a list of nested dictionaries. This worked as expected via:

   comm.gather(nested_dicts, root=0) (*)

But, as I have scaled, I encountered the error:

   SystemError: Negative size passed to PyBytes_FromStringAndSize

This probably means the total size of the incoming data at the root is overflowing the infamous 32bit count limit of MPI. 

In other words, the concatenation of all pickled streams at root=0 is larger than ~2Gbytes (INT_MAX = 2^31-1).

mpi4py tries to detect and explicitly error on these overflows, but this particular case escaped my checks and instead you get a rather cryptic error.

 

Thus, I instead tried to use comm.Gather(sendbuf, recvbuf, root=0). Now I am receiving the error:

   TypeError: a bytes-like object is required, not 'dict'

Uppercase Gather() is to be used with buffer-like objects, e.g. numpy arrays, and not complex nested Python objects like lists/tuples//user-defined classes.

 

Where I replaced (*) with:

   recvbuf = None
   if rank == 0:
       recvbuf = np.empty([size, len(nested_dicts)])
   comm.Gather(nested_dicts, recvbuf, root=0)

I would appreciate any help.

Unfortunately, the only solution to this is to use an MPI-4 implementation with support for large-count (64bit) messages. Up to today, only MPICH v4.x supports MPI-4. And to take advantage of the new large-count support, you should use mpi4py from git master branch.

Oh, I forgot... You could try to use mpi4py.util.pkl5 (https://mpi4py.readthedocs.io/en/latest/mpi4py.util.pkl5.html). To use this module, you basically create a communicator wrapper object that redefines methods send, recv, and so on, and use a different way to communicate objects (multiple messages, and workarounds for the 32bit count limit of MPI-3.x).

Here you have an example that is able to communicate more than 4GB with lowercase send: https://mpi4py.readthedocs.io/en/latest/mpi4py.util.pkl5.html#test-pkl5-2 This example uses a large numpy array, but it could be any picklable Python object.

Unfortunately, you need gather(), and support for gather()/scatter()/allgather()/alltoall() is only available in mpi4py git master branch. 

Installing mpi4py from git master is easy, though: 

python -m pip install https://github.com/mpi4py/mpi4py/tarball/master

--

Lisandro Dalcin
============
Senior Research Scientist
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/

[Jacob Wren]

Thank you for the workaround, Lisandro.

[Jacob Wren]

Hi Lisandro,

> Up to today, only MPICH v4.x supports MPI-4.

   I am working on a cluster that only provides Open MPI. Any idea if Open MPI will support MPI-4 in the near future?

Thanks,

Jake

On Thursday, November 10, 2022 at 12:55:58 AM UTC-6 dal...@gmail.com wrote:

[Lisandro Dalcin]

On Mon, 14 Nov 2022 at 14:42, Jacob Wren <jacob...@gmail.com> wrote:

Hi Lisandro,

> Up to today, only MPICH v4.x supports MPI-4.

   I am working on a cluster that only provides Open MPI. Any idea if Open MPI will support MPI-4 in the near future?

Open MPI from its git repository main branch (and v5.0.x branch which will become the upcoming release 5.0.0) already have partial support for MPI-4. All new MPI-4 features are already implemented, with the exception of the large-count MPI APIs, and adding all these new routines may take them a while. I'm not a member of the Open MPI team, you should really ask them about the timeline. Moreover, asking them about it may help to put some pressure on them to prioritize work.

Regards,

[Jacob Wren]

Hi Lisandro,

> Unfortunately, the only solution to this is to use an MPI-4 implementation with support for large-count (64bit) messages. Up to today, only MPICH v4.x supports MPI-4. And to take advantage of the new large-count support, you should use mpi4py from git master branch.

With the above implementation, is it still possible that the total size of the incoming data at the root is too large? 

I will be gathering several million pairs of floats. Is there a way to know whether this will be a problem ex-ante?

I suppose I could use a numpy array of object datatype, but I prefer the simplicity of lower case gather().

Thanks,

Jake

[Lisandro Dalcin]

On Mon, 21 Nov 2022 at 18:34 Jacob Wren <jacob...@gmail.com> wrote:

Hi Lisandro,

> Unfortunately, the only solution to this is to use an MPI-4 implementation with support for large-count (64bit) messages. Up to today, only MPICH v4.x supports MPI-4. And to take advantage of the new large-count support, you should use mpi4py from git master branch.

With the above implementation, is it still possible that the total size of the incoming data at the root is too large? 

Well, yes… the limit is what a 64bit signed integer can represent. But 2^63-1 is such a large number that you will run out of memory in the root node for the foreseeable future.

I will be gathering several million pairs of floats. Is there a way to know whether this will be a problem ex-ante?

Several millions should not be a problem as long as you have enough memory.

I suppose I could use a numpy array of object datatype, but I prefer the simplicity of lower case gather().

An ndarray with object datatype will not provide any benefit IMHO.

Perhaps you could build two arrays, one for keys and another for values, and use upper case comm.Gather() for each? Or even use a datatype for a pair key,value and a single array, thus gathering in a single call?

Lowercase methods are convenient, but they are slower and requires extra memory because of pickling. For large data, it may be worth to optimize and use the upper case methods with numpy arrays.

Regards,

Lisandro