Gathering large arrays leads to SystemError

[sebastie...@gmail.com]

Hi,
while trying to gather large (32 x 512 x 512 x 6) arrays of 64 bit floats, I get the following error message

Execution time 5209.503359302878 s.
Traceback (most recent call last):
  File "simulation-PETSc.py", line 148, in <module>
    gathered = comm.gather(tau_loc, root)
  File "Comm.pyx", line 1266, in mpi4py.MPI.Comm.gather (src/mpi4py.MPI.c:101808
)
  File "msgpickle.pxi", line 664, in mpi4py.MPI.PyMPI_gather (src/mpi4py.MPI.c:4
3881)
  File "msgpickle.pxi", line 179, in mpi4py.MPI.Pickle.allocv (src/mpi4py.MPI.c:
38302)
  File "msgpickle.pxi", line 127, in mpi4py.MPI.Pickle.alloc (src/mpi4py.MPI.c:3
7601)
SystemError: Negative size passed to PyBytes_FromStringAndSize

I suspect it has something to do with 32bit int overflow, but I am not too sure.

Any Ideas?

Sébastien

[Lisandro Dalcin]

Indeed, it is likely a 32bits overflow. I would suggest to use the
uppercase spelling Gather() method instead to communicate such large
arrays. It is faster (no picking used under the hood), and you will
alleviate the overflow issues by a factor of 8. Eventually, for
messages with more of 2GB entries, you will need to split the
communication in two successive calls. Other approach is to employ
user defined datatypes (MPI.Datatype.Create_contiguous()) and use
these datatypes to perform the Gather() calls.

--
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169

[Sébastien Brisard]

Hi Lissandro,

Thanks for the suggestion. I didn't now about uppercase versions of Gather(). Not sure about the differences between the two, I might need to get back to you ;-). I'll make sure to test your solution by the end of the day, and keep you posted.
 

Eventually, for
messages with more of 2GB entries, you will need to split the
communication in two successive calls. Other approach is to employ
user defined datatypes (MPI.Datatype.Create_contiguous()) and use
these datatypes to perform the Gather() calls.

That's a very interesting suggestion. mpy4py seems to be very rich in useful functionalities. What would you suggest to discover them all (appart from reading the sources...).

[Lisandro Dalcin]

On 6 July 2014 10:36, Sébastien Brisard <sebastie...@gmail.com> wrote:
>
> Thanks for the suggestion. I didn't now about uppercase versions of
> Gather(). Not sure about the differences between the two, I might need to
> get back to you ;-). I'll make sure to test your solution by the end of the
> day, and keep you posted.
>

See here for a executive description (and some examples) on the
difference between the lower/uppercase versions
https://pythonhosted.org/mpi4py/usrman/tutorial.html

In short, the lower-case methods use pickle under the hood to
serialize/deserialize Python objects. This is good because of its
generality and simplicity, but you also have to pay for the pickle
overhead. The upper-case methods are almost direct calls to MPI,
communicating the memory buffer of array-like objects, so you have no
other overhead than the usual Python function call. Using the
upper-case version is slightly more involved (e.g, you need to
preallocate arrays to receive messages), but at the same time its
usage is more similar to how you would use MPI from C/C++/Fortran.

>>
>> Eventually, for
>> messages with more of 2GB entries, you will need to split the
>> communication in two successive calls. Other approach is to employ
>> user defined datatypes (MPI.Datatype.Create_contiguous()) and use
>> these datatypes to perform the Gather() calls.
>
>
> That's a very interesting suggestion. mpy4py seems to be very rich in useful
> functionalities. What would you suggest to discover them all (appart from
> reading the sources...).
>

Well, mpi4py supports almost all of MPI, and follows closely the MPI-2
C++ bindings. So if you look at the MPI standard itself or any other
book/tutorial about MPI, you should easily find the corresponding
functionality in mpi4py. For example, I mentioned
MPI.Datatype.Create_contiguous(), that's the Python version of the
C/Fortran call MPI_Type_contiguous() (the MPI-2 C++ bindings, it is
MPI::Datatype::Create_contiguous()). And of course, you can ask here
any time.

[Sébastien Brisard]

Hi Lisandro,

comm.Scatterv / comm.Gatherv is exactly what I needed. Actually, for my usecase, using these commands simplifies the script (because I'm scattering a large, contiguous ndarray along the first dimension). The computation is now running, we'll see in a few hours if everything went OK. Thanks again for your help, and your involvment in mpi4py/petsc4py.

Sébastien