displacement overflow during the Scatterv

[Zhihao Cui]

Dear all,

I recently encountered an overflow error when using Scatterv to distribute a large array.

I did make a segmentation of the array, but it seems the displacement arg will itself be overflowed:

    eri_mo = mpi.scatter(eri_sliced, root=0, data=eri_data)
  File "/home/zhcui/program/mpi4pyscf/mpi4pyscf/tools/mpi.py", line 273, in scatter
    comm.Scatterv([sendbuf, counts_seg, displs+p0, mpi_dtype],
  File "mpi4py/MPI/Comm.pyx", line 626, in mpi4py.MPI.Comm.Scatterv
  File "mpi4py/MPI/msgbuffer.pxi", line 538, in mpi4py.MPI._p_msg_cco.for_scatter
  File "mpi4py/MPI/msgbuffer.pxi", line 440, in mpi4py.MPI._p_msg_cco.for_cco_send
  File "mpi4py/MPI/msgbuffer.pxi", line 313, in mpi4py.MPI.message_vector
  File "mpi4py/MPI/asarray.pxi", line 22, in mpi4py.MPI.chkarray
  File "mpi4py/MPI/asarray.pxi", line 15, in mpi4py.MPI.getarray
OverflowError: value too large to convert to int

My wrapper is like the following:

INT_MAX = 2147483647
BLKSIZE = INT_MAX // 32 + 1

def prange(start, stop, step):
    nsteps = (stop - start + step - 1) // step
    nsteps = max(comm.allgather(nsteps))
    for i in range(nsteps):
        i0 = min(stop, start + i * step)
        i1 = min(stop, i0 + step)
        yield i0, i1

def scatter(sendbuf, root=0, data=None):
    if rank == root:
        mpi_dtype = numpy.result_type(*sendbuf).char
        shape = comm.scatter([x.shape for x in sendbuf])
        counts = numpy.asarray([x.size for x in sendbuf])
        comm.bcast((mpi_dtype, counts))
        if data is None:
            sendbuf = [numpy.asarray(x, mpi_dtype).ravel() for x in sendbuf]
            sendbuf = numpy.hstack(sendbuf)
        else:
            sendbuf = numpy.asarray(data, order='C')
    else:
        shape = comm.scatter(None)
        mpi_dtype, counts = comm.bcast(None)

    displs = numpy.append(0, numpy.cumsum(counts[:-1]))
    recvbuf = numpy.empty(numpy.prod(shape), dtype=mpi_dtype)
   
    #DONOT use lib.prange. lib.prange may terminate early in some processes
    for p0, p1 in prange(0, numpy.max(counts), BLKSIZE):
        counts_seg = _segment_counts(counts, p0, p1)
        comm.Scatterv([sendbuf, counts_seg, displs+p0, mpi_dtype],
                      [recvbuf[p0:p1], mpi_dtype], root)
    return recvbuf.reshape(shape)

It seems that the data is already chuncked, but the displs may be larger than INT size.

Is there a way to avoid that? Thanks!

Best,

Zhihao

[Ahmed Chabib]

Hi dear,  Zhihao Cui

how did you solve this issue? 

Best regards 

[Lisandro Dalcin]

Dear Ahmed,

What kind of data are you trying to communicate? Just NumPy arrays? Or Python classes with large arrays inside?

The definitive solution to this issue is to use mpi4py from the git repository master branch with an MPI-4.0 implementation.

MPI-4.0 added large-count versions of many routines allowing the specification of sizes beyond the 2GB limit of 32 bit integers.

Up to today, only MPICH v4.0.x supports the MPI-4.0 standard.

Alternatively, you could use the mpi4py.util.pkl5 module as a workaround.

mpi4py.util.pkl5 allows you to send efficiently (via pickle protocol 5) objects weighing more than 2GB.

If you only need point-to-point send/recv communication, then you can use mpi4py.util.pkl5 from latest mpi4py release 3.1.x.

If you need collectives like scatter/gather/allgather/alltoall, then you have to use mpi4py from git repository master branch. 

Please note that these collectives are all implemented with a simple loop doing send/recv.

Please have a glance to the latest (master) docs: https://mpi4py.readthedocs.io/en/latest/mpi4py.util.pkl5.html

Regards,

--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpi4py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/bc7cde8c-4966-4832-817b-7678bc195a33n%40googlegroups.com.

--

Lisandro Dalcin
============
Senior Research Scientist
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/