recv() doesn't start receiving an existing isend() request

[Prashant]

Hi,

I am trying to send a large chunk of data using the non-blocking
isend() function. On the receiving end, I expected it to start
receiving immediately on calling the recv() function as long as an
isend request is alive. However, this is not the case for large data,
it seems. A snippet of the code is pasted below:

# File mpitest.py begin
comm = MPI.COMM_WORLD
time0 = time.time()
reqlist = []
data = ['myid_particles:'+str(comm.rank)]*10000000
otherrank = 1 if comm.rank==0 else 0

reqlist += [comm.isend(data,dest=otherrank,tag=1)]
time1 = time.time() - time0
if comm.rank==1:
time.sleep(10)
time2 = time.time() - time0
a = comm.recv(source=otherrank,tag=1)
time3 = time.time() - time0

print str(comm.rank)+': send at t = '+str(time1)
print str(comm.rank)+': recv at t = ('+str(time2)+','+str(time3)+')'

MPI.Prequest.Waitall(reqlist)
# END


This code is run for 2 processors. The output for processor 0 is:
0: send at t = 2.36448693275
0: recv at t = (2.36450195312,13.0527698994)

and the output for processor 1 is:
1: send at t = 2.15210485458
1: recv at t = (12.1510648727,13.0352680683)

So, processor 0 waits till processor 1 starts another recv at around
12. seconds and then only receives the data.

This problem does not arise if the data to be transferred is of a
small size.

Is this behavior expected or is there some workaround? Thanks a lot.

[Lisandro Dalcin]

Yes, this behavior is expected. It should be related to the
implementation of the progress engine of your MPI. Could you try
change you code to perform a regular send BUT using a thread? Take a
look at the code in demo/threads
(http://code.google.com/p/mpi4py/source/browse/trunk/demo/threads/sendrecv.py)...
Note that you will need an MPI implementation that REALLY works safe
with threads (e.g. MPICH2 should do the job).

Below is my version, please improve it as your convenience.

# File mpitest.py begin

from mpi4py import MPI
import time

comm = MPI.COMM_WORLD

reqlist = []
data = ['myid_particles:'+str(comm.rank)]*10000000
otherrank = 1 if comm.rank==0 else 0

import threading
def _send():
global time0, time1
comm.send(data,dest=otherrank,tag=1)

time1 = time.time() - time0

send_thread = threading.Thread(target=_send)

time0 = time1 = time2 = time3 = 0
time0 = time.time()
send_thread.start()

if comm.rank==1:
time.sleep(10)
time2 = time.time() - time0
a = comm.recv(source=otherrank,tag=1)
time3 = time.time() - time0

send_thread.join()

print str(comm.rank)+': send at t = '+str(time1)
print str(comm.rank)+': recv at t = ('+str(time2)+','+str(time3)+')'

# END

--
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169