Irecv and strings
Vincent FR
still discovering mpi4py, great job!
I have a very basical question: I would like to do non blocking
receive with STRING messages (size below 256 characters).
I was about to use "irecv" but it does not exist.
Then, reading this forum and the post "How to use Irecv and Waitany?",
I switch to "Irecv" but actually I can't figure out what is the best
practice to adopt: converting string to array? Create_contiguous or
something like this (painful)?
Any suggestion would be very welcome!
Thanks in advance,
Vincent
Lisandro Dalcin
> Hi,
>
> still discovering mpi4py, great job!
>
> I have a very basical question: I would like to do non blocking
> receive with STRING messages (size below 256 characters).
> I was about to use "irecv" but it does not exist.
There is not easy way to implement 'irecv' for Python objects.
> Then, reading this forum and the post "How to use Irecv and Waitany?",
> I switch to "Irecv" but actually I can't figure out what is the best
> practice to adopt: converting string to array? Create_contiguous or
> something like this (painful)?
>
In Python, 'str' objects are immutable, so you cannot receive a
message on a string, you have to use an array instead. Take a look at
the example below:
from mpi4py import MPI
from array import array
comm = MPI.COMM_WORLD
if comm.rank == 0:
comm.Send("hello",dest=1)
elif comm.rank == 1:
buf = array('c', '\0') * 256
r = comm.Irecv(buf,source=0)
status = MPI.Status()
r.Wait(status)
n = status.Get_count(MPI.CHAR)
s = buf[:n].tostring()
print s
--
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169
Vincent FR
many thanks for this nice answer!
I don't especially understand how it works (I mean, passing a string
to Send, but not an array... and receiving an aray), but it is
perfectly working.
Regards,
Vincent
On 4 août, 16:45, Lisandro Dalcin <dalc...@gmail.com> wrote:
Lisandro Dalcin
> Dear Lisandro,
>
> many thanks for this nice answer!
> I don't especially understand how it works (I mean, passing a string
> to Send, but not an array... and receiving an aray), but it is
> perfectly working.
> Regards,
>
All this happens through Python's buffer interface. You can send a
string because it exposes it's underlying memory buffer as READONLY.
For Send(), that's all mpi4py needs. However, for Recv() you cannot,
as the buffer is readonly (if you try, mpi4py will generate an error).
Then, you have to create an "array.array" instance (these objects are
mutable, then their memory buffers are READ/WRITE) to receive the
message. Also, you need to figure out the actual message size using
"status".
Also note that this trick also works for pickled communication of
Python objects, so you code yourself something like the missing
"irecv()" method:
from mpi4py import MPI
from array import array
import pickle
comm = MPI.COMM_WORLD
if comm.rank == 0:
msg = {'a': 1, 'b':2, 'c': [1,2,3]}
comm.send(msg,dest=1)
elif comm.rank == 1:
buf = array('c', '\0') * 1024
r = comm.Irecv(buf,source=0)
status = MPI.Status()
r.Wait(status)
n = status.Get_count(MPI.CHAR)
s = buf[:n].tostring()
print pickle.loads(s)
Matthew Turk
On Fri, Aug 5, 2011 at 10:42 AM, Lisandro Dalcin <dal...@gmail.com> wrote:
> On 5 August 2011 08:45, Vincent FR <vincen...@univ-lyon1.fr> wrote:
>> Dear Lisandro,
>>
>> many thanks for this nice answer!
>> I don't especially understand how it works (I mean, passing a string
>> to Send, but not an array... and receiving an aray), but it is
>> perfectly working.
>> Regards,
>>
>
> All this happens through Python's buffer interface. You can send a
> string because it exposes it's underlying memory buffer as READONLY.
> For Send(), that's all mpi4py needs. However, for Recv() you cannot,
> as the buffer is readonly (if you try, mpi4py will generate an error).
> Then, you have to create an "array.array" instance (these objects are
> mutable, then their memory buffers are READ/WRITE) to receive the
> message. Also, you need to figure out the actual message size using
> "status".
>
> Also note that this trick also works for pickled communication of
> Python objects, so you code yourself something like the missing
> "irecv()" method:
This is a very cool trick, certainly very useful when sending small
objects back and forth. When one usually sends (implicitly pickled)
objects, how does mpi4py set up the buffers on the receive end? Does
it do this through a two-step communication, or is it somehow
communicated in a header or status object? Would your method (even
applied to blocking send/recv) reduce the overall time of
communication, when sending just very short objects back and forth?
Thanks,
Matt
>
>
> from mpi4py import MPI
> from array import array
> import pickle
>
> comm = MPI.COMM_WORLD
> if comm.rank == 0:
> msg = {'a': 1, 'b':2, 'c': [1,2,3]}
> comm.send(msg,dest=1)
> elif comm.rank == 1:
> buf = array('c', '\0') * 1024
> r = comm.Irecv(buf,source=0)
> status = MPI.Status()
> r.Wait(status)
> n = status.Get_count(MPI.CHAR)
> s = buf[:n].tostring()
> print pickle.loads(s)
>
>
>
>
> --
> Lisandro Dalcin
> ---------------
> CIMEC (INTEC/CONICET-UNL)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> 3000 Santa Fe, Argentina
> Tel: +54-342-4511594 (ext 1011)
> Tel/Fax: +54-342-4511169
>
> --
> You received this message because you are subscribed to the Google Groups "mpi4py" group.
> To post to this group, send email to mpi...@googlegroups.com.
> To unsubscribe from this group, send email to mpi4py+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mpi4py?hl=en.
>
>
Lisandro Dalcin
> Hi Lisandro,
>
> On Fri, Aug 5, 2011 at 10:42 AM, Lisandro Dalcin <dal...@gmail.com> wrote:
>> On 5 August 2011 08:45, Vincent FR <vincen...@univ-lyon1.fr> wrote:
>>> Dear Lisandro,
>>>
>>> many thanks for this nice answer!
>>> I don't especially understand how it works (I mean, passing a string
>>> to Send, but not an array... and receiving an aray), but it is
>>> perfectly working.
>>> Regards,
>>>
>>
>> All this happens through Python's buffer interface. You can send a
>> string because it exposes it's underlying memory buffer as READONLY.
>> For Send(), that's all mpi4py needs. However, for Recv() you cannot,
>> as the buffer is readonly (if you try, mpi4py will generate an error).
>> Then, you have to create an "array.array" instance (these objects are
>> mutable, then their memory buffers are READ/WRITE) to receive the
>> message. Also, you need to figure out the actual message size using
>> "status".
>>
>> Also note that this trick also works for pickled communication of
>> Python objects, so you code yourself something like the missing
>> "irecv()" method:
>
> This is a very cool trick, certainly very useful when sending small
> objects back and forth. When one usually sends (implicitly pickled)
> objects, how does mpi4py set up the buffers on the receive end?
Here is the code:
http://code.google.com/p/mpi4py/source/browse/trunk/src/MPI/pickled.pxi#182
It is based on MPI_Probe() and MPI_Get_count(), followed by the actual
MPI_Recv() call. The Probe() blocks waiting for a matching message to
arrive, and Get_count() lets you preallocate the buffer required to
receive the message.
> Does
> it do this through a two-step communication, or is it somehow
> communicated in a header or status object?
A single message involved. This is the only way to support thread
safety. Additionally, this makes Send()/send() and Recv()/recv()
somewhat compatible (as in my previous example).
> Would your method (even
> applied to blocking send/recv) reduce the overall time of
> communication, when sending just very short objects back and forth?
>
No, not at all. Of course, if you have to send the same object to many
processes (but not all, otherwise you use bcast()), you can pickle
once and Send() to many. Also, using Irecv() can be better if you have
to receive many messages from different sources, as you are not
enforcing a specific communication order and then MPI has more freedom
to make progress.
PS: I'm still thinking on ways of support "irecv()" for pickled
objects. Once I have something more refined, I'll comment on that
here.