Output synchronized with bcast?

10 views
Skip to first unread message

Wirawan Purwanto

unread,
Jan 21, 2021, 6:23:25 PMJan 21
to mpi4py
Hi all,

I am trying to understand whether my observation below is as expected. Here is a sample code, where I did bcast, then barrier:

from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
if rank == 0:
   message = "SENT_FROM_RANK_0"
else:
   message = "Empty"
print("Content of message in rank ", rank, " before broadcast: ", message)
message = comm.bcast(message, root=0)
comm.barrier()
print("Content of message in rank ", rank, " after broadcast:  ", message)

Without comm.barrier() up there, the order of "before" and "after" outputs are mixed: -- for 4 processes below:

Content of message in rank 2 before broadcast: Empty
Content of message in rank 3 before broadcast: Empty
Content of message in rank 0 before broadcast: SENT_FROM_RANK_0
Content of message in rank 0 after broadcast: SENT_FROM_RANK_0
Content of message in rank 2 after broadcast: SENT_FROM_RANK_0
Content of message in rank 1 before broadcast: Empty
Content of message in rank 3 after broadcast: SENT_FROM_RANK_0
Content of message in rank 1 after broadcast: SENT_FROM_RANK_0

But without the extra comm.barrier(), the program always prints all the "before" lines before all the "after":

Content of message in rank  0  before broadcast:  SENT_FROM_RANK_0
Content of message in rank  2  before broadcast:  Empty
Content of message in rank  3  before broadcast:  Empty
Content of message in rank  1  before broadcast:  Empty
Content of message in rank  3  after broadcast:   SENT_FROM_RANK_0
Content of message in rank  2  after broadcast:   SENT_FROM_RANK_0
Content of message in rank  0  after broadcast:   SENT_FROM_RANK_0
Content of message in rank  1  after broadcast:   SENT_FROM_RANK_0

I tried several times with a different number of ranks (2, 64, 128) -- always result in this kind of ordering. But a post somewhere else (https://stackoverflow.com/questions/5182045/openmpi-mpi-barrier-problems) seems to indicates that stdout is not synchronized across processes. I did no expect such a synchronization to be honest.

Out of curiosity I remade the program in C and ran it...and it did exactly the same thing. 

Can somebody explain what happened? Why does the bcast followed by barrier lead to synchronization of output?

Thanks,
Wirawan

Wirawan Purwanto

unread,
Jan 21, 2021, 6:25:00 PMJan 21
to mpi4py
I need to clarify the implementation detail. I was using OpenMPI 3.1.4 for the underlying MPI, and mpi4py 3.0.3.

Wirawan

Aamir Shafi

unread,
Jan 21, 2021, 9:36:48 PMJan 21
to mpi...@googlegroups.com
Can you please clarify which output is with barrier() and which one is without barrier()? Your email says that both outputs are without barrier().

Regards,
--Aamir

--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpi4py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/f4bad641-a868-43e9-a438-698689ae9fc5n%40googlegroups.com.

Lisandro Dalcin

unread,
Jan 22, 2021, 3:52:48 AMJan 22
to mpi...@googlegroups.com
On Fri, 22 Jan 2021 at 02:23, Wirawan Purwanto <wira...@gmail.com> wrote:

Out of curiosity I remade the program in C and ran it...and it did exactly the same thing. 

Can somebody explain what happened? Why does the bcast followed by barrier lead to synchronization of output?


The MPI standard explicitly states that there is no guarantee in the order of standard output. 
This is all implementation dependent. An output is usually not synchronized.

A quick way to (attempt to) get synchronized standard output is the following:
This code basically uses send/recv to serialize execution.

--
Lisandro Dalcin
============
Senior Research Scientist
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/

Wirawan Purwanto

unread,
Jan 22, 2021, 3:02:19 PMJan 22
to mpi4py
Sorry for my confusing message. The second one is *with* the extra barrier:

But WITH the extra comm.barrier(), the program always prints all the "before" lines before all the "after":

Content of message in rank  0  before broadcast:  SENT_FROM_RANK_0
Content of message in rank  2  before broadcast:  Empty
Content of message in rank  3  before broadcast:  Empty
Content of message in rank  1  before broadcast:  Empty
Content of message in rank  3  after broadcast:   SENT_FROM_RANK_0
Content of message in rank  2  after broadcast:   SENT_FROM_RANK_0
Content of message in rank  0  after broadcast:   SENT_FROM_RANK_0
Content of message in rank  1  after broadcast:   SENT_FROM_RANK_0

But to Lisandro's point (thanks for confirming that!) this behavior is not in the formal spec of MPI, so it should not be relied upon. I did see some cases (in other program) where this (synchronization of output) indeed is NOT honored at the barrier().

Wirawan
Reply all
Reply to author
Forward
0 new messages