Output synchronized with bcast?

[Wirawan Purwanto]

Hi all,

I am trying to understand whether my observation below is as expected. Here is a sample code, where I did bcast, then barrier:

from mpi4py import MPI

comm = MPI.COMM_WORLD

rank = comm.Get_rank()

size = comm.Get_size()

if rank == 0:

   message = "SENT_FROM_RANK_0"

else:

   message = "Empty"

print("Content of message in rank ", rank, " before broadcast: ", message)

message = comm.bcast(message, root=0)

comm.barrier()

print("Content of message in rank ", rank, " after broadcast:  ", message)

Without comm.barrier() up there, the order of "before" and "after" outputs are mixed: -- for 4 processes below:

Content of message in rank 2 before broadcast: Empty

Content of message in rank 3 before broadcast: Empty

Content of message in rank 0 before broadcast: SENT_FROM_RANK_0

Content of message in rank 0 after broadcast: SENT_FROM_RANK_0

Content of message in rank 2 after broadcast: SENT_FROM_RANK_0

Content of message in rank 1 before broadcast: Empty

Content of message in rank 3 after broadcast: SENT_FROM_RANK_0

Content of message in rank 1 after broadcast: SENT_FROM_RANK_0

But without the extra comm.barrier(), the program always prints all the "before" lines before all the "after":

Content of message in rank  0  before broadcast:  SENT_FROM_RANK_0

Content of message in rank  2  before broadcast:  Empty

Content of message in rank  3  before broadcast:  Empty

Content of message in rank  1  before broadcast:  Empty

Content of message in rank  3  after broadcast:   SENT_FROM_RANK_0

Content of message in rank  2  after broadcast:   SENT_FROM_RANK_0

Content of message in rank  0  after broadcast:   SENT_FROM_RANK_0

Content of message in rank  1  after broadcast:   SENT_FROM_RANK_0

I tried several times with a different number of ranks (2, 64, 128) -- always result in this kind of ordering. But a post somewhere else (https://stackoverflow.com/questions/5182045/openmpi-mpi-barrier-problems) seems to indicates that stdout is not synchronized across processes. I did no expect such a synchronization to be honest.

Out of curiosity I remade the program in C and ran it...and it did exactly the same thing. 

Can somebody explain what happened? Why does the bcast followed by barrier lead to synchronization of output?

Thanks,

Wirawan

[Wirawan Purwanto]

I need to clarify the implementation detail. I was using OpenMPI 3.1.4 for the underlying MPI, and mpi4py 3.0.3.

Wirawan

[Aamir Shafi]

Can you please clarify which output is with barrier() and which one is without barrier()? Your email says that both outputs are without barrier().

Regards,

--Aamir

--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpi4py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/f4bad641-a868-43e9-a438-698689ae9fc5n%40googlegroups.com.

[Lisandro Dalcin]

On Fri, 22 Jan 2021 at 02:23, Wirawan Purwanto <wira...@gmail.com> wrote:

Out of curiosity I remade the program in C and ran it...and it did exactly the same thing. 

Can somebody explain what happened? Why does the bcast followed by barrier lead to synchronization of output?

The MPI standard explicitly states that there is no guarantee in the order of standard output. 

This is all implementation dependent. An output is usually not synchronized.

A quick way to (attempt to) get synchronized standard output is the following:

https://github.com/mpi4py/mpi4py/blob/master/src/mpi4py/bench.py#L7

This code basically uses send/recv to serialize execution.

--

Lisandro Dalcin
============
Senior Research Scientist
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/

[Wirawan Purwanto]

Sorry for my confusing message. The second one is *with* the extra barrier:

But WITH the extra comm.barrier(), the program always prints all the "before" lines before all the "after":

Content of message in rank  0  before broadcast:  SENT_FROM_RANK_0

Content of message in rank  2  before broadcast:  Empty

Content of message in rank  3  before broadcast:  Empty

Content of message in rank  1  before broadcast:  Empty

Content of message in rank  3  after broadcast:   SENT_FROM_RANK_0

Content of message in rank  2  after broadcast:   SENT_FROM_RANK_0

Content of message in rank  0  after broadcast:   SENT_FROM_RANK_0

Content of message in rank  1  after broadcast:   SENT_FROM_RANK_0

But to Lisandro's point (thanks for confirming that!) this behavior is not in the formal spec of MPI, so it should not be relied upon. I did see some cases (in other program) where this (synchronization of output) indeed is NOT honored at the barrier().

Wirawan