logging and MPIPoolExecutor

[Jan Kwakkel]

I hope this is the right place to ask this question.


I am trying to set up logging while using an MPIPoolExecutor. I found a few earlier discussions both here (i.e., https://groups.google.com/g/mpi4py/c/SaNzc8bdj6U) and on StackOverflow and Github (e.g., https://gist.github.com/sixy6e/ed35ea88ba0627e0f7dfdf115a3bf4d1). My understanding is that the best option would be to use MPI.File to write the log messages to a shared file. This works fine when not using a pool, but if I try the code below with mpiexec -n 4 python -m mpi4py.futures <module_name>.py, it just hangs. If I change the comm to COMM_SELF, it sort of works, but messages are missing.

Any suggestions on how to make this work would be welcome.


#!/usr/bin/env python

"""

"""
import mpi4py
from mpi4py import MPI
from mpi4py.futures import MPIPoolExecutor



def write_rank(i):
    comm = MPI.COMM_WORLD
    rank = comm.Get_rank()
    file = MPI.File.Open(comm, 'test.log',
                         MPI.MODE_WRONLY|MPI.MODE_CREATE)

    file.Write_shared(f"message {i} from {rank}\n")
    file.Sync()
    file.Close()

    return i


if __name__ == '__main__':
    n = 3
    with MPIPoolExecutor() as executor:
        # results = [r for r in ]
        # mpi_handler = MPIFileHandler("test.log")
        for result in executor.map(write_rank, range(n)):
            print(result)