I hope this is the right place to ask this question.
I am trying to set up logging while using an MPIPoolExecutor. I found a few earlier discussions both here (i.e.,
https://groups.google.com/g/mpi4py/c/SaNzc8bdj6U) and on StackOverflow and Github (e.g.,
https://gist.github.com/sixy6e/ed35ea88ba0627e0f7dfdf115a3bf4d1). My understanding is that the best option would be to use MPI.File to write the log messages to a shared file. This works fine when not using a pool, but if I try the code below with mpiexec -n 4 python -m mpi4py.futures <module_name>.py, it just hangs. If I change the comm to COMM_SELF, it sort of works, but messages are missing.
Any suggestions on how to make this work would be welcome.
#!/usr/bin/env python
"""
"""
import mpi4py
from mpi4py import MPI
from mpi4py.futures import MPIPoolExecutor
def write_rank(i):
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
file = MPI.File.Open(comm, 'test.log',
MPI.MODE_WRONLY|MPI.MODE_CREATE)
file.Write_shared(f"message {i} from {rank}\n")
file.Sync()
file.Close()
return i
if __name__ == '__main__':
n = 3
with MPIPoolExecutor() as executor:
# results = [r for r in ]
# mpi_handler = MPIFileHandler("test.log")
for result in executor.map(write_rank, range(n)):
print(result)