MPI parallelization in python to parallelize a "for loop" gives empty values
gogineni kailashnath
Hello, I want to parallelize the "for loop" iterations using OpenMP or MPI gather in python. The code is shown below. I also referred to the manual of MPI in python (MPI) and started with the code.
Here's my code:
Original code:
In the code, idxes is a list of 1024 random numbers. The for loop iterates over each element in the list of idxes 1024 times and retrieves the data stored in the self.storage and stores the information.
What I'm trying to do:
In my new implementation, I'm trying to make each process work on its local portion of idxes, collects the corresponding samples into local buffers, and converts the local buffers into numpy arrays. Then, the samples are gathered from all processes to the root process using the MPI.Comm.gather method. Finally, the gathered samples are concatenated into numpy arrays on the root process and returned.
What is the error:
When I try to print the lists, I observed that local_obses_t, local_actions, local_rewards, local_obses_tp1, local_dones have some data inside them, but somehow when I use gather to to gather all the samples from all the processes to the root process, I get None.
I'm not understanding the reason behind this bug. Can anyone help with it?
Command I used:
mpiexec -n 2 python program.py
The code works with mpiexec -n 1 python program.py, but does not work when I increase the value more than 1.
Any help would be appreciated.
I posted this problem on stack overflow: https://stackoverflow.com/questions/75989704/mpi-parallelization-in-python-to-parallelize-a-for-loop-gives-empty-values
Kailash
gogineni kailashnath
I want to parallelize the "for loop" iterations using OpenMP or MPI gather in python. The code is shown below. I also referred to the manual of MPI in python (MPI) and started with the code.
I posted my code and the error I get in stack overflow.