Using mpi4py instead of multiprocessing

[Jiadun Liu]

To whom it may concern,

Currently I'm using multiprocessing to run independent simulation in different directories.

Now I want to run the program on more than one nodes. I think I have to use mpi4py.

The attached file is run as follows.

[bwjliu@int1 codeTBD]$ python Iso.py 

2

0

1

3

Is that possible that I use Spawn in Iso.py with keeping using python Iso.py in the terminal command line so that the independent program load.py will be spawn on more than one nodes?

Best regards,

Jiadun

[Jason Maldonis]

Hi Jiadun,

I believe I wrote some examples for almost exactly what you want to do.  Please see my examples here, and this one in particular.  I wrote these just a few weeks ago, so if they are helpful please let me know, and if they can be improved somehow I would appreciate any suggestions you have.

In order to specify different directories for your jobs, you will have to use mpi4py's "Info" object. This was mentioned in a recent thread or two.

Also, if you want to run on more than one node, then yes mpi4py is the best way to do that in my experience.

Best,

Jason

--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpi4py+unsubscribe@googlegroups.com.
To post to this group, send email to mpi...@googlegroups.com.
Visit this group at https://groups.google.com/group/mpi4py.
To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/CAMoctxcVgPmU9Egb2r6KTBFHEJ03UH%3Df8J4JJUN-oYU920k5EA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Jiadun Liu]

Hi Jason,

I will try it.

Best regards,

Jiadun

[Jiadun Liu]

Hi Jason,

It seems that the example spawn_split.py has syntax error.

bwjliu@bwjliu-HP-ZBook-15:~/mpiLearning/mpi-parallelization-master$ mpiexec -np 4  -oversubscribe  -mca btl tcp,sm,self  python spawn_multiple.py

  File "spawn_multiple.py", line 43

    args = [["spawn_multiple_worker.py"] + [*data] for data in args]

                                            ^

SyntaxError: invalid syntax

  File "spawn_multiple.py", line 43

    args = [["spawn_multiple_worker.py"] + [*data] for data in args]

                                            ^

SyntaxError: invalid syntax

  File "  File "spawn_multiple.py", line 43

    args = [["spawn_multiple_worker.py"] + [*data] for data in args]

                                            ^

SyntaxError: invalid syntax

spawn_multiple.py", line 43

    args = [["spawn_multiple_worker.py"] + [*data] for data in args]

                                            ^

SyntaxError: invalid syntax

-------------------------------------------------------

Primary job  terminated normally, but 1 process returned

a non-zero exit code.. Per user-direction, the job has been aborted.

-------------------------------------------------------

--------------------------------------------------------------------------

mpiexec detected that one or more processes exited with non-zero status, thus causing

the job to be terminated. The first process to do so was:

  Process name: [[11004,1],1]

  Exit code:    1

Best,

Jiadun

[Jason Maldonis]

Hi Jiadun,

My examples were written for Python 3.x+ (I'm not sure if they work on all versions of 3.x or not). My guess is that the start in [*data] is invalid syntax in older versions of Python. If you are already using Python 3, then you should just be able to upgrade to the latest version and everything will work.  If you are using Python 2, you could consider switching or you'll have to fix the examples for yourself.

Best,

Jason

--

You received this message because you are subscribed to the Google Groups "mpi4py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpi4py+unsubscribe@googlegroups.com.
To post to this group, send email to mpi...@googlegroups.com.
Visit this group at https://groups.google.com/group/mpi4py.

To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/305e102b-98c9-477f-9b08-6cb083e7d450%40googlegroups.com.