calling external executor

[RITE Zhang]

Hi

I have encountered a problem for a long time.

In my python program using mpi4py, I need to spawn processes that call an external executor ('solver') as follows.

with open('log', 'wb') as outfile:

                p = subprocess.run(['./solver'], stdout=outfile, check=True) 

The problem is the code always going through the code, without really running the external executor. It gives the following error message.

subprocess.CalledProcessError: Command '['./solver']' returned non-zero exit status 15.

Rr if I use os.system('./solver'), it gets stuck

If not using mpi4py, for example, using multiprocessing, there is no problem. The solver itself does not use mpi.

I work under Linux.

Could anyone give a solution? Thank you.

[Lisandro Dalcin]

What backend MPI implementation are you using?

Some MPI implementations have problems with the fork() system call, and that my be used by Python's subprocess module to implement its functionalities.

--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpi4py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/f76b4342-ebe8-4590-b5e5-196fabcee8a2n%40googlegroups.com.

--

Lisandro Dalcin
============
Research Scientist
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/

[Yi Zhang]

Thank you. It seems mpich of conda.
It is a cluster with impi as default. I have tried to install mpi4py
with impi, however, always get failed, though I point mpicc to the
mpiicc of impi/bin.

> You received this message because you are subscribed to a topic in the Google Groups "mpi4py" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/mpi4py/oOKkWfqb19I/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to mpi4py+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/CAEcYPwC0wUycd5Upe65zxyVqdP9u2xWeud7_jBTbPth4zKraPw%40mail.gmail.com.

[Lisandro Dalcin]

On Wed, 2 Sep 2020 at 18:36, Yi Zhang <zhang...@gmail.com> wrote:

Thank you. It seems mpich of conda.

I'm not sure what's going on. Could you send a minimal self-contained example reproducing the problem?

 

It is a cluster with impi as default. I have tried to install mpi4py
with impi, however, always get failed, though I point mpicc to the
mpiicc of impi/bin.

Oh, do not use mpiicc, just use plan "mpicc" (which in turn uses GCC). You probably need this because your Python was built with GCC, and the Intel compiler cannot understand some of the flags Python's distutils set. In short, there is absolutely no need to use the Intel compiler to build mpi4py. Using GCC is just fine, and it will save you some trouble.

[Yi Zhang]

The problem is solved with your remind. Thank you.

> --
> You received this message because you are subscribed to a topic in the Google Groups "mpi4py" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/mpi4py/oOKkWfqb19I/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to mpi4py+un...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/CAEcYPwACvcPrZvdU1vCGSYcBaQBfhRCdFQ9WcwF8UqFQPjytTQ%40mail.gmail.com.