mpi4py with non-multithreaded OpenMPI

[Dag Sverre Seljebotn]

I'm trying to use mpi4py on a particular cluster and it's not going too
well. I think it has to do with threading: MPI.Query_thread() returns 0,
apparently *MPI_THREAD_SINGLE,* with the OpenMPI provided by the cluster.

Is mpi4py even supported in this mode? I mean, multithreading is used in
Python itself, right?

Eventually, things crashes with this message:

ERROR; return code from pthread_create() is 11
Error detail: Resource temporarily unavailable

I get this warning when starting. I don't think I or any of my libraries
call fork() directly, though threads are created. Are pthreads the culprit?

---
An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process. Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your MPI job may hang, crash, or produce silent
data corruption. The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

Local host: compute-25-30.local (PID 4098)
MPI_COMM_WORLD rank: 25

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
---

Thanks for any experiences with similar setups.

Dag Sverre

[Lisandro Dalcin]

On 13 August 2010 17:14, Dag Sverre Seljebotn

<da...@student.matnat.uio.no> wrote:
> I'm trying to use mpi4py on a particular cluster and it's not going too
> well. I think it has to do with threading: MPI.Query_thread() returns 0,
> apparently *MPI_THREAD_SINGLE,* with the OpenMPI provided by the cluster.
>

By default, mpi4py calls MPI_Init_thread() asking for
MPI_THREAD_MULTIPLE. Of course, MPI could not provide such level. This
is the case of Open MPI, when configured with default options, MPI
thread support is disabled, then the provided thread level will be
SINGLE(=0).

> Is mpi4py even supported in this mode? I mean, multithreading is used in
> Python itself, right?
>

Well, the actual question should be "Is MPI even supported in this
mode?" ... The answer is "no", still however things could work (GIL, I
love you!) unless your MPI implementation actively prevents you from
creating new threads.

> Eventually, things crashes with this message:
>
> ERROR; return code from pthread_create() is 11
>   Error detail: Resource temporarily unavailable
>
> I get this warning when starting. I don't think I or any of my libraries
> call fork() directly, though threads are created. Are pthreads the culprit?
>

Interesting... Perhaps Open MPI is behind you installing some pthread
hook to generate an error if a new thread is created? What Open MPI
version are you using?

> ---
> An MPI process has executed an operation involving a call to the
> "fork()" system call to create a child process.  Open MPI is currently
> operating in a condition that could result in memory corruption or
> other system errors; your MPI job may hang, crash, or produce silent
> data corruption.  The use of fork() (or system() or other calls that
> create child processes) is strongly discouraged.
> The process that invoked fork was:
>
>  Local host:          compute-25-30.local (PID 4098)
>  MPI_COMM_WORLD rank: 25
>
> If you are *absolutely sure* that your application will successfully
> and correctly survive a call to fork(), you may disable this warning
> by setting the mpi_warn_on_fork MCA parameter to 0.
> ---
>

Hold on... Are you 100% sure your code and libs does not call fork()
... Could you double check using strace ? BTW, Did you tried "mpiexec
-mca mpi_warn_on_fork 0 ...."

>
> Thanks for any experiences with similar setups.
>

You have two options:

1) Build Open MPI with MPI thread support explicitly enabled when you
./configure it.

2) Switch to MPICH2.

--
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

[Dag Sverre Seljebotn]

Lisandro Dalcin wrote:
> On 13 August 2010 17:14, Dag Sverre Seljebotn
> <da...@student.matnat.uio.no> wrote:
>
>> I'm trying to use mpi4py on a particular cluster and it's not going too
>> well. I think it has to do with threading: MPI.Query_thread() returns 0,
>> apparently *MPI_THREAD_SINGLE,* with the OpenMPI provided by the cluster.
>>
>>
>
> By default, mpi4py calls MPI_Init_thread() asking for
> MPI_THREAD_MULTIPLE. Of course, MPI could not provide such level. This
> is the case of Open MPI, when configured with default options, MPI
> thread support is disabled, then the provided thread level will be
> SINGLE(=0).
>
>> Is mpi4py even supported in this mode? I mean, multithreading is used in
>> Python itself, right?
>>
>>
>
> Well, the actual question should be "Is MPI even supported in this
> mode?" ... The answer is "no", still however things could work (GIL, I
> love you!) unless your MPI implementation actively prevents you from
> creating new threads.
>

Thanks for confirming my suspicions. I expected MPI_THREAD_FUNNELED in
this case, and half expected mpi4py to balk unless one had at least
MPI_THREAD_FUNNELED, however it appears that it's just OpenMPI reporting
wrongly according to some internet posts (what's the point of a standard
at all if you don't adhere to it...)

>> ---
>> An MPI process has executed an operation involving a call to the
>> "fork()" system call to create a child process. Open MPI is currently
>> operating in a condition that could result in memory corruption or
>> other system errors; your MPI job may hang, crash, or produce silent
>> data corruption. The use of fork() (or system() or other calls that
>> create child processes) is strongly discouraged.
>> The process that invoked fork was:
>>
>> Local host: compute-25-30.local (PID 4098)
>> MPI_COMM_WORLD rank: 25
>>
>> If you are *absolutely sure* that your application will successfully
>> and correctly survive a call to fork(), you may disable this warning
>> by setting the mpi_warn_on_fork MCA parameter to 0.
>> ---
>>
>>
>
> Hold on... Are you 100% sure your code and libs does not call fork()
> ... Could you double check using strace ? BTW, Did you tried "mpiexec
> -mca mpi_warn_on_fork 0 ...."
>

strace didn't show any fork calls... and disabling the warning did
nothing but disable the warning, but thanks!

Thanks; I'll just have to keep trying various things.

Dag Sverre

[Lisandro Dalcin]

On 14 August 2010 14:55, Dag Sverre Seljebotn

<da...@student.matnat.uio.no> wrote:
>
> Thanks for confirming my suspicions. I expected MPI_THREAD_FUNNELED in this
> case,

Indeed. MPI_THREAD_FUNNELED support is a reasonable expectation, but
still it is not trivial or immediate to support. Suppose the MPI
implementation uses dlopen() you dynamically load components at
runtime (e.g. Open MPI). Then any call to dlerror()&dlopen()&dlerror()
(the only sane way to actually check for errors) have to be protected
against possible usages of theses calls in other threads...

> and half expected mpi4py to balk unless one had at least
> MPI_THREAD_FUNNELED,

What should mpi4py balk about? you call
MPI_Init_thread(required,&provided), then if provided<required, then
it is user choice what to do next. Why should mpi4py do something
different? What if a user do not actually need thread support, because
her code does not use threads? See this demo:
http://code.google.com/p/mpi4py/source/browse/trunk/demo/nxtval/nxtval-threads.py#54,
IMHO, user code has to decide what to do if provided<required.

> however it appears that it's just OpenMPI reporting
> wrongly according to some internet posts (what's the point of a standard at
> all if you don't adhere to it...)
>

Could you drop a link? Why do you say Open MPI is not adhering to the standard?

[Dag Sverre Seljebotn]

Lisandro Dalcin wrote:
> On 14 August 2010 14:55, Dag Sverre Seljebotn
> <da...@student.matnat.uio.no> wrote:
>
>> Thanks for confirming my suspicions. I expected MPI_THREAD_FUNNELED in this
>> case,
>>
>
> Indeed. MPI_THREAD_FUNNELED support is a reasonable expectation, but
> still it is not trivial or immediate to support. Suppose the MPI
> implementation uses dlopen() you dynamically load components at
> runtime (e.g. Open MPI). Then any call to dlerror()&dlopen()&dlerror()
> (the only sane way to actually check for errors) have to be protected
> against possible usages of theses calls in other threads...
>

Ah, I see. So there's a wide range of program behavior between being
strictly SINGLE and needing full FUNNELED support, meaning Open MPI
perhaps cannot promise more than SINGLE but multi-threaded might still work.

OK, this finally makes sense to me then. I was wondering for a while
what the purpose of SINGLE was at all, or how an MPI implementation
could possibly be non-FUNNELED-compatible.

>
>> and half expected mpi4py to balk unless one had at least
>> MPI_THREAD_FUNNELED,
>>
>
> What should mpi4py balk about? you call
>

I'm sorry, I just assumed that the mere presence of threads in Python's
core (?) would itself mean one would never be compliant with a SINGLE
situation, but I was wrong.

> Could you drop a link? Why do you say Open MPI is not adhering to the standard?
>

http://www.open-mpi.org/community/lists/users/2007/05/3330.php seemed to
imply that OpenMPI did something wrong.

Anyway, thanks a lot for educating me, this all makes a lot more sense now.

Dag Sverre

[Lisandro Dalcin]

On 14 August 2010 15:52, Dag Sverre Seljebotn

<da...@student.matnat.uio.no> wrote:
>
> Ah, I see. So there's a wide range of program behavior between being
> strictly SINGLE and needing full FUNNELED support, meaning Open MPI perhaps
> cannot promise more than SINGLE but multi-threaded might still work.
>

Yet, with SINGLE support multithreading MIGHT work, but you do not
have any guarantee. You need at least FUNNELED to be safe about
spawning (non-MPI) threads.

> OK, this finally makes sense to me then. I was wondering for a while what
> the purpose of SINGLE was at all, or how an MPI implementation could
> possibly be non-FUNNELED-compatible.
>

You see, the infamous GIL make our minds think that multi-threading is
easier than what actually is.

>
> I'm sorry, I just assumed that the mere presence of threads in Python's core
> (?) would itself mean one would never be compliant with a SINGLE situation,
> but I was wrong.
>

Do you mean core Python supporting threads? To put it clear: If MPI
provides SINGLE support, then you have no guarantees that spawning a
new Python thread will work. Moreover, if an MPI implementation
actively disables thread creation by making pthread_create() fail, I
would not say such behavior is non-compliant with the MPI standard.