Attempting to use an MPI routine before initializing MPICH

[blo...@lbl.gov]

Experiencing the above error generated in wrapped C++ code. I can use MPI via mpi4py from Python but when I call my wrapped function I get the above and then the program aborts. I've used ldd and compared MPI.so to my .so files and all link to same mpi library. What else can cause this issue?

[Lisandro Dalcin]

What OS are you running on? Are you executing "from mpi4py import MPI"
before calling your wrapped C++ code ?


--
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 0109
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

[blo...@lbl.gov]

On Saturday, July 8, 2017 at 11:36:09 PM UTC-7, Lisandro Dalcin wrote:

On 8 July 2017 at 03:56,  <blo...@lbl.gov> wrote:
> Experiencing the above error generated in wrapped C++ code. I can use MPI
> via mpi4py from Python but when I call my wrapped function I get the above
> and then the program aborts. I've used ldd and compared MPI.so to my .so
> files and all link to same mpi library. What else can cause this issue?
>

What OS are you running on? Are you executing "from mpi4py import MPI"
before calling your wrapped C++ code ?

 

In the Python code I had been "from mpi4py import *"

This is on a Cray System, it's Linux but I think they have their own flavor. I stumbled upon the following work around, setting LD_PRELOAD to the MPI library I linked to. So it does seem some issue with dynamic linking. Unfortunately I am linking in MPI consistently so I don't really know why I am having the issue.

[Lisandro Dalcin]

Oh, be very carefull, maybe some parts of your code is linking MPI
statically, that would be VERY bad.

Instead of LD_PRELOAD, you could try the following (at the very
beginning of your script)

from mpi4py import MPI
from mpi4py import dl
lib = dl.dlopen("libmpi.so.1", dl.RTLD_GLOBAL|dl.RTLD_NOW|dl.RTLD_NOLOAD)
assert lib, dl.dlerror()

Of course, replace "libmpi.so.1" with the actual name of the library,
i.e. the one you used with LD_PRELOAD, or the one you get from ldd on
mpi4py's MPI.so extension module.

Anyway, this is just a workaround. I still think some of your modules
are linking MPI statically, that's the reason you see these errors.

[blo...@lbl.gov]

On Tuesday, July 11, 2017 at 3:01:48 AM UTC-7, Lisandro Dalcin wrote:

Anyway, this is just a workaround. I still think some of your modules
are linking MPI statically, that's the reason you see these errors.

Hey, that is a good point, static linking could be an issue. but I'm linking libraries dynamically, and in the build explicitly point to .so files with full paths. running under strace I was able to see that I had an old install of mpi4py in ~/.local (ref). It was causing the issues. I don't understand why it didn't cause problems before, as home folders are cross mounted on all of our systems and the particular install was compiled on a different system a number of years ago.

For future: 

strace was key to track this problem down, it showed both .so that were loaded as well as py/pyc. Anyone experience this issue should take a look at strace output.

[Lisandro Dalcin]

On 12 July 2017 at 20:34, <blo...@lbl.gov> wrote:
>
> For future:
> strace was key to track this problem down, it showed both .so that were
> loaded as well as py/pyc. Anyone experience this issue should take a look at
> strace output.
>

Though I acknowledge strace it the most powerful way to debug these
issues, you can also pass the "-v" flag to the Python interpreter and
it will print the full path of .so/.py/.pyc files for every "import"
statement in Python code.

[godenw...@gmail.com]

Hello, Burlen:

I have same problems as you mentioned on same type of system(https://bitbucket.org/mpi4py/mpi4py/issues/107/compiling-problem-for-swig-demo), maybe you could provide me more details about how to use strace to solve this problem.

Thanks a lot for your help.

Sincerely,

Zhe

在 2017年7月12日星期三 UTC-4下午1:34:12，Burlen Loring写道：