Re: [mpi4py] 'mpi.h' file not found error upgrade mpi4py via pip

[Lisandro Dalcin]

On 5 October 2012 14:38, <neha...@gmail.com> wrote:
> Hi mpi4py team,
>
> First, thanks for the nice package.
>
> I recently upgraded to OS X Mountain Lion and have attempted to upgrade my
> installation of mpi4py via pip with:
> sudo pip install --upgrade mpi4py
>
> And this results in quite a bit of output (attached in pip.log) with many
> errors associated with:
> fatal error: 'mpi.h' file not found
>
> And installation fails with code 1, ending with:
> #include <mpi.h>
>
> ^
> 1 error generated.
>
> failure.
>
> removing: _configtest.c _configtest.o
>
> failure.
>
> error: Cannot find 'mpi.h' header. Check your configuration!!!
>
> ----------------------------------------
> Rolling back uninstall of mpi4py
> Command
> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
> -c "import
> setuptools;__file__='/tmp/pip-build/mpi4py/setup.py';exec(compile(open(__file__).read().replace('\r\n',
> '\n'), __file__, 'exec'))" install --record
> /tmp/pip-qt8hfl-record/install-record.txt
> --single-version-externally-managed failed with error code 1 in
> /tmp/pip-build/mpi4py
>
> However, mpi.h does exist as openmpi is installed (via macports) on the
> system with header files residing in directory at:
> [/opt/local/include/openmpi]
> 10:28:19-nehalecky@nico:openmpi $ pwd
> /opt/local/include/openmpi
>
> [/opt/local/include/openmpi]
> 10:28:22-nehalecky@nico:openmpi $ ls
> mpi-ext.h mpi_portable_platform.h mpif-config.h
> mpif.h vampirtrace/
> mpi.h mpif-common.h mpif-mpi-io.h
> openmpi/
>
> It seems other python packages (and their subsequent installers) have had to
> manage this, such as py-tables, as the default location for mpi headers has
> been changed to:
> ${prefix}/include/openmpi
>
> I thought it might be helpful to point this out so that the mpi4py installer
> can be updated accordingly. Also, if you have any tips to getting it
> upgraded, easily, in the meantime, I would appreciate it!
>
> Thanks again.
> Saludos,
> Nicholaus
>
> --
>
>

$ export PATH=/opt/local/bin
$ export LD_LIBRAY_PATH=/opt/local/lib
$ sudo -E pip install --upgrade mpi4py

PS: Please note that you will likely need to also set PATH and
LD_LIBRAY_PATH in your ~/.bashrc

--
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

[neha...@gmail.com]

Thanks for the reply.

The path to /opt/local/bin already existed in environment variable $PATH. Added env var $LD_LIBRAY_PATH=/opt/local/lib to ~/.profile and reran pip upgrade but am still receiving the same error. Emphasize that the file not being found is a header (mpi.h), and is located in /opt/local/include/openmpi  

Is their an environment variable that I can define to include this path in the installer search?

Regards.

[Lisandro Dalcin]

El 05/10/2012 17:01, <neha...@gmail.com> escribió:
>
> Thanks for the reply.
>
> The path to /opt/local/bin already existed in environment variable $PATH. Added env var $LD_LIBRAY_PATH=/opt/local/lib to ~/.profile and reran pip upgrade but am still receiving the same error. Emphasize that the file not being found is a header (mpi.h), and is located in /opt/local/include/openmpi  
> Is their an environment variable that I can define to include this path in the installer search?
>

Sorry. I got confused. Basically, you should add to PATH the directory containing the mpicc compiler wrapper, and LD_LIBRARY_PATH to the libmpi.dylib shared library.

Can you try this? If it fails please send us the output of 'mpicc -show'.

> --
>  
>  

[neha...@gmail.com]

Sorry. I got confused. Basically, you should add to PATH the directory containing the mpicc compiler wrapper, and LD_LIBRARY_PATH to the libmpi.dylib shared library. 

$PATH contains the directory to openmpi binary:

[/opt/local/bin]
11:20:15-nehalecky@nico:bin $ ll openmpi*
lrwxr-xr-x  1 root  admin    39B Oct  4 11:12 openmpic++@ -> /opt/local/lib/openmpi/bin/opal_wrapper
lrwxr-xr-x  1 root  admin    39B Oct  4 11:12 openmpicc@ -> /opt/local/lib/openmpi/bin/opal_wrapper
lrwxr-xr-x  1 root  admin    39B Oct  4 11:12 openmpicxx@ -> /opt/local/lib/openmpi/bin/opal_wrapper
lrwxr-xr-x  1 root  admin    34B Oct  4 11:12 openmpiexec@ -> /opt/local/lib/openmpi/bin/orterun
lrwxr-xr-x  1 root  admin    39B Oct  4 11:12 openmpif77@ -> /opt/local/lib/openmpi/bin/opal_wrapper
lrwxr-xr-x  1 root  admin    39B Oct  4 11:12 openmpif90@ -> /opt/local/lib/openmpi/bin/opal_wrapper
lrwxr-xr-x  1 root  admin    34B Oct  4 11:12 openmpirun@ -> /opt/local/lib/openmpi/bin/orterun

Also, now both env vars $LD_LIBRARY_PATH and $DYLD_LIBRARY_PATH contain the paths of both the libraries and headers/includes 

I am still receiving the same error:

 fatal error: 'mpi.h' file not found

 

Can you try this? If it fails please send us the output of 'mpicc -show'. 

Here is the output: 

11:22:14-nehalecky@nico:bin $ openmpicc -show
/usr/bin/clang -I/opt/local/include/openmpi -L/opt/local/lib -lmpi -lm

Thank you for taking the time in assisting me with this!

Regards.

[Lisandro Dalcin]

On 7 October 2012 15:27, <neha...@gmail.com> wrote:
>
> Here is the output:
>
> 11:22:14-nehalecky@nico:bin $ openmpicc -show
> /usr/bin/clang -I/opt/local/include/openmpi -L/opt/local/lib -lmpi -lm
>
> Thank you for taking the time in assisting me with this!
> Regards.
>

Oh! So the compiler wrapper is not "mpicc" but "openmpicc" ? This is
certainly not standard practice! Please try the following (I'm
assuming openmpicc is in your $PATH):

$ export MPICC=openmpicc

$ sudo -E pip install --upgrade mpi4py

[neha...@gmail.com]

Oh! So the compiler wrapper is not "mpicc" but "openmpicc" ? This is 
certainly not standard practice! Please try the following (I'm 
assuming openmpicc is in your $PATH):  

 

$ export  MPICC=openmpicc 
$ sudo -E pip install --upgrade mpi4py 

Lisandro, that worked! Thanks so much!

Just to note: "openmpicc" is the one of the two variants of mpi (along with mpih) that exist in the OS X package repository macports. I simply installed it via the package manage and the complier installed as "openmipcc", and indeed exists on my path as such. If this isn't standard practice, who might I inform that this is causing hiccups elsewhere with software that is dependent on mpicc? I'd be glad to try and inform them but perhaps you might have a little bit more clout being that you develop mpi4py? :) 

Thanks again, 

Nicholaus

[Lisandro Dalcin]

On 9 October 2012 13:28, <neha...@gmail.com> wrote:
>
>> Oh! So the compiler wrapper is not "mpicc" but "openmpicc" ? This is
>> certainly not standard practice! Please try the following (I'm
>> assuming openmpicc is in your $PATH):
>>
>>
>>
>> $ export MPICC=openmpicc
>> $ sudo -E pip install --upgrade mpi4py
>
>
> Lisandro, that worked! Thanks so much!
>
> Just to note: "openmpicc" is the one of the two variants of mpi (along with
> mpih) that exist in the OS X package repository macports. I simply installed
> it via the package manage and the complier installed as "openmipcc", and
> indeed exists on my path as such. If this isn't standard practice, who might
> I inform that this is causing hiccups elsewhere with software that is
> dependent on mpicc? I'd be glad to try and inform them but perhaps you might
> have a little bit more clout being that you develop mpi4py? :)
>

Well, if you install MPICH2 or OpenMPI from sources, both install
"mpicc" in $prefix/bin, so they conflicts. Perhaps MacPorts folks do
install as "openmpicc" in order to avoid such conflicts with other MPI
implementations. Is MPICH2 also available from MacPorts? (I'm not a
Mac user).

Perhaps I should update mpi4py to look for "openmpicc" in the next
release. I would need a little more info, can you help me? Basically,
I would need a listing of all the files that the MacPort package
installs. Is easy for you to get such information?

[neha...@gmail.com]

Well, if you install MPICH2 or OpenMPI from sources, both install 
"mpicc" in $prefix/bin, so they conflicts. Perhaps MacPorts folks do 
install as "openmpicc" in order to avoid such conflicts with other MPI 
implementations. Is MPICH2 also available from MacPorts? (I'm not a 
Mac user). 
Perhaps I should update mpi4py to look for "openmpicc" in the next 
release. I would need a little more info, can you help me? Basically, 
I would need a listing of all the files that the MacPort package 
installs. Is easy for you to get such information? 

Lisandro,

Attached is the output from the port contents command, which lists all files installed from the given port (quite a few). I didn't get to spend much time looking at them all (there are quite a few), however, I noticed a few .txt files with 'wrapper data' in the title that I though might be useful for you. They are attached in the .zip file. 

[~/Sites/notes.nicholaushalecky.com]

13:57:02-nehalecky@nico:notes.nicholaushalecky.com (master) $ port contents openmpi > openmpi-contents.out

Please let me know if you can use any other information.

[Lisandro Dalcin]

On 21 October 2012 18:50, <neha...@gmail.com> wrote:
> Attached is the output from the port contents command, which lists all files
> installed from the given port (quite a few). I didn't get to spend much time
> looking at them all (there are quite a few), however, I noticed a few .txt
> files with 'wrapper data' in the title that I though might be useful for
> you. They are attached in the .zip file.

That info helped a lot. So it seems "openmpicc" is the name of the
compiler wrapper you have to invoke to compile MPI programs. Now, try
to install mpi4py using pip like this:

$ MPICC=openmpicc sudo -E pip install --upgrade mpi4py

PS1: You really need to pass the -E flag to sudo, otherwise the MPICC
environ var will not be seen by the subprocess.

PS2: You will likely need to add the following line in your ~/.basrc file:
export DYLD_LIBRARY_PATH=/opt/local/lib:$DYLD_LIBRARY_PATH

Let me know if you have success.