Failure installing mpi4py using conda and specifying mpicc file

[Breno Vincenzo de Almeida]

Hello everyone,

I am  trying to install mpi4py using conda while specifying the path for my installed OpenMPI library, which I am using for PETSc, without conda installing its own mpich.

My OS is Ubuntu 18.0.4. I am using the latest Anaconda release (installed sometime this week).

The OpenMPI bin directory is set to the environment variable PATH.

And the libraries are set to LD_LIBRARY_PATH.

I tried installing mpi4py with:

env MPICC=path/to/ompi/bin/mpicc conda install -c anaconda mpi4py

 

But I get this message, before accepting/declining to continue:

The following NEW packages will be INSTALLED:
  mpi                anaconda/linux-64::mpi-1.0-mpich
  mpi4py             anaconda/linux-64::mpi4py-3.0.3-py37h028fd6f_0
  mpich              anaconda/linux-64::mpich-3.3.2-hc856adb_0

Which shows that the MPICC environment didn't work, since conda tries to install mpich.

Just to be sure, I had conda install mpi4py with the above configuration and then tried to run the following simple program, called mpi4pyTest.py:

from mpi4py import MPI
comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()
print('Hello from processor {} of {}'.format(rank+1,size))

I ran it with  mpirun -n 4 python mpi4pyTest.py   and the output was:

Hello from processor 1 of 1
Hello from processor 1 of 1
Hello from processor 1 of 1
Hello from processor 1 of 1

Which confirmed that mpi4py was not installed using my own mpi.

Is it possible to install mpi4py with Anaconda and link it to an existing MPI installation?

Thank you very much in advance.

Best regards,

Breno Vincenzo de Almeida

DMC FEM Unicamp

Antonio da Costa Santos Av.

13083-890 Campinas, SP - Brazil

Tel: +55 19 35213464

 www.fem.unicamp.br/~ltm

~~~~~~~~  |_-|-/\/\  ~~~~~~~~

[Lisandro Dalcin]

Conda packages are binary packages, they are not meant to be used in the way you want. What you are trying to do may be possible, but it would require some minor understanding on what conda packages are, how to conda install without dependencies, how shared libraries work, how to play tricks with LD_LIBRARY_PATH and maybe LD_PRELOAD, etc.

My advice would be: do not install mpi4py using conda, use instead "pip install mpi4py", which will install from the source release tarball on PyPI using the `mpicc` in your path.

--
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpi4py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/CANV_%3D0h3oF%2BSMFrmQh0Xod7YCi85Li8-FhHxNLiQEeav6bZ%2BmQ%40mail.gmail.com.

--

Lisandro Dalcin
============
Research Scientist
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/

[Leo Fang]

Hi Breno,

What you did doesn't make sense. The conda version of mpi4py sets either mpich or openmpi (both are also available on conda) as its runtime dependency, so when you do "conda install", either of them will also be installed. You should follow Lisandro's instruction and just do "pip install mpi4py" in you conda environment.

Best,

Leo 

Lisandro Dalcin於 2020年4月3日星期五 UTC-4上午7時12分33秒寫道：

To unsubscribe from this group and stop receiving emails from it, send an email to mpi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/CANV_%3D0h3oF%2BSMFrmQh0Xod7YCi85Li8-FhHxNLiQEeav6bZ%2BmQ%40mail.gmail.com.

[Breno Vincenzo de Almeida]

Hello,

I've managed to "fix" this issue.

I conda removed mpi4py, mpi and mpich. And just for good measure I conda removed petsc4py as well.

Next I installed mpi4py with:

env MPICC=path/to/openmpi/mpicc pip install --no-chache-dir mpi4py

I consider this a workaround, not an actual fix, because I couldn't install mpi4py with conda.

Now I'm having trouble with petsc4py, but that's a topic for a different forum.

Best regards,

Breno Vincenzo de Almeida

DMC FEM Unicamp

Antonio da Costa Santos Av.

13083-890 Campinas, SP - Brazil

Tel: +55 19 35213464

 www.fem.unicamp.br/~ltm

~~~~~~~~  |_-|-/\/\  ~~~~~~~~

[Leo Fang]

Glad to know it works (as it should), Breno. There's nothing to be "fixed" on the conda side, because this (not letting users worry about compiling) is how conda works.

Best,

Leo 

Breno Vincenzo de Almeida於 2020年4月3日星期五 UTC-4下午12時13分54秒寫道：

[Lisandro Dalcin]

On Fri, 3 Apr 2020 at 19:13, Breno Vincenzo de Almeida <balm...@fem.unicamp.br> wrote:

I consider this a workaround, not an actual fix, because I couldn't install mpi4py with conda.

As we stressed before, conda is a package manager for pre-built, BINARY packages. Would you consider a bug that if you yum/dnf/apt/brew install mpi4py, then you cannot use your own external MPI build?

[Chang Liao]

Hi,

I have had a similar issue earlier but I was able to fix it after some testing.

I was using mpi4py on an HPC cluster so the setup was slightly different. I hope my experience may help folks who use HPC instead of Ubuntu on a single machine.

Again, as discussed, the mpi4py should use system-wide mpicc instead of its own (and I wonder when we actually use the conda mpi.)

I used the following steps:

1. ssh into the cluster
2. load anaconda into system
3. create a conda environment for this task mpienv
4. activate the environment
5. load compiler, I used gcc
6. load openmpi
7. export MPICC="$(which mpicc)"
8. pip install mpi4py,  you should not use conde-forge channel because it will install mpi again.
9. test with python -c "import mpi4py"

I had provided some additional information here as well: https://code.changliao.us/2020/01/battle-11-mpi4py-on-cluster.html

Good luck!

Best Regards!

Chang Liao

Website: http://www.changliao.us

Skype: changliao1025

--

You received this message because you are subscribed to the Google Groups "mpi4py" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mpi4py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mpi4py/CANV_%3D0hkgNoHwaO3SwFxBLyvZKuOnRm07%2B35-zg6LirMUsX%3DOg%40mail.gmail.com.

[balm...@fem.unicamp.br]

Dear Lisandro Dalcin and Leo Fang,

I would like to apologize for not answering you. For some reason I did not receive any e-mail notifications of your answers. I have actually found this by chance right now, after googling mpi4py.

That "answer" I wrote, which is exactly what Lisandro instructed me to do, is what I did when someone else gave me the exact same advice in StackOverflow. I then passed on what I did via e-mail for completeness, not realizing this topic was opened in the mpi4py Google Group and that you had already answered me. In hindsight, I should have checked if/when the topic was opened. I am still curious why I didn't receive any e-mail notifications of your answers.

Python was (and still is) new to me - I'm still learning how to program anything past what I could write in Matlab, which was basically to solve linear electro-mechanical structural problems using the Finite Element Method - and I have relatively recently begun to understand the differences between pip and conda to install packages and how to play with LD_LIBRARY_PATH and LD_PRELOAD (because of Intel MKL). Now that I reread my question/comment I realize how dumb they sound! So sorry! Please disregard them!

Thus, I am sorry if I seemed pretentious with my comment. That was very far from my intention.

I intend to program more complicated structural problems in a cluster and may have new (hopefully not as dumb) questions. Please have patience with me! hahahah

I really appreciate your work. Thank you for your attention.

Best regards,

Breno