how to get mpicc (or alternatively, build mpi4py) on windows

[Francois]

Hello,

I am trying to build mpi4py on windows, but I can't find any mpi C
compiler on windows. After installing MPICH2, I do not get any mpicc
in the bin directory, and I can barely find topics on internet that
mention mpicc on windows.
If mpicc indeed does not exist on windows, is there an alternate way
of building and installing mpi4py on windows ?

Any help is greatly appreciated.
Best,

Francois

------------------------
Francois Monard
PhD student in Applied Mathematics, Columbia University
www.columbia.edu/~fm2234

[Lisandro Dalcin]

On Wed, Nov 11, 2009 at 7:27 PM, Francois <francoi...@gmail.com> wrote:
>
> Hello,
>

Hi, Francois.

Disclaimer: my Windows expertise is almost NULL. Any
comments/suggestions from your Windows-user perspective will be highly
appreciated.

> I am trying to build mpi4py on windows, but I can't find any mpi C
> compiler on windows. After installing MPICH2, I do not get any mpicc
> in the bin directory, and I can barely find topics on internet that
> mention mpicc on windows.
>

These 'mpicc' compiler wrappers are not available on Windows, then we
have to "manually" pass include dirs, libraries, and library dirs...
And you should be able to use either MSVC or MinGW ...

> If mpicc indeed does not exist on windows, is there an alternate way
> of building and installing mpi4py on windows ?

Of course, and it is more or less straightforward, as described below...

>
> Any help is greatly appreciated.
> Best,
>

Look at the mpi.cfg file, it should have a section like this:

[mpich2_win]
mpi_dir = $ProgramFiles\MPICH2
include_dirs = %(mpi_dir)s\include
libraries = mpi
library_dirs = %(mpi_dir)s\lib

If you installed MPICH2 in the standard location (i.e, "C:\Program
Files"), that's enough... if not, you have to adjust it...

BTW, this is documented at
http://mpi4py.scipy.org/docs/usrman/install.html#building . This way
is rather general and OS-agnostic, perhaps these docs should have a
more specific instructions for POSIX / OS X / Windows. Contributions
welcome!

Now, all you need to do:

$ python setup.py build --mpi=mpich2_win

After that, you could install system-wide:

$ python setup.py install

or install just for your user if you have Python 2.6

$ python setup.py install --user

of perhaps make a binary MSI installer to make it easy to install
mpi4py in other machines/nodes using 'msiexec' in a batch file (never
used it, google for it):

$ python setup.py bdist_msi

of make an older-format, legacy binary installer:

$ python setup.py bdist_wininst

I think is easy enough, and you have fairly many options for the install step.

PS: In case of trouble, keep comming.

Regards

--
Lisandro Dalcín
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

[francois monard]

Thank you very much for the quick reply, I was able to build and install it. 

Apologies for not looking a little further into the mpi.cfg file, I got stuck on that mpicc idea, it was indeed fairly simple to build...

as far as windows is concerned, I will contribute if I have some relevant comments. However I never really took part in such a discussion group, so if you tell me what kind of questions are of interest, I will provide what I know on the selected topics. 

Thanks again,

Best regards,

Francois

--
------------------------------------------------------------
Francois Monard
------------------------------------------------------------

[Lisandro Dalcin]

On Wed, Nov 11, 2009 at 8:36 PM, francois monard
<francoi...@gmail.com> wrote:
> Thank you very much for the quick reply, I was able to build and install
> it.
> Apologies for not looking a little further into the mpi.cfg file, I got
> stuck on that mpicc idea, it was indeed fairly simple to build...
> as far as windows is concerned, I will contribute if I have some relevant
> comments.

OK.

> However I never really took part in such a discussion group, so if
> you tell me what kind of questions are of interest, I will provide what I
> know on the selected topics.

Well, if at some point you find anything that is annoying or complex
from the Windows-user perspective, just come back. Then we could start
a specific 'Windows issues' section or something like that in the
documentation. Right now, it seems all is working just fine for you...
So I would say that you have to use the beast a little.

In short, I think mpi4py is lacking some serious, production-level
usage on Windows; I just run the testsuite from time to time, in a
VirtualBox'ed XP and a DualCore Dell notebook. For example, if you can
do any testing (like just running your own code) on an networked
environment (I mean, a cluster of Windows machines), that would by
itself be a VERY useful contribution.

Regards,

[francois monard]

Hi Lisandro, I have a couple remarks:

I ran the runalltests.py file and passed all tests but one, the error message is given below :

>> Running test_exceptions

>> ....................................F........

>> ======================================================================

>> FAIL: testGetStatus (test_exceptions.TestExcRequestNull)

>> ----------------------------------------------------------------------

>> Traceback (most recent call last):

>>   File "test_exceptions.py", line 65, in testGetStatus

>>     self.assertRaisesMPI(MPI.ERR_REQUEST, MPI.REQUEST_NULL.Get_status)

>>   File "mpiunittest.py", line 35, in assertRaisesMPI

>>     raise self.failureException("%s not raised" % excName)

>> AssertionError: Exception not raised

>> 

>> ----------------------------------------------------------------------

>> Ran 45 tests in 0.006s

>> 

>> FAILED (failures=1)

Also, when running the mandelbrot demo, the plt.show() never displays. If I take out the alarm signal part of the code (for which I get the error "'module' object has no attribute 'SIGALRM'", although I checked that the <signal.h> library has that variable in it, so I don't know what's going on here), the process simply hangs. I run on cygwin, for which when I run sequential python code, the figures appear with no problem. 

My machine is an Intel quad core with vista 32bit on it. I see there are a few issues discussed on the web about how vista has a pretty tightened security, and I had to disable some administrator things in the user settings in order to get mpiexec to be able to connect to my computer. Could that be a similar issue ?

Thanks a lot,

Francois

[Lisandro Dalcin]

On Thu, Nov 12, 2009 at 1:34 PM, francois monard
<francoi...@gmail.com> wrote:
> Hi Lisandro, I have a couple remarks:
> I ran the runalltests.py file and passed all tests but one, the error
> message is given below :
>>> Running test_exceptions
>>> ....................................F........
>>> ======================================================================
>>> FAIL: testGetStatus (test_exceptions.TestExcRequestNull)
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>   File "test_exceptions.py", line 65, in testGetStatus
>>>     self.assertRaisesMPI(MPI.ERR_REQUEST, MPI.REQUEST_NULL.Get_status)
>>>   File "mpiunittest.py", line 35, in assertRaisesMPI
>>>     raise self.failureException("%s not raised" % excName)
>>> AssertionError: Exception not raised
>>>
>>> ----------------------------------------------------------------------
>>> Ran 45 tests in 0.006s
>>>
>>> FAILED (failures=1)

Ir you are using latest MPICH2 1.2.0, this is expected. The behavior
of MPI_Request_get_status() with MPI_REQUEST_NULL changed in recent
MPI-2.2 specification, and MPICH2 1.2.0 is up-to-date with the latest
MPI-2.2 std.

The latest mpi4py release IS NOT updated for MPI-2.2, though the -dev
code in SVN at Google Code DO IS updated.

> Also, when running the mandelbrot demo, the plt.show() never displays. If I
> take out the alarm signal part of the code (for which I get the error
> "'module' object has no attribute 'SIGALRM'", although I checked that the
> <signal.h> library has that variable in it, so I don't know what's going on
> here), the process simply hangs.

Oh, yes. Remove all that alarm stuff... it is there just for making
'make -C demo' work nicely in my Linux box... And if you cannot make
the plot appear, I doubt I can help you with that :-(

perhaps the problem is related to this line:

if rank != 0: raise SystemExit

could you try to modify the code to make the plot only if rank==0, and
add a comm.Barrier() at the very end of the script?

> I run on cygwin, for which when I run
> sequential python code, the figures appear with no problem.
>

Mmm.. then yes, it should be a issue with raise SystemExit and the
automatic call to MPI_Finalize()... Could you try my previous
suggestion?

> My machine is an Intel quad core with vista 32bit on it. I see there are a
> few issues discussed on the web about how vista has a pretty tightened
> security, and I had to disable some administrator things in the user
> settings in order to get mpiexec to be able to connect to my computer.

I have exactly the same problem in a Dell notebook at home with Vista.
My user has admin rights, but I cannot make all work out of the box.
First I have to open a console with admin privs, next execute "smpd
-stop", next 'smpd -start'... After that, I'm able to use mpiexec...

I've decided to NOT disable security stuff in Vista, just because I do
not trust MS a single bit, Win always had and have a lot of holes, and
I do not want to get my system hacked when my girlfriend visit a
random website or click on an "Accept" button she should not have to
:-)...

>
> Could that be a similar issue ?
>

I really have no idea, sorry...

[francois monard]

I tried using the following syntax, but in vain. 

if rank==0:

    M = np.zeros([h,w], dtype='i')

    M[indices, :] = cdata

    

    from matplotlib import pyplot as plt

    plt.figure(rank)

    plt.imshow(M, aspect='equal')

    plt.spectral()

    plt.show()

comm.Barrier()

When I abort manually, the three other processes do not say anything, while proc 0 mentions 'mpiexec aborting job'... When I used raise SystemExit, it would do the same thing. 

Did the plots work on your laptop with vista ?

 

> My machine is an Intel quad core with vista 32bit on it. I see there are a
> few issues discussed on the web about how vista has a pretty tightened
> security, and I had to disable some administrator things in the user
> settings in order to get mpiexec to be able to connect to my computer.

I have exactly the same problem in a Dell notebook at home with Vista.
My user has admin rights, but I cannot make all work out of the box.
First I have to open a console with admin privs, next execute "smpd
-stop", next 'smpd -start'... After that, I'm able to use mpiexec...

I've decided to NOT disable security stuff in Vista, just because I do
not trust MS a single bit, Win always had and have a lot of holes, and
I do not want to get my system hacked when my girlfriend visit a
random website or click on an "Accept" button she should not have to
:-)...

I see what you mean, I don't trust MS either and often wonder why I am trying so hard to make Python work out on Windows... I am just a little too accustomed to it I guess... As far as security, all I disabled was the screen going dark all the time in order to ask the administrator (i.e. the same person !) if he agrees with the user's decision; I believe softwares that are not completely up-to-date with win's new security barriers might not do it and therefore might hang.... however the firewall is still turned on for sure. 

 

>
> Could that be a similar issue ?
>

I really have no idea, sorry...

thanks a lot for the support. I'll keep looking at it and will let you know if I find out anything. 

[francois monard]

is there anything else than python.exe that is used for displaying figures, i.e. another executable that would require authorization from the firewall ?

[Lisandro Dalcin]

On Thu, Nov 12, 2009 at 2:41 PM, francois monard
<francoi...@gmail.com> wrote:
>
>> > I run on cygwin, for which when I run
>> > sequential python code, the figures appear with no problem.
>> >

OK.. Wait a minute... I never ever tried mpi4py inside cygwin.

>>
>> Mmm.. then yes, it should be a issue with raise SystemExit and the
>> automatic call to MPI_Finalize()... Could you try my previous
>> suggestion?
>>
> I tried using the following syntax, but in vain.
> if rank==0:
>     M = np.zeros([h,w], dtype='i')
>     M[indices, :] = cdata
>
>     from matplotlib import pyplot as plt
>     plt.figure(rank)
>     plt.imshow(M, aspect='equal')
>     plt.spectral()
>     plt.show()
> comm.Barrier()
> When I abort manually, the three other processes do not say anything, while
> proc 0 mentions 'mpiexec aborting job'... When I used raise SystemExit, it
> would do the same thing.

Mmm... this is very strange... Can plot in sequential outside cygwin?

>
> Did the plots work on your laptop with vista ?
>

I never made any plotting with matplotlib on XP/Vista, whatever WinDog...

>
> As far as security, all I disabled was the
> screen going dark all the time in order to ask the administrator (i.e. the
> same person !) if he agrees with the user's decision; I believe softwares
> that are not completely up-to-date with win's new security barriers might
> not do it and therefore might hang.... however the firewall is still turned
> on for sure.
>

Mmm... I still fail to see you issues related to a security problem...
Once mpiexec started the parallel run and the first MPI communication
call success, all should work.

[Lisandro Dalcin]

On Thu, Nov 12, 2009 at 2:48 PM, francois monard
<francoi...@gmail.com> wrote:
> is there anything else than python.exe that is used for displaying figures,
> i.e. another executable that would require authorization from the firewall ?
>

AFAIK, python.exe should be the only executable involved... Just in
case, you could try authorize any other exe you may have in
C:\Python2X

[francois monard]

indeed, this is probably not a security problem since the process still hangs even when I turn the firewall off. 

By the way, I also installed everything on my laptop with XP (mingw/cygwin/mpich2/mpi4py) and obtain the same issue, namely matplotlib.pyplot.show() will hang indefinitely and figures won't popup.

any ideas, besides throwing away all Windows, are welcome :)

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "mpi4py" group.
To post to this group, send email to mpi...@googlegroups.com
To unsubscribe from this group, send email to mpi4py+un...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/mpi4py?hl=en
-~----------~----~----~----~------~----~------~--~---

[Lisandro Dalcin]

On Mon, Nov 16, 2009 at 9:05 PM, francois monard
<francoi...@gmail.com> wrote:
> indeed, this is probably not a security problem since the process still
> hangs even when I turn the firewall off.
> By the way, I also installed everything on my laptop with XP
> (mingw/cygwin/mpich2/mpi4py) and obtain the same issue, namely
> matplotlib.pyplot.show() will hang indefinitely and figures won't popup.

OK. Confirmed. When launching with mpiexec (even with -n 1), the plot
never appears, the process just hangs and never finish.

>
> any ideas, besides throwing away all Windows, are welcome :)
>

You just banned the only suggestion I had in mind :-)

More seriously... Do you really need this working? I'm starting to
suspect that this may be an issue related to MPICH2 not supporting
threads, and matplotlib do requiring them...

> --

>
> You received this message because you are subscribed to the Google Groups
> "mpi4py" group.

> To post to this group, send email to mpi...@googlegroups.com.

> To unsubscribe from this group, send email to

> mpi4py+un...@googlegroups.com.

> For more options, visit this group at

> http://groups.google.com/group/mpi4py?hl=.

[Lisandro Dalcin]

On 16 November 2009 21:38, Lisandro Dalcin <dal...@gmail.com> wrote:
> On Mon, Nov 16, 2009 at 9:05 PM, francois monard
> <francoi...@gmail.com> wrote:
>> indeed, this is probably not a security problem since the process still
>> hangs even when I turn the firewall off.
>> By the way, I also installed everything on my laptop with XP
>> (mingw/cygwin/mpich2/mpi4py) and obtain the same issue, namely
>> matplotlib.pyplot.show() will hang indefinitely and figures won't popup.
>
> OK. Confirmed. When launching with mpiexec (even with -n 1), the plot
> never appears, the process just hangs and never finish.
>

A follow-up on this, just in case you are still interested ...

you have to run like this (run mpiexec -help2 for explanations on
-localroot flag):

mpiexec -localroot -n 5 python demos\mandelbrot\mandelbrot.py

--
Lisandro Dalcin

[lucian...@gmail.com]

Hi there, 

The instructions for installing mpi4py were very useful, and I want to thank you for sharing them. I have a problem though: when I try to run any code (from the demos), I get this error:

job aborted:

[ranks] message

[0-4] fatal error

Fatal error in MPI_Comm_get_errhandler: Invalid communicator, error stack:

MPI_Comm_get_errhandler(comm=0x0, errhandler=0x100A8264) failed

Invalid communicator

---- error analysis -----

[0-4] on Lucian-PC

mpi has detected a fatal error and aborted python

---- error analysis -----

I am using Win7 and MPI from Microsoft Windows with Python 2.7

Any ideas why this is happening?

Thanks.

[Lisandro Dalcin]

On 25 March 2014 14:47, <lucian...@gmail.com> wrote:
> Hi there,
>
> The instructions for installing mpi4py were very useful, and I want to thank
> you for sharing them. I have a problem though: when I try to run any code
> (from the demos), I get this error:
>
> job aborted:
> [ranks] message
>
> [0-4] fatal error
> Fatal error in MPI_Comm_get_errhandler: Invalid communicator, error stack:
> MPI_Comm_get_errhandler(comm=0x0, errhandler=0x100A8264) failed
> Invalid communicator
>
> ---- error analysis -----
>
> [0-4] on Lucian-PC
> mpi has detected a fatal error and aborted python
>
> ---- error analysis -----
>
> I am using Win7 and MPI from Microsoft Windows with Python 2.7
>
> Any ideas why this is happening?
>

Any chance mpi4py got linked against the wrong library? Do you have
any other MPI installed in your system? Can you send us the log of the
build?


--
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169

[lucian...@gmail.com]

Thanks for the fast reply.

After checking everything from the start, I found out that I hadn't registered the MPI to my user :D... but was no able to to do for MS-MPI, so i switch to MPICH2 (mpich2-1.0.3-1-win32-ia32). So sorry for my mistake. Even so, I still have some problems.

The rebuilding log looks like:

C:\Users\Lucian\Downloads\mpi4py-1.3>python setup.py build --mpi=mpich2-win32

running build

running build_src

running build_py

running build_clib

MPI configuration: [mpich2-win32] from 'mpi.cfg'

running build_ext

MPI configuration: [mpich2-win32] from 'mpi.cfg'

running build_exe

MPI configuration: [mpich2-win32] from 'mpi.cfg'

Now, when I try to run an application, I get:

c:\Users\Lucian\Desktop>mpiexec -n 4 python hello.py

Fatal protocol error: check version between Mpiexec.exe, Msmpi.dll, and Smpd.exe.

Fatal protocol error: check version between Mpiexec.exe, Msmpi.dll, and Smpd.exe.

Fatal protocol error: check version between Mpiexec.exe, Msmpi.dll, and Smpd.exe.

Fatal protocol error: check version between Mpiexec.exe, Msmpi.dll, and Smpd.exe.

Aren't these version of MPI compatible?

[Lisandro Dalcin]

On 29 March 2014 14:39, <lucian...@gmail.com> wrote:
> Thanks for the fast reply.
>
> After checking everything from the start, I found out that I hadn't
> registered the MPI to my user :D... but was no able to to do for MS-MPI, so
> i switch to MPICH2 (mpich2-1.0.3-1-win32-ia32). So sorry for my mistake.
> Even so, I still have some problems.
>
> The rebuilding log looks like:
>
> C:\Users\Lucian\Downloads\mpi4py-1.3>python setup.py build
> --mpi=mpich2-win32
> running build
> running build_src
> running build_py
> running build_clib
> MPI configuration: [mpich2-win32] from 'mpi.cfg'
> running build_ext
> MPI configuration: [mpich2-win32] from 'mpi.cfg'
> running build_exe
> MPI configuration: [mpich2-win32] from 'mpi.cfg'
>

Please remove the build/ folder before rebuilding:

RMDIR /S /Q build

and after that:

python setup.py build --mpi=mpich2-win32 install

>
> Now, when I try to run an application, I get:
>
> c:\Users\Lucian\Desktop>mpiexec -n 4 python hello.py
> Fatal protocol error: check version between Mpiexec.exe, Msmpi.dll, and
> Smpd.exe.
> Fatal protocol error: check version between Mpiexec.exe, Msmpi.dll, and
> Smpd.exe.
> Fatal protocol error: check version between Mpiexec.exe, Msmpi.dll, and
> Smpd.exe.
> Fatal protocol error: check version between Mpiexec.exe, Msmpi.dll, and
> Smpd.exe.
>

This probably related to a "broken" rebuild, follow my steps above

> Aren't these version of MPI compatible?
>

Do you mean MS-MPI and MPICH2? The first is a derivative of the
second, but I would not expect them to be compatible.