Re: PyPar Build Question

[Ole Nielsen]

Hi Jay

No trouble at all,  Jay.

I cannot say, I understand the issue fully, but I think you are right about Python expecting to use the same version of gcc that it (itself) was compiled with. And I am not sure about the subtleties on Macs.

Perhaps, you can try to just compile it (i.e. not using the distutils) by running (and perhaps modifying) the little script:

python compile_pypar_locally.py

I got this output:

ro@homedesk:~/sandpit/pypar/source$ python compile_pypar_locally.py 

mpicc -c mpiext.c -I/usr/include/python2.7 -I/usr/lib/pymodules/python2.7/numpy/core/include -o mpiext.o -Wall

mpicc -shared mpiext.o  -o mpiext.so

MPI initialised OK

And then running the test

ro@homedesk:~/sandpit/pypar/source$ mpirun -np 4 python test_pypar.py 

I got the output listed below.

My environment here at home is 

• Ubuntu 11.04 - 32 bit
• gcc (Ubuntu/Linaro 4.5.2-8ubuntu4) 4.5.2

with the dependencies

sudo apt-get install openmpi-common openmpi-bin libopenmpi-dev 

Hope this helps - otherwise, perhaps someone on this mailing list can chime in.

Cheers

Ole

#-------------- Test Output -----------------

ro@homedesk:~/sandpit/pypar/source$ mpirun -np 4 python test_pypar.py 

Pypar (version 2.1.4) initialised MPI OK with 4 processors

I am processor 0 of 4 on node "homedesk"

I am processor 1 of 4 on node "homedesk"

I am processor 3 of 4 on node "homedesk"

Buffered communication of numeric integer arrays OK

Buffered communication of numeric long integer arrays OK

Buffered communication of numeric real arrays OK

Buffered communication of numeric complex arrays OK

I am processor 2 of 4 on node "homedesk"

Buffered communication of strings OK

Buffered communication of general structures OK

Buffered communication of 2D real arrays OK

Buffered communication of 2D complex arrays OK

Buffered communication of 3D real real arrays OK

Buffered communication of 4D real real arrays OK

Buffered communication of 5D real real arrays OK

Buffered communication of 5D double arrays OK

Buffered communication of 5D complex arrays OK

Simplified communication of numeric integer arrays OK

Simplified communication of long integer real arrays OK

Simplified communication of numeric real arrays OK

Simplified communication of numeric complex arrays OK

Simplified communication of strings OK

Simplified communication of general structures OK

Simplified communication of 2D real arrays OK

Simplified communication of 3D real arrays OK

Simplified communication of 4D real arrays OK

Simplified communication of 5D real arrays OK

Simplified communication of 5D double arrays OK

Simplified communication of 5D complex real arrays OK

Broadcast communication of strings OK

Broadcast communication of numeric integer array OK

Broadcast communication of numeric real array OK

Broadcast communication of 2D numeric real array OK

Broadcast communication of 3D numeric real array OK

Broadcast communication of 3D numeric complex array OK

Broadcast communication of general structures OK

Scatter communication of strings OK

Scatter communication of numeric integer array OK

Scatter communication of numeric real arrays OK

Scatter communication of numeric complex array OK

Gather communication of strings OK

Gather communication of numeric integer array OK

Gather communication of numeric real array OK

Gather communication of numeric complex arrays OK

Gather communication of 2D numeric complex arrays OK

Buffered reduce using pypar.SUM OK

Buffered reduce using pypar.MAX OK

Buffered reduce using pypar.MIN OK

Buffered reduce using pypar.PROD OK

Buffered reduce using pypar.LAND OK

Buffered reduce using pypar.BAND OK

Buffered reduce using pypar.LOR OK

Buffered reduce using pypar.BOR OK

Buffered reduce using pypar.LXOR OK

Buffered reduce using pypar.BXOR OK

Status object (numeric integer arrays) OK

Status object (numeric real arrays) OK

Status object (strings) OK

Status object (more general structures) OK

Status object (general structures) OK

Status object (buffered numeric integer arrays) OK

Status object (buffered numeric real arrays) OK

Status object (buffered strings) OK

Status object (buffered general structures) OK

On 23 September 2012 07:17, jay larson <jay.l...@anu.edu.au> wrote:

Hi Ole,

Sorry to bother you on a Sunday morning.  I am trying to get the current (from google code) version of pypar to build on an OS X 10.8 (Mountain Lion) machine.

I am hitting a snag when I do a setup.py

<spew>
larson@calhoun:source>python setup.py build
/usr/local/bin/gcc -I/opt/openmpi/include -L/opt/openmpi/lib -lmpi -lm

running build
running build_py
running build_ext
building 'pypar.mpiext' extension
creating build/temp.macosx-10.6-intel-2.7
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -g -O2 -DNDEBUG -g -O3 -I/opt/openmpi/include -I/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c mpiext.c -o build/temp.macosx-10.6-intel-2.7/mpiext.o -fPIC
unable to execute gcc-4.2: No such file or directory
error: command 'gcc-4.2' failed with exit status 1
</spew>

It appears the problem stems from an incompatibility between the gcc version python expects and the one actually
on my system; I have the gcc 4.8.0 series in /usr/local/bin, Apple sticks us with gcc 4.2.1 in their python.  I thought
there was some fairly gentle way of persuading the distutils-based setup.py to accept my gcc 4.8 as the C compiler,
but I'm just grasping at straws here...Is there a simple command-line option way to force this?  Or, do I have to hack
setup.py?

BTW, I tried setting the CC environment variable (export CC=/usr/local/bin/gcc , which is my 4.8 version), but it caused
the build to fail with tons of 'bad register' errors.  I also tried just supplying it as an invocation option

<moreSpew>
/usr/local/bin/gcc -I/opt/openmpi/include -L/opt/openmpi/lib -lmpi -lm

running build
running build_py
running build_ext
error: don't know how to compile C/C++ code on platform 'posix' with 'gcc' compiler
</moreSpew>

FWIW, I'm locked into 4.8 because the associated gfortran is better than its 4.2 counterpart...

Thanks for any advice/help you can offer in this matter...I'd like to think I'm doing something dumb that has a straightforward
fix...

All the Best,
                  Jay

[Ole Nielsen]

Please let (and pypar-discuss) how you go and if you find a good solution.

Cheers

Ole 

On 23 September 2012 09:49, Jay Larson <jay.l...@anu.edu.au> wrote:

Thanks for your speedy reply, Ole!  I will try this once I am back to my laptop. It's a relief that I am not immediately destined to have to install a separate python to bind to the gcc 4.8  series. FWIW my openmpi was built with gcc 4.8, so mpicc should get me there. 

Thanks again...

Jay

Sent from my iPhone

[Ole Nielsen]

Thanks Jay

Well done.

However, it still annoys me that we have to invoke the CDLL as prior to around 2009 pypar would run nicely without and just compile the c-extension against the MPI library.

But this seems to be the case no more and the CDLL call has caused many problems like the one you experienced.

I'd be grateful is someone could help us out here and get pypar back to where it was  -  light and easy to install across platforms.

Cheers and thanks

Ole 

 

On 27 September 2012 08:49, jay larson <jay.l...@anu.edu.au> wrote:

Hi Ole (and the good folks on pypar-discuss)...

Sorry for the slow-ish reply; I was at CTAC 2012 and was too beat to chase this
in my spare time.

Thank you--yes, your advice worked a treat with one wrinkle--my OpenMPI installation
uses Apple's .dylib convention for its shared libraries.  I had to change one line of pypar.py:

Change this:

mpi = CDLL('libmpi.so.0', RTLD_GLOBAL)

to this:

mpi = CDLL('libmpi.dylib', RTLD_GLOBAL)

The test suit ran just fine with that change.  For the sake of record-keeping, here's my config

OS X 10.8.2 "Mountain Lion"
GNU c/c++ compilers version 4.8.0
OpenMPI version 1.6.1
And I'm using Apple's Python -- see below
python
Python 2.7.2 (v2.7.2:8527427914a2, Jun 11 2011, 15:22:34)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin

All the Best,
                    Jay