parallel cython on mac os Sierra (possibly gcc issue)

[Peter]

I'm having my first try at parallel cython using prange (I have successfully used the multiprocessing module but the syntax is really clumsy), and am trying to follow the instructions at http://cython.readthedocs.io/en/latest/src/userguide/parallelism.html , and also looking at http://nealhughes.net/parallelcomp2/

I get this error (**** is instead of some personal details in my path):

# python cyrun_ptest.pyx

Traceback (most recent call last):

  File "cyrun_ptest.pyx", line 4, in <module>

    import cyrun_ptest2

ImportError: dlopen(/Users/****/work/cyrun_ptest2.so, 2): Symbol not found: _GOMP_parallel

  Referenced from: /Users/****/work/cyrun_ptest2.so

  Expected in: flat namespace

 in /Users/****/work/cyrun_ptest2.so

"import cyrun_ptest2" is the very first non-comment line in cyrun_ptest.pyx.

My compile file looks like this:

from distutils.core import setup

from distutils.extension import Extension

from Cython.Build import cythonize

from Cython.Distutils import build_ext

ext_modules = [

    Extension(

        "cyrun_ptest",

        ["cyrun_ptest.pyx"],

        extra_compile_args=["-fopenmp" ],

        extra_link_args=['-fopenmp'],

    )

]

setup(

       name='cyrun_ptest',

       cmdclass = {"build_ext": build_ext},

       ext_modules = ext_modules

     )

A bit of commenting and uncommenting suggests that the -fopenmp is what's tripping it up. In fact earlier I was being told the gcc did not support the option -fopenmp . I got around that by pointing to the brew gcc (/usr/local/bin) instead of the Mac-supplied one (/usr/bin), but it seems I didn't completely fix things.

Does anyone have an idea how to fix this?

Thanks in advance

[Jérôme Kieffer]

On Sun, 19 Nov 2017 21:31:22 -0800 (PST)
Peter <ozpeter...@gmail.com> wrote:

> Does anyone have an idea how to fix this?

Complain to Apple which does not support OpenMP, more exactly they
dropped the support of OpenMP with 10.6 when they switched from gcc 4.2
to clang (without OpenMP). Even if OpenMP is now supported by the
latest version of clang under linux, it is not the case on Apple'OS.

To be able to use OpenMP, you need to provide a compiler which supports
OpenMP like a geniune gcc.

Cheers,

Jerome

[Peter]

I am already pointing to the gcc from Brew (gcc 7.2.0). Is that not "a genuine gcc"? Output of "gcc -v" is below.

$ gcc -v

Using built-in specs.

COLLECT_GCC=gcc

COLLECT_LTO_WRAPPER=/usr/local/Cellar/gcc/7.2.0/libexec/gcc/x86_64-apple-darwin16.7.0/7.2.0/lto-wrapper

Target: x86_64-apple-darwin16.7.0

Configured with: ../configure --build=x86_64-apple-darwin16.7.0 --prefix=/usr/local/Cellar/gcc/7.2.0 --libdir=/usr/local/Cellar/gcc/7.2.0/lib/gcc/7 --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-7 --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-isl=/usr/local/opt/isl --with-system-zlib --enable-checking=release --with-pkgversion='Homebrew GCC 7.2.0' --with-bugurl=https://github.com/Homebrew/homebrew-core/issues --disable-nls

Thread model: posix

gcc version 7.2.0 (Homebrew GCC 7.2.0) 

On Monday, November 20, 2017 at 5:24:37 PM UTC+10:30, Jerome Kieffer wrote:

On Sun, 19 Nov 2017 21:31:22 -0800 (PST)
Peter <ozpeter...@gmail.com> wrote:

> Does anyone have an idea how to fix this?

[Stefan Behnel]

Am 20. November 2017 23:34:33 MEZ schrieb Peter:
>I am already pointing to the gcc from Brew (gcc 7.2.0). Is that not "a
>genuine gcc"?

You might be compiling with gcc but linking with clang. Male sure you use gcc for both.

Stefan

[Peter]

Compile file:

from distutils.core import setup

from distutils.extension import Extension

from Cython.Build import cythonize

from Cython.Distutils import build_ext

ext_modules = [

    Extension(

        "cyrun_ptest",

        ["cyrun_ptest.pyx"],

        extra_compile_args=["-fopenmp" ],

        extra_link_args=['-fopenmp'],

    )

]

setup(

       name='cyrun_ptest',

       cmdclass = {"build_ext": build_ext},

       ext_modules = ext_modules

     )

Compile command:

python cyrun_ptest_compile.py build_ext --inplace > cyrun_ptest_compile.log 2>&1

Output of compile (i.e. the contents of cyrun_ptest_compile.log) (again **** is my name though it's not hard to guess from my posting name anyway :) :

running build_ext

cythoning cyrun_ptest.pyx to cyrun_ptest.c

building 'cyrun_ptest' extension

gcc -fno-strict-aliasing -I/Users/****/miniconda2/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/****/miniconda2/include/python2.7 -c cyrun_ptest.c -o build/temp.macosx-10.6-x86_64-2.7/cyrun_ptest.o -fopenmp

gcc -bundle -undefined dynamic_lookup -L/Users/****/miniconda2/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.6-x86_64-2.7/cyrun_ptest.o -L/Users/****/miniconda2/lib -o /Users/****/Documents/work/cyrun_ptest.so -fopenmp

I thought that last line did the linking, which would mean I am linking with gcc already. Or does something look wrong?

Thanks again,

Peter

[Francesc Alted]

Hi,

It is not enough to use a naked 'gcc' because Apple calls their own toolchain (clang/LLVM) internally:

Francescs-MacBook-Pro:~ faltet$ gcc -v

Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/c++/4.2.1

Apple LLVM version 9.0.0 (clang-900.0.38)

Target: x86_64-apple-darwin17.2.0

Thread model: posix

InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Better use something like gcc-6 (or gcc-7 in your case):

Francescs-MacBook-Pro:~ faltet$ gcc-6 -v

Using built-in specs.

COLLECT_GCC=gcc-6

COLLECT_LTO_WRAPPER=/usr/local/Cellar/gcc/6.3.0_1/libexec/gcc/x86_64-apple-darwin16.3.0/6.3.0/lto-wrapper

Target: x86_64-apple-darwin16.3.0

Configured with: ../configure --build=x86_64-apple-darwin16.3.0 --prefix=/usr/local/Cellar/gcc/6.3.0_1 --libdir=/usr/local/Cellar/gcc/6.3.0_1/lib/gcc/6 --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-6 --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-isl=/usr/local/opt/isl --with-system-zlib --enable-libstdcxx-time=yes --enable-stage1-checking --enable-checking=release --enable-lto --with-build-config=bootstrap-debug --disable-werror --with-pkgversion='Homebrew GCC 6.3.0_1' --with-bugurl=https://github.com/Homebrew/homebrew-core/issues --enable-plugin --disable-nls --enable-multilib

Thread model: posix

gcc version 6.3.0 (Homebrew GCC 6.3.0_1) 

You can enforce this by setting CC envvar to gcc-X (X being 4,5,6,7...) before you issue any command using a compiler.

Francesc

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Francesc Alted

[Peter]

On Tuesday, November 21, 2017 at 6:31:26 PM UTC+10:30, Francesc Alted wrote:

Hi,

It is not enough to use a naked 'gcc' because Apple calls their own toolchain (clang/LLVM) internally:

Francescs-MacBook-Pro:~ faltet$ gcc -v

Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/c++/4.2.1

Apple LLVM version 9.0.0 (clang-900.0.38)

Target: x86_64-apple-darwin17.2.0

Thread model: posix

InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Better use something like gcc-6 (or gcc-7 in your case):

Francescs-MacBook-Pro:~ faltet$ gcc-6 -v

Using built-in specs.

COLLECT_GCC=gcc-6

COLLECT_LTO_WRAPPER=/usr/local/Cellar/gcc/6.3.0_1/libexec/gcc/x86_64-apple-darwin16.3.0/6.3.0/lto-wrapper

Target: x86_64-apple-darwin16.3.0

Configured with: ../configure --build=x86_64-apple-darwin16.3.0 --prefix=/usr/local/Cellar/gcc/6.3.0_1 --libdir=/usr/local/Cellar/gcc/6.3.0_1/lib/gcc/6 --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-6 --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-isl=/usr/local/opt/isl --with-system-zlib --enable-libstdcxx-time=yes --enable-stage1-checking --enable-checking=release --enable-lto --with-build-config=bootstrap-debug --disable-werror --with-pkgversion='Homebrew GCC 6.3.0_1' --with-bugurl=https://github.com/Homebrew/homebrew-core/issues --enable-plugin --disable-nls --enable-multilib

Thread model: posix

gcc version 6.3.0 (Homebrew GCC 6.3.0_1) 

I think I am getting that already. My output of gcc -v is below. It is very similar to your latter one, and has no mention of clang:

Using built-in specs.

COLLECT_GCC=gcc

COLLECT_LTO_WRAPPER=/usr/local/Cellar/gcc/7.2.0/libexec/gcc/x86_64-apple-darwin16.7.0/7.2.0/lto-wrapper

Target: x86_64-apple-darwin16.7.0

Configured with: ../configure --build=x86_64-apple-darwin16.7.0 --prefix=/usr/local/Cellar/gcc/7.2.0 --libdir=/usr/local/Cellar/gcc/7.2.0/lib/gcc/7 --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-7 --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-isl=/usr/local/opt/isl --with-system-zlib --enable-checking=release --with-pkgversion='Homebrew GCC 7.2.0' --with-bugurl=https://github.com/Homebrew/homebrew-core/issues --disable-nls

Thread model: posix

gcc version 7.2.0 (Homebrew GCC 7.2.0) 

Though I don't understand it all, it seems to be getting everything from /usr/local/bin (where homebrew gcc is) and related paths; rather than from /usr/bin (where the shipped Apple gcc is) and related paths. Or is there something I am missing?

 

You can enforce this by setting CC envvar to gcc-X (X being 4,5,6,7...) before you issue any command using a compiler

CC isn't defined in my shell. I tried defining it anyway (export CC=gcc) but it didn't change anything. I'm still getting (where **** replaces some of my path):

ImportError: dlopen(/Users/****/cyrun_ptest2.so, 2): Symbol not found: _GOMP_parallel

Any ideas how to fix this, anyone? Surely someone is running parallel cython on Mac OS!

[Gregor Thalhammer]

I have success using cython with OpenMP on macOS 10.13, with gcc 7.2 installed via homebrew, and using anaconda python (2.7).

Built an cython extension via

export CC=gcc-7; python setup.py build_ext —inplace

Perhaps just upgrade your gcc via brew? My version is

gcc-7 -v

Using built-in specs.

COLLECT_GCC=gcc-7

COLLECT_LTO_WRAPPER=/usr/local/Cellar/gcc/7.2.0/libexec/gcc/x86_64-apple-darwin17.2.0/7.2.0/lto-wrapper

Target: x86_64-apple-darwin17.2.0

Configured with: ../configure --build=x86_64-apple-darwin17.2.0 --prefix=/usr/local/Cellar/gcc/7.2.0 --libdir=/usr/local/Cellar/gcc/7.2.0/lib/gcc/7 --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-7 --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-isl=/usr/local/opt/isl --with-system-zlib --enable-checking=release --with-pkgversion='Homebrew GCC 7.2.0' --with-bugurl=https://github.com/Homebrew/homebrew-core/issues --disable-nls --with-native-system-header-dir=/usr/include --with-sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk

Thread model: posix

gcc version 7.2.0 (Homebrew GCC 7.2.0) 

Gregor

-- 

--- 
You received this message because you are subscribed to the Google Groups "cython-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.

[Peter]

Sorry for long delay - a combination of personal and work distractions. I'd like to bump this thread again and ask if anyone has idea why I can't use "prange", getting the compile error 

ImportError: dlopen(/Users/****/Documents/postdoc/work/cyrun_ptest2.so, 2): Symbol not found: _GOMP_parallel

Anyway, replying to the last poster (Gregor), if you look at my output of gcc -v (it's in the earlier posts quoted in your reply), you will see it is almost identical to yours. The only differences are:

* I have --build=x86_64-apple-darwin16.7.0 whereas you have --build=x86_64-apple-darwin17.2.0  (I doubt this matters because another replier buulds with darwin16.3.0)

* yours was built with two extra switches:  --with-native-system-header-dir=/usr/include  --with-sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk

I don't see how either of these matter, but maybe they do.

Any suggestions  appreciated.

Peter

[Gregor Thalhammer]

Am 18.12.2017 um 02:17 schrieb Peter <ozpeter...@gmail.com>:

Sorry for long delay - a combination of personal and work distractions. I'd like to bump this thread again and ask if anyone has idea why I can't use "prange", getting the compile error 

ImportError: dlopen(/Users/****/Documents/postdoc/work/cyrun_ptest2.so, 2): Symbol not found: _GOMP_parallel

This is an indication that building is ok (despite slightly different versions than my), but somehow dynamic linking to libgomp fails. Perhaps it is not installed, version mismatch?

My compiled module ist linked to:

Gregors-MBP:cyparallel gregor$ otool -L test1.so 

test1.so:

/usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)

/usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

best

Gregor

[John Bauer]

I suspect that --enable-languages=objc,ob-c++ is the key.  I made some efforts to compile Python from source, which similarly failed to link.  Mostly this is successful with CLANG.  The discussion I found pointed to OpenSSL, which was released hurriedly.  It is apparently a mix of objective-c and c++ which CLANG would compile but gcc would not.  I stopped trying, since that discussion implied that it was only possible to use CLANG.  But perhaps with some additional compiler directives it would link.

[ewadew Hufewaw]

I encounter this problem and find out that  -Wl,-rpath,/your/gcc/lib should be added to the link process, like

ex = Extension('test', ['test.pyx'], extra_link_args=['-Wl,-rpath,/usr/local/Cellar/gcc@7/7.3.0/lib/gcc/7'])

It solves my problem