OSX 10.8: setuptools generates gcc command that wants to use to use .c instead of .cpp

[Justin Israel]

This is a strange issue that happens for my project only when someone with OSX 10.8 (mountain lion) tries to build the extension. I don't have 10.8 so I am still waiting on a colleague to forward me a dump of the literal failure, but I figured I could start the question now and provide more info if needed.

The issue is pretty straight forward. On <=10.7.x, when building the extension in place, it properly respects the language="c++" Extension attribute. It cythonizes the pyx => cpp, and runs the proper gcc command to build.

On 10.8.x, it seems it is cythonizing the cpp file just fine, but then it generates a gcc command that is trying to compile a .c that does not exist, obviously. The way I can get it to succeed is if I modify my setup file to not use cython. Then I manually cythonize the pyx => cpp, and have it use the cpp source directly in the setuptools Extension (I have one of those common use_cython style flag setups that I just force to False). 

Does anyone offhand know what might be an issue on osx 10.8? I know apple mucked with the compiler setups a whole bunch, and have seen my fair share of posts containing "problems compiling XYZ on 10.8", requiring some special adjustment. I know that the person who encountered this error had freshly installed the xcode command-line tools to get their compilers, and had no previous setup for building from source code until they were trying to build my extension.

Example setup.py code looks like:

...

cmdclass = {}

if use_cython:

    SRC = 'foo.pyx'

    cmdclass['build_ext'] = build_ext

else:

    SRC = 'foo.cpp'

...

ext = Extension('plow.client.plow',

    [SRC],

    language="c++",

    libraries=[...], 

    extra_compile_args=cflags,

    extra_link_args=ldflags,

    define_macros=[...]

)

...

setup(

   ...

    ext_modules = [ext],

    cmdclass=cmdclass,

   ...

)

[Chris Barker - NOAA Federal]

On Fri, May 17, 2013 at 6:55 PM, Justin Israel <justin...@gmail.com> wrote:
> This is a strange issue that happens for my project only when someone with
> OSX 10.8 (mountain lion) tries to build the extension.

You might try the python-mac list -- there are some smart and helpful
folks there.

> I don't have 10.8 so
> I am still waiting on a colleague to forward me a dump of the literal
> failure,

That's going to be tough...I would at least put together a really
simple, small example that you can post so others can easily test.
(sorry -- I"m still avoiding 10.8 for this reason, among others...)

> On 10.8.x, it seems it is cythonizing the cpp file just fine,

so Cython is doing the right thing

> but then it
> generates a gcc command that is trying to compile a .c that does not exist,
> obviously.

that is really wierd -- that command is generated by distutils, so
should have nothign do do with what compiler the user is using...

What version of Python is this? what build? That may be where your
problem is. I generally recommend using the binaries from Python.org
-- at least that way you've got a common system folks can help with.
(doing it all in macports or homebrew is another option).

> The way I can get it to succeed is if I modify my setup file to
> not use cython. Then I manually cythonize the pyx => cpp, and have it use
> the cpp source directly in the setuptools Extension (I have one of those
> common use_cython style flag setups that I just force to False).

so the system can build C++ extensions -- and distutils seems to do it
right. This is really, really odd!

> Does anyone offhand know what might be an issue on osx 10.8? I know apple
> mucked with the compiler setups a whole bunch, and have seen my fair share
> of posts containing "problems compiling XYZ on 10.8", requiring some special
> adjustment.

yes, but this doesn't seem to fit -- it's not even calling g++, I take it?

Good luck!

-Chris


--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

[Justin Israel]

Thanks for the reply!

It would be the apple python 2.7.x that ships with 10.8

Cython is doing the right thing in terms of cythonizing the file, but without delving into the source I had assumed the custom Extension class that is being used from cython was doing something wrong. I figured it was ignoring that language=c++ flag and returning the wrong output source type to be compiled?

The reason it works with the workaround is that I don't let the cython Extension handle the conversion. I just explicitly tell it to use the pregenerated .cpp source. At that point the right gcc command is used.

You are correct that when it fails under the normal cythonizing approach, it does not get to the linker step with g++. That is because it failed at the first gcc compile of the cythonized source file, which it thinks should be a .c file and not .cpp

--

---
You received this message because you are subscribed to a topic in the Google Groups "cython-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cython-users/6TPt_q9DS5I/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Chris Barker - NOAA Federal]

On Tue, May 21, 2013 at 1:22 PM, Justin Israel <justin...@gmail.com> wrote:
> It would be the apple python 2.7.x that ships with 10.8

I wonder if it's something funky with Apple's distutils setup.

I'd have them try the python.org binary Python.

In any case, it doesn't sound like a Cython thing to me, but hard to
tell without getting my hands on it.

-Chris

> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Robert Bradshaw]

I'm not sure what exactly the issue is, but the line

cmdclass['build_ext'] = build_ext

replaces the built-in build_ext with the Cython-provided one. Instead
I would reccommend using the cythonize() command, found at
http://docs.cython.org/src/reference/compilation.html (to switch
between the options, you could provide a fake_cythonize that simply
replaces each .pyx extension with .cpp).



On Tue, May 21, 2013 at 2:30 PM, Chris Barker - NOAA Federal

[Justin Israel]

Awesome. I am going to try out replacing with the cythonize command.

[Justin Israel]

Hmm... Am I crazy or is this a bug (cython 0.19) in cythonize?

Example cythonize command:

ext_list = cythonize(['src/plow.pyx'], language="c++", force=True)
ext = ext_list[0]
print e.sources #['src/plow.c']
print e.language # None

From what I examined in Cython/Build/Dependencies.py, it looks like it does not maintain the language flag being passed in at all:

from Cython.Build.Dependencies import *
options = {'language': 'c++'}
# at this point in the original source code, even though it created both a c and cpp options object
# it calls create_extension_list with the c version. For testing, I am explicitly passing the cpp version
cpp_options = CompilationOptions(**options); cpp_options.cplus = True
ctx = cpp_options.create_context()
module_list = create_extension_list(['src/plow.pyx'], ctx=ctx)
m = module_list[0]
print m.language # None
print m.sources # ["src/plow.pyx"]

The language value gets lost when it is parsing the string, finding everything, and building back up an Extension. Now, if I were to create my own Extension object and pass that in, the logic in create_extension_list short-circuits early and uses that original object, thus preserving my flag:

from distutils.extension import Extension
ex = Extension('plow', ['src/plow.pyx'], language="c++")
e = cythonize([ex], force=True)[0]
print e.sources # ['src/plow.cpp']
print e.language # 'cpp'

Does this look right? I'm fine accomplishing it this way, by passing in an Extension instance, but the source codes logic looked strange in cythonize.