HOWTO: Fix gcc "No such file or directory" error when installing Cython extension modules with pip or easy_install

[mwi...@eng.ucsd.edu]

It took me a few hours to figure this out.

For people creating Cython packages, if you notice that this:

  $ python setup.py build_ext -i

works fine, while this:

  $ pip install .

breaks with "gcc: error: hello.c: No such file or directory", you have two options:

1. Install the 'distribute' package, or
2. Include the following magic in your setup.py:

    # Needed to fix pip                                                             
    # See https://pypi.python.org/pypi/setuptools_cython/,                          
    # http://mail.python.org/pipermail/distutils-sig/2007-September/thread.html#8204
    import sys                                                                      
    if 'setuptools.extension' in sys.modules:                                       
        m = sys.modules['setuptools.extension']                                     
        m.Extension.__dict__ = m._Extension.__dict__ 

The reason is that Cython and setup.py use "distutils", the oldest python packaging system. However, easy_install and pip use "setuptools", a slightly newer but still old packaging system that monkeypatched support for Pyrex .pyx files. Setuptools will go through all the Extension()s and replace ".pyx" files with ".c" files without compiling them. The newer "distribute" package *reimplements* setuptools without breaking cython, which is why pip/easy_install mysteriously work if you happen to have distribute installed.

I'm not sure if this is a bug or what's happening here...

Just figured I'd post this for others who are having the same issue.

[Stefan Behnel]

mwi...@eng.ucsd.edu, 08.07.2013 07:24:

> It took me a few hours to figure this out.
>
> For people creating Cython packages, if you notice that this:
>
> $ python setup.py build_ext -i
>
> works fine, while this:
>
> $ pip install .
>
> breaks with "gcc: error: hello.c: No such file or directory", you have two
> options:
>
> 1. Install the 'distribute' package, or
> 2. Include the following magic in your setup.py:
>
> # Needed to fix
> pip
> # See
> https://pypi.python.org/pypi/setuptools_cython/,
> #
> http://mail.python.org/pipermail/distutils-sig/2007-September/thread.html#8204
>
> import
> sys
> if 'setuptools.extension' in
> sys.modules:
> m =
> sys.modules['setuptools.extension']
> m.Extension.__dict__ = m._Extension.__dict__

This is a rather excessive hack.

> The reason is that Cython and setup.py use "distutils", the oldest python
> packaging system. However, easy_install and pip use "setuptools", a
> slightly newer but still old packaging system that monkeypatched support
> for Pyrex .pyx files. Setuptools will go through all the Extension()s and
> replace ".pyx" files with ".c" files without compiling them. The newer
> "distribute" package *reimplements* setuptools without breaking cython,
> which is why pip/easy_install mysteriously work if you happen to have
> distribute installed.
>
> I'm not sure if this is a bug or what's happening here...
>
> Just figured I'd post this for others who are having the same issue.

Yes, that's a well known bug in the old setuptools package. I hope that
post-distribute-merge versions finally fix this.

The canonical answer is to not use the old distutils integration but to use
cythonize() in your setup.py instead.

http://docs.cython.org/src/reference/compilation.html#compiling-with-distutils

Stefan

[mwi...@eng.ucsd.edu]

On Sunday, July 7, 2013 11:39:06 PM UTC-6, Stefan Behnel wrote:

The canonical answer is to not use the old distutils integration but to use
cythonize() in your setup.py instead.

http://docs.cython.org/src/reference/compilation.html#compiling-with-distutils

Is it possible to use cythonize() with include_dirs = [numpy.get_include()] ? If I use a comment like this in my .pyx:

# distutils: include_dirs = [numpy.get_include()]

then it's interpreted as a literal string, eg. "-Inumpy.get_include()" is passed as a compiler flag.

[Stefan Behnel]

mwi...@eng.ucsd.edu, 08.07.2013 17:08:

> On Sunday, July 7, 2013 11:39:06 PM UTC-6, Stefan Behnel wrote:
>>
>> The canonical answer is to not use the old distutils integration but to
>> use
>> cythonize() in your setup.py instead.
>>

>> ht<http://docs.cython.org/src/reference/compilation.html#compiling-with-distutils>
>> tp://docs.cython.org/src/reference/compilation.html#compiling-with-distutils<http://docs.cython.org/src/reference/compilation.html#compiling-with-distutils>

>>
>
> Is it possible to use cythonize() with include_dirs = [numpy.get_include()]
> ? If I use a comment like this in my .pyx:
>
> # distutils: include_dirs = [numpy.get_include()]
>
> then it's interpreted as a literal string, eg. "-Inumpy.get_include()" is
> passed as a compiler flag.

We only recently updated the documentation, here's the in-dev version:

https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/src/reference/compilation.html

Seems like they still don't contain a mention of the aliasing support, but
IMHO the best way to do it is by creating Extension() instances anyway, as
that makes it very easy to disable the Cython compilation in the build.

https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/src/userguide/source_files_and_compilation.html#distributing-cython-modules

(and yes, the docs totally need some merging - they are way too redundant...)

Stefan

[Stefan Behnel]

Stefan Behnel, 08.07.2013 18:02:

> https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/src/userguide/source_files_and_compilation.html#distributing-cython-modules
>
> (and yes, the docs totally need some merging - they are way too redundant...)

I moved at least that section over to the reference page now. Some help
with further cleanups would be warmly appreciated.

Stefan