Building distutils setup.py to find PXD files for Pure Python file

[Ian Bell]

Dear Cython users,

After more than 12 hours of fighting with Cython+distutils, I am getting quite frustrated.  I am trying to do something that seems like it should be quite trivial - compile a pure Python file with an augmenting PXD file backpack using Cython+distutils.  I have taken the pure-Python files listed below from Stefan's talk, and I cannot for the life of me build a setup.py that will compile properly and - this is the part I can't get to work - include the typedefs from the PXD.  If I can get this to work, I should be able to manage everything else for my modules.

I have the two files integrate_py.py and integrate_py.pxd in the same directory, and when I run python setup.py build_ext --inplace, python builds example.pyd, but doesn't include the PXD typdefs in the c-code it generates.  Can someone point me to my mistake?  I feel like this is a pretty common use-case of pure Python mode with Cython, and it would be great to have this better documented - I would be happy to contribute in this regards.  Other users seem to have run into the same problems before: http://codespeak.net/pipermail/cython-dev/2009-September/006837.html

For what it's worth, if at a command prompt in the source folder I do cython.py -a example_py.py, it builds the example.c properly (including the typedefs from the PXD file).  Any thoughts.

Thanks again,
Ian

#### integrate_py.py ###########
# integrate_py.py

from math import sin

def f(x):
    return sin(x**2)

def integrate_f(a, b, N):

    dx = (b-a)/N
    s = 0
    for i in range(N):
        s += f(a+i*dx)
    return s * dx

#### integrate_py.pxd ########
# integrate_py.pxd
cimport cython

cpdef double f(double x)

@cython.locals(dx=double, s=double, i=int)
cpdef integrate_f(double a, double b, int N)

#### setup.py ##############
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [Extension("example", ["integrate_py.py"])]

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = ext_modules
)

[Stefan Behnel]

Ian Bell, 27.04.2012 13:53:

> After more than 12 hours of fighting with Cython+distutils, I am getting
> quite frustrated. I am trying to do something that seems like it should be
> quite trivial - compile a pure Python file with an augmenting PXD file
> backpack using Cython+distutils. I have taken the pure-Python files listed

> below from Stefan's talk <http://www.behnel.de/cython200910/talk.html>, and

> I cannot for the life of me build a setup.py that will compile properly and
> - this is the part I can't get to work - include the typedefs from the
> PXD. If I can get this to work, I should be able to manage everything else
> for my modules.

> [...]

> #### setup.py ##############
> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
>
> ext_modules = [Extension("example", ["integrate_py.py"])]
>
> setup(
> cmdclass = {'build_ext': build_ext},
> ext_modules = ext_modules
> )

That's the "old" way of doing it, although it should still work. Not sure
what the problem might be here. It may be an issue that you are renaming
your extension from "integrate_py" to "example" above. By common
convention, the module name part of the source file name is the same as
that of the extension, so that it's easier for human readers to make the
link ("readability counts").

In any case, the new and preferred way is described here:

http://wiki.cython.org/enhancements/distutils_preprocessing

And, yes, it would be good to visibly present this in the documentation.
The build documentation is mostly ok but lacks this feature:

http://docs.cython.org/src/userguide/source_files_and_compilation.html

and especially the quick start part of the build docs appears way too
dominated by the Sage notebook image to be accessible:

http://docs.cython.org/src/quickstart/build.html

Stefan

[Ian Bell]

Stefan,
 

In any case, the new and preferred way is described here:

http://wiki.cython.org/enhancements/distutils_preprocessing

And, yes, it would be good to visibly present this in the documentation.
The build documentation is mostly ok but lacks this feature:

http://docs.cython.org/src/userguide/source_files_and_compilation.html

The build docs are in good shape - lots of useful information, but from browsing the mailing lists, seems like quite a few people are having varied issues with PXD files and distutils.   Would be great to have a (nominally) working example in the documentation online and/or a downloadable version.  Also, an example of getting Cython distutils parameters would be great.

Just for fun, tried a new setup.py file, like the link you recommended using cythonize. Works a charm.  Weird.  I think I am in business for now.  Thanks for your help and patience.

So it seems that cythonize does something slightly different than the Cython build_ext command.  I suppose this shouldn't be the expected behavior?

Cheers,
Ian

##### setup2.py  #######
from distutils.core import setup
from Cython.Build import cythonize

setup(
  name = 'MyProject',
  ext_modules = cythonize(["integrate_py.py"]),
)

[mark florisson]

Have you tried using Cython.Distutils.extension.Extension instead of
distutils' Extension class?

[Ian Bell]

Interestingly, if I use Cython.Distutils.extension.Extension but leave the extension module name as example (see setup4.py below), it doesn't include the PXD file.

But if I change the module name to integrate_py in setup.py(as below), it does work and finds the PXD file properly.  The file I list below for setup.py DOES work, either using

from distutils.core import Extension

or

from Cython.Distutils.extension import Extension.  I would argue this is a bug, or at worst should spit out a warning at build time.  It is quite unexpected behavior, especially considering the documentation of pure Python mode in the docs online.

#### setup.py #####
from distutils.core import setup,Extension
from Cython.Distutils import build_ext

#Note: the name of the module MUST be the same as the name of the pure Python file
ext_modules = [Extension("integrate_py", ["integrate_py.py"])]

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = ext_modules
)

#### setup4.py #####

#NB: This code doesn't include the PXD properly
from distutils.core import setup
from Cython.Distutils.extension import Extension

[Stefan Behnel]

Ian Bell, 29.04.2012 02:06:
> On Sun, Apr 29, 2012 at 12:20 AM, mark florisson wrote:

>> On 27 April 2012 14:03, Ian Bell wrote:
>>>> In any case, the new and preferred way is described here:
>>>>
>>>> http://wiki.cython.org/enhancements/distutils_preprocessing
>>>>
>>>> And, yes, it would be good to visibly present this in the documentation.
>>>> The build documentation is mostly ok but lacks this feature:
>>>>
>>>> http://docs.cython.org/src/userguide/source_files_and_compilation.html
>>>>
>>> The build docs are in good shape - lots of useful information, but from
>>> browsing the mailing lists, seems like quite a few people are having
>> varied
>>> issues with PXD files and distutils. Would be great to have a
>> (nominally)
>>> working example in the documentation online and/or a downloadable
>> version.
>>> Also, an example of getting Cython distutils parameters would be great.
>>>
>>> Just for fun, tried a new setup.py file, like the link you recommended
>> using
>>> cythonize. Works a charm. Weird. I think I am in business for now.
>> Thanks
>>> for your help and patience.
>>>
>>> So it seems that cythonize does something slightly different than the
>>> Cython
>>> build_ext command. I suppose this shouldn't be the expected behavior?
>>>

>>> ##### setup2.py #######
>>> from distutils.core import setup
>>> from Cython.Build import cythonize
>>>
>>> setup(
>>> name = 'MyProject',
>>> ext_modules = cythonize(["integrate_py.py"]),
>>> )
>

> But if I change the module name to integrate_py in setup.py(as below), it
> does work and finds the PXD file properly. The file I list below for
> setup.py DOES work, either using
>
> from distutils.core import Extension
>
> or
>
> from Cython.Distutils.extension import Extension. I would argue this is a
> bug

Agreed. It should consider the name of the source file, not the name of the
target extension module (even if using different names for both is
confusing to humans).

Stefan