setup.py question: Why does the Cythonize step always go off before anything starts?

[Tom Swirly]

Hello!  Too much of my time has gone into this tiny question...

Python 3.4, cross-platform compilation (currently on OS/X).

I have added other targets to my setuptools setup.py like "generate" that creates generated files.  So far so good, but the Cythonize step always goes off before the generated file step, when of course I want it to come after.

This is particularly annoying when there's a mistake in my scripts which generates a .pyx file that can't be compiled, because then it always tries and fails to cythonize the .pyx before it even tries to get to the code generation - so I can't even revert the change and fix.

Here's the setup.py:  https://github.com/rec/tada/blob/templates/setup.py

--

     /t

http://radio.swirly.com - art music radio 24/7 366/1000

[Robert Bradshaw]

setup.py is a python script, cythonize is executed at
https://github.com/rec/tada/blob/templates/setup.py#L110, and your
generate is likely deferred until during the setuptools.setup call at
https://github.com/rec/tada/blob/templates/setup.py#L127

One option you could do is synchronize things better is create your
own build_ext that call cythonize later, e.g.

from setuptools.command.build_ext import build_ext as _build_ext

class build_ext(_build_ext):
def finalize_options(self):
from Cython.Build import cythonize
self.distribution.ext_modules[:] = cythonize(
self.distribution.ext_modules)
super(build_ext, self).finalize_options()

setup(..., cmdclass={'build_ext': build_ext, ...})

[Tom Swirly]

Aha, got it, the cythonize step actually does the cythonization occurs right at the moment of reading setup.py and not part of the build!

Sort of obnoxious in a build system - pretty well anything you prepare should give you back something that operates later and not do something right now - because of exactly this issue.

I'll experiment with your solution later but it has the Ring Of Truth to it/I'll bet I can make it work because it obviously defers the Cythonization till later (which should be the default state of affairs, but oh, well...)

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Swirly]

Had a few minutes to spare and it worked right first time:  https://github.com/rec/tada/commit/e54c020e383dc6e52b4e6d2fcffdc8f34bcd4b8f

Another happy customer!

[Jeroen Demeyer]

On 2016-06-13 18:00, Robert Bradshaw wrote:
> One option you could do is synchronize things better is create your
> own build_ext that call cythonize later

Cython has its own Cython.Distutils.build_ext, why does it not just call
cythonize() like you suggest?

Cython's build_ext seems to reinvent part of cythonize() but in a worse
way: it doesn't do the sophisticated dependency checking that
cythonize() does.

[Tom Swirly]

On Tue, Jun 14, 2016 at 10:51 AM, Erik Bray <erik....@gmail.com> wrote:

Coincidentally, I was discussing this issue just the other day on sage-devel: https://groups.google.com/d/msg/sage-devel/LIbAfzFWpOU/scrRQRPJDQAJ

This has been a pet peeve of mine for a while.  The current Cython docs, which demonstrate how to use cythonize() in setup.py, are fine for getting off the ground with Cython projects.  But it's poor advice when it comes to packaging and distribution of Cython projects.  So I think the docs should be supplemented to explain some of what I explained in the post linked above, and to demonstrate a recipe for distributing Cython projects.

Yes, agree 100%.  I asked a question here earlier about this, but got it so wrong I didn't get applicable answers.  

There should be some nice template you can just paste into your code and go.

Indeed - there should be a "sample empty Cython project" that has all the parts you'd need for almost any Cython project - but had a single function that returned "hello, world" and a Python unit test for that function! - so you can copy all the parts of this and start working immediately with a test framework.

(and yes, I'm willing to help bell the cat.  In fact, I currently have pretty well all these parts put together, with that extra generated code step we're discussing, so it might be just ripping things out...)

 

This also raises the issue of what ever happened to Cython.Distutils.BuildExt, which still exists in the source, and can still be invoked by setuptools, but which does not take advantage of the enhancements that cythonize offers.  It might be good if Cython.Distutils.BuildExt were updated to take better advantage of cythonize, as well as implement good practices like shipping C sources in source tarballs (which may also require updating setuptools' sdist command).  My guess is simply that nobody wanted to maintain the distutils commands, but I'm not sure.  By bringing this up I'm de-facto offering to help with the issue if we can agree on how or whether to improve matters.

+1 on this too!

(er, what do you mean by "updating" setuptools?  Monkey patching - or getting a change into setuptools' source code itself?)

I commented elsewhere that I like the idea of distributing the .cpp code, but I really hate the idea of checking it into a repository (because it fouls your diffs).  This is the best of both worlds.

I'm going to start a new thread on "Cython setup.py best practices and sample project" and if someone else wants to follow up on Cython.Distutils.BuildExt too...?

[Robert Bradshaw]

cythonize was introduced due to the pain of mixing with the many other
projects that extend distutils themselves. However, it's clear that it
does have its drawbacks.

The existing Cython.Distutils.BuildExt is quite complicated, has many
(likely barely used) features, and I'd worry trying to change that
would break people for little benefit. I created

https://github.com/cython/cython/commit/9f0e25f72b581201233a2a814961c4bc9c77ba00

instead, which is simple and does the right thing. It could be
interesting exploring doing this as part of sdist as well. Might still
be worth gutting/fixing Cython.Distutils.BuildExt, I just don't know
how many people are using its esoteric features.

More contributions welcome.

> (er, what do you mean by "updating" setuptools? Monkey patching - or
> getting a change into setuptools' source code itself?)
>
> I commented elsewhere that I like the idea of distributing the .cpp code,
> but I really hate the idea of checking it into a repository (because it
> fouls your diffs). This is the best of both worlds.
>
>
> I'm going to start a new thread on "Cython setup.py best practices and
> sample project" and if someone else wants to follow up on
> Cython.Distutils.BuildExt too...?
>
>
> --
> /t
>
> http://radio.swirly.com - art music radio 24/7 366/1000
>

[Tom Swirly]

That small change would seem to satisfies my needs for the foreseeable future, at least...

[Erik Bray]

Coincidentally, I was discussing this issue just the other day on sage-devel: https://groups.google.com/d/msg/sage-devel/LIbAfzFWpOU/scrRQRPJDQAJ

This has been a pet peeve of mine for a while.  The current Cython docs, which demonstrate how to use cythonize() in setup.py, are fine for getting off the ground with Cython projects.  But it's poor advice when it comes to packaging and distribution of Cython projects.  So I think the docs should be supplemented to explain some of what I explained in the post linked above, and to demonstrate a recipe for distributing Cython projects.

This also raises the issue of what ever happened to Cython.Distutils.BuildExt, which still exists in the source, and can still be invoked by setuptools, but which does not take advantage of the enhancements that cythonize offers.  It might be good if Cython.Distutils.BuildExt were updated to take better advantage of cythonize, as well as implement good practices like shipping C sources in source tarballs (which may also require updating setuptools' sdist command).  My guess is simply that nobody wanted to maintain the distutils commands, but I'm not sure.  By bringing this up I'm de-facto offering to help with the issue if we can agree on how or whether to improve matters.

Best,

Erik 

[Erik Bray]

On Tue, Jun 14, 2016 at 5:54 PM, Tom Swirly <t...@swirly.com> wrote:
> On Tue, Jun 14, 2016 at 10:51 AM, Erik Bray <erik....@gmail.com> wrote:
>>
>> Coincidentally, I was discussing this issue just the other day on
>> sage-devel:
>> https://groups.google.com/d/msg/sage-devel/LIbAfzFWpOU/scrRQRPJDQAJ
>>
>> This has been a pet peeve of mine for a while. The current Cython docs,
>> which demonstrate how to use cythonize() in setup.py, are fine for getting
>> off the ground with Cython projects. But it's poor advice when it comes to
>> packaging and distribution of Cython projects. So I think the docs should
>> be supplemented to explain some of what I explained in the post linked
>> above, and to demonstrate a recipe for distributing Cython projects.
>
>
> Yes, agree 100%. I asked a question here earlier about this, but got it so
> wrong I didn't get applicable answers.
>
> There should be some nice template you can just paste into your code and go.
>
> Indeed - there should be a "sample empty Cython project" that has all the
> parts you'd need for almost any Cython project - but had a single function
> that returned "hello, world" and a Python unit test for that function! - so
> you can copy all the parts of this and start working immediately with a test
> framework.

For what it's worth the astropy package-template works quite well for
this purpose: https://github.com/astropy/package-template
There's absolutely nothing Astronomy-specific about it, and the only
part that's really astropy-specific is the use, by default, of the
Astropy test runner (which itself is nothing special--just a wrapper
around py.test--and one day should be made a separate package).

I'm not necessarily advocating its use for packages that have nothing
to do with astropy, and it still has some bugs in it too that I've
been agitating about for a while. There's been talk about splitting
off a version of it that's less tied to astropy, but at the moment
there aren't enough resources to support such a thing. But it has the
right general idea.

[Tom Swirly]

Well, at the very least, this does more or less hit the spot as "a fairly full featured Cython package to point people to when they ask questions".

Honestly, I suspect this might be "good enough" for most people, but I'll contemplate what a slightly more generic package might look like - or if you wanted to emit such a thing, I'd code review it.  :-)