Re: [cython-users] Tips for building/packaging moderately complex Cython projects?

[Emil Vaughan]

I'm not an expert on this by any means, but in my project Flagmatic I
successfully used multiple Extensions, and some used different
extra_compile_args:

https://github.com/emil79/flagmatic/blob/master/pkg/setup.py

Emil

On 31 January 2013 12:01, Matthew Honnibal <honn...@gmail.com> wrote:
> Hi all,
>
> I've been using Cython as the main development language for a small project
> (an English statistical parser), about 2000 lines of code. Currently my
> build system is an ugly mix of make and setup scripts, and I'm confused
> about how to best build a package that has several Cython modules.
>
> My project has three packages, all of which have cython code, and additional
> C/C++ code. One of them has several Cython modules.
>
> Some questions:
>
> 1) When I have multiple Cython modules in a single package, can I use a
> single setup script? What if one of the modules uses different C/C++
> extensions? I tried just listing several modules in ext_modules, but this
> didn't work immediately, and to be honest distutils confuses me, so I
> settled for a hacky solution to get it working. If someone has an example
> package with multiple Cython modules that would be great. I looked through
> the pandas code, but had trouble understanding it, as it involves
> subclassing Extension in a pretty complicated way.
>
> 2) Can I have the generated .cpp files sit in some subdirectory, such as
> src, rather than in the main package folder? What's the best practice for
> directory structure in general?
>
> 3) What's the best way to setup other packages that a main package depends
> on? Are general-purpose build tools such as make the best system for this?
> Also, I found that distutils' "clean" wasn't wiping the Cython-generated cpp
> files, which was further incentive to use make.
>
> 4) I wanted to name two modules "io" and "state", but this caused a
> namespace clash. Is there a good work-around, or should I settle for
> renaming the modules?
>
> Thanks for any advice,
> Matt.
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Chris Barker - NOAA Federal]

On Thu, Jan 31, 2013 at 4:01 AM, Matthew Honnibal <honn...@gmail.com> wrote:

> I've been using Cython as the main development language for a small project
> (an English statistical parser), about 2000 lines of code. Currently my
> build system is an ugly mix of make and setup scripts, and I'm confused
> about how to best build a package that has several Cython modules.

yeah, I'd try to clean that up!

> 1) When I have multiple Cython modules in a single package, can I use a
> single setup script?

yes.

> What if one of the modules uses different C/C++
> extensions?

yes.

How are you linking the C++ == where it got tricky for us was we had a
bunch of C++ that many of the cython modules have to link to -- we
found that we had to do this quite differently on Windows and OS-X
(haven't tested, but I suspect Linux would be like OS-X)

> If someone has an example
> package with multiple Cython modules that would be great.

enclosed -- a bit ugly, but it all gets built with one setup.py
(except he Windows DLL, which is build independently, though called
from the setup.py)

> the pandas code, but had trouble understanding it, as it involves
> subclassing Extension in a pretty complicated way.

we avoided that.

> 2) Can I have the generated .cpp files sit in some subdirectory, such as
> src, rather than in the main package folder?

I don't think so -- at least not easily.

> What's the best practice for
> directory structure in general?

for Cython, the *.pyx, *.pyd files in dirs that mimic your package
stucture -- jsut like python files would (even though there are more
of them) and the generated *.c or *.cpp get laid in next to them.

> 3) What's the best way to setup other packages that a main package depends
> on? Are general-purpose build tools such as make the best system for this?

if you mean python/cython packages, then, no I wouldn't use make --
stick with distutils. If you mean other, non-python dependencies, then
whatever the native system is (i.e. make on *nix)

> Also, I found that distutils' "clean" wasn't wiping the Cython-generated cpp
> files, which was further incentive to use make.

yeah, distuils "clean" is not so smart -- we hand-wrote a "cleanall"
for that reason -- hacky, but pretty easy.

> 4) I wanted to name two modules "io" and "state", but this caused a
> namespace clash. Is there a good work-around, or should I settle for
> renaming the modules?

what was the clash? If you put them in a package, it shouldn't be an
issue if they match std lib names. Though you probably don't want to
re-use common stdlib names anyway.

HTH,
-Chris


--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

[Robert Bradshaw]

On Thu, Jan 31, 2013 at 4:01 AM, Matthew Honnibal <honn...@gmail.com> wrote:

> Hi all,

>
> I've been using Cython as the main development language for a small project
> (an English statistical parser), about 2000 lines of code. Currently my
> build system is an ugly mix of make and setup scripts, and I'm confused
> about how to best build a package that has several Cython modules.
>

> My project has three packages, all of which have cython code, and additional
> C/C++ code. One of them has several Cython modules.
>
> Some questions:
>

> 1) When I have multiple Cython modules in a single package, can I use a
> single setup script?

Yes. Sage is probably the largest Cython project out there (several
hundred thousand lines of Cython code) and we have a single setup
script (though we split the list of extensions into a separate file
and do some extra work to build things in parallel, so it's not the
easiest example to follow.)

> What if one of the modules uses different C/C++ extensions?

Not quite following you here...

> I tried just listing several modules in ext_modules, but this
> didn't work immediately, and to be honest distutils confuses me, so I
> settled for a hacky solution to get it working.

What was the issue? Did cythonize(...)
http://wiki.cython.org/enhancements/distutils_preprocessing not work
for you?

> If someone has an example
> package with multiple Cython modules that would be great.

Most projects I know of have multiple Cython modules.

> I looked through

> the pandas code, but had trouble understanding it, as it involves
> subclassing Extension in a pretty complicated way.
>

> 2) Can I have the generated .cpp files sit in some subdirectory, such as

> src, rather than in the main package folder? What's the best practice for
> directory structure in general?

No, this is not supported. The best practice is to lay things out just
as they would be in Python (with .py, .pyc, .so, .dll, .c[pp], .pyx,
.pxd files sitting in the directory structure according to the
packages they belong to).

> 3) What's the best way to setup other packages that a main package depends
> on? Are general-purpose build tools such as make the best system for this?

Python package management is a bit of a mess, but rolling your own is
probably worse.

> Also, I found that distutils' "clean" wasn't wiping the Cython-generated cpp
> files, which was further incentive to use make.

Make is probably more pain then it's worth.

> 4) I wanted to name two modules "io" and "state", but this caused a
> namespace clash. Is there a good work-around, or should I settle for
> renaming the modules?
>