Re: [cython-users] Best practice for using pkg-config

[Robert Bradshaw]

On Thu, May 23, 2013 at 2:05 PM, Wichert Akkerman <wic...@wiggy.net> wrote:
> Is there a best practice for using pkg-config? I was trying to use cythonize
> since the documentation suggests this is the current preferred method, but
> there doesn't appear to be any API documentation listing its parameters. The
> only working thing I could come up with is to manually modify the extensions
> it returns:
>
> from distutils.core import setup
> import subprocess
> from Cython.Build import cythonize
>
> flags = subprocess.check_output(['pkg-config', '--cflags-only-I', 'pkgA
> pkgB'])
> include_dirs = [flag[2:] for flag in flags.split()]
> flags = subprocess.check_output(['pkg-config', '--libs-only-L', 'pkgA
> pkgB'])
> library_dirs = [flag[2:] for flag in flags.split()]
>
> extensions = cythonize('*.pyx')
> for ext in extensions:
> ext.include_dirs.extend(include_dirs)
> ext.library_dirs.extend(library_dirs)
>
> setup(name='mypkg', ext_modules=extensions)
>
> This feels unnecessarily complex and definitely is not readable. It would be
> very nice if you could do this:
>
> setup(
> name='mypkg',
> ext_modules=cythonize('*.pyx', pkg_config=['pkgA', 'pkgB']),
> )
>
> Or perhaps like this in a .pyx file:
>
> # distutils: language = c++
> # distutils: pkg_config = pkgA pkgB

Perhaps you should do

extensions = cythonize(Extension('*.pyx', [arbitrary Extension arguments]))

- Robert

[Wichert Akkerman]

Thanks, that is at least a bit more readable. Did I miss any documentation which states cythonize takes Extension arguments, or is that an undocumented feature?

Wichert.

[Robert Bradshaw]

Cythonize is severely undocumented. It's on my todo list, but I won't
complain if someone beats me to it :).

- Robert

[Wichert Akkerman]

I'ld be happy to submit the occasional pull request with things I find if that helps.

Wichert.

[Robert Bradshaw]

That would be great. The sources are at
https://github.com/cython/cython/blob/master/docs/src/reference/compilation.rst

[Wichert Akkerman]

Done: https://github.com/cython/cython/pull/234

W.

[Nikita Nemkin]

On Fri, 24 May 2013 13:26:47 +0600, Wichert Akkerman <wic...@wiggy.net>

wrote:

> On May 24, 2013, at 09:19 , Robert Bradshaw <robe...@gmail.com> wrote:
>> On Thu, May 23, 2013 at 11:53 PM, Wichert Akkerman <wic...@wiggy.net>
>> wrote:
>>>
>>> On May 24, 2013, at 08:52 , Robert Bradshaw <robe...@gmail.com> wrote:
>>>> On Thu, May 23, 2013 at 11:25 PM, Wichert Akkerman
>>>> <wic...@wiggy.net> wrote:
>>>>>
>>>>> Thanks, that is at least a bit more readable. Did I miss any
>>>>> documentation which states cythonize takes Extension arguments, or
>>>>> is that an undocumented feature?
>>>>
>>>> Cythonize is severely undocumented. It's on my todo list, but I won't
>>>> complain if someone beats me to it :).
>>>
>>> I'ld be happy to submit the occasional pull request with things I find
>>> if that helps.
>>
>> That would be great. The sources are at
>> https://github.com/cython/cython/blob/master/docs/src/reference/compilation.rst
>
> Done: https://github.com/cython/cython/pull/234

It does not quite work like that. cythonize kwargs are Cython options,
not extension options. To pass extension options you have to create
Extension objects explicitly.

cythonize API is pretty confusing. It seems to target specific
use case (Sage?), instead of more common use cases.


Best regards,
Nikita Nemkin

[Wichert Akkerman]

On May 24, 2013, at 13:20 , "Nikita Nemkin" <nik...@nemkin.ru> wrote:
> On Fri, 24 May 2013 13:26:47 +0600, Wichert Akkerman <wic...@wiggy.net> wrote:
>>
>> Done: https://github.com/cython/cython/pull/234
>
> It does not quite work like that. cythonize kwargs are Cython options,
> not extension options. To pass extension options you have to create
> Extension objects explicitly.

I'm just going with what Robert said:

extensions = cythonize(Extension('*.pyx', [arbitrary Extension arguments]))

which seems to work at least for include path. Perhaps those happen to match the Cython options in this case? If so, where can I find the list of valid Cython options?

Wichert.

[Robert Bradshaw]

On Fri, May 24, 2013 at 5:35 AM, Wichert Akkerman <wic...@wiggy.net> wrote:
>
> On May 24, 2013, at 13:20 , "Nikita Nemkin" <nik...@nemkin.ru> wrote:
>> On Fri, 24 May 2013 13:26:47 +0600, Wichert Akkerman <wic...@wiggy.net> wrote:
>>>
>>> Done: https://github.com/cython/cython/pull/234
>>
>> It does not quite work like that. cythonize kwargs are Cython options,
>> not extension options. To pass extension options you have to create
>> Extension objects explicitly.
>
> I'm just going with what Robert said:
>
> extensions = cythonize(Extension('*.pyx', [arbitrary Extension arguments]))
>
> which seems to work at least for include path. Perhaps those happen to match the Cython options in this case?

Correct.

> If so, where can I find the list of valid Cython options?

In the code.

- Robert

[Robert Bradshaw]

Yep. It should work for the (very) common case as well, but apparently
there's something missing in between?

The original proposal is up at
http://wiki.cython.org/enhancements/distutils_preprocessing , but
feedback is still welcome.

- Robert

[Nikita Nemkin]

On Sat, 25 May 2013 01:24:11 +0600, Robert Bradshaw <robe...@gmail.com>
wrote:

> On Fri, May 24, 2013 at 5:35 AM, Wichert Akkerman <wic...@wiggy.net>
> wrote:
>>
>> On May 24, 2013, at 13:20 , "Nikita Nemkin" <nik...@nemkin.ru> wrote:
>>> On Fri, 24 May 2013 13:26:47 +0600, Wichert Akkerman
>>> <wic...@wiggy.net> wrote:
>>>>
>>>> Done: https://github.com/cython/cython/pull/234
>>>
>>> It does not quite work like that. cythonize kwargs are Cython options,
>>> not extension options. To pass extension options you have to create
>>> Extension objects explicitly.
>>
>> I'm just going with what Robert said:
>>
>> extensions = cythonize(Extension('*.pyx', [arbitrary Extension
>> arguments]))
>>
>> which seems to work at least for include path. Perhaps those happen to
>> match the Cython options in this case?

This code:

from distutils.core import setup
from Cython.Build import cythonize

ext_modules = cythonize(
'test.pyx',
include_dirs=['TEST_INCLUDE'],
library_dirs=['TEST_LIB'])

print ext_modules[0].include_dirs, ext_modules[0].library_dirs

produces two empty lists.

cythonize does have "include_path", but it is not related to Extension's
"include_dirs".


Best regards,
Nikita Nemkin

[Wichert Akkerman]

Ok, so trying to trace that: cythonize takes a **option parameter, which it uses to create two CompilationOptions instances (one for C and one for C++, but completely identical in all other respects). CompilationOptions is documented (in its docstring) to take include_path, but no linking options other compiler flags at all. I've updated my pull request to reflect this. I hope it is accurate now.

Wichert.

[Robert Bradshaw]

Looks good; thanks!

[Stefan Behnel]

Am 24.05.2013 08:25, schrieb Wichert Akkerman:
> On May 24, 2013, at 00:45, Robert Bradshaw wrote:

As Robert noted, it's lacking documentation. Another major feature here,
besides support for passing options, is that you can do

extensions = [ Extension(
sources=["module.pyx" if USE_CYTHON else "module.c"],
**more_opts)
)]

if USE_CYTHON:
extensions = cythonize(extensions)

setup(...)

That makes it very easy to ship pre-compiled (and tested!) C sources in
source releases.

Stefan

[Robert Bradshaw]

Yep, or even something like

if USE_CYTHON:
from Cython.Build import cythonize
else:
def cythonize(extensions):
for ext in extensions:
if ext.sources[0].endswith('.pyx'):
ext.sources[0] = ext.sources[0][:-3] + 'c'

- Robert

[Nikita Nemkin]

On Thu, 30 May 2013 08:30:20 +0600, Robert Bradshaw <robe...@gmail.com>
wrote:

Wy not pass .c/.cpp filenames to cythonize and let it find corresponding
.pyx files by itself? Then double branching could be avoided.

I think it won't even break backwards compatibility.

Recently added build_dir option may need some special handling though.


Best regards,
Nikita Nemkin