Conditional compilation of Cython code / preprocessor / external conditions

[Yury V. Zaytsev]

Hi folks,

Is it possible to conditionally compile Cython code depending on some
externally defined value?

From reading the documentation, I've got an impression that presently
Cython doesn't have any kind of preprocessor, and I hope that I'm just
missing something here:

http://docs.cython.org/src/userguide/language_basics.html

My use case is as follows:

The application that I'm wrapping links against a number of libraries,
and some might be unavailable at compile time, so the corresponding
parts of the Cython code shouldn't be compiled either.

During the build, a preprocessor constant is defined, such as
HAVE_LIBFOO that I can get by including config.h or pass via the
autotools build system to the external tools. However, I can't figure
out how to use that from Cython...

So far, I was able to come up with an idea of replicating the API of the
missing libraries with dummies on the C++ level, such that the Cython
code compiles either way.

Or else, I can make a poor man's preprocessor based on sed and just
strip parts marked for conditional compilation from the PYX files.

Both solutions look really disgusting and unmaintainable. I would
greatly appreciate better ideas! To me it sounds like a rather common
problem, so somebody must have come up with a solution already...

Thanks,

--
Sincerely yours,
Yury V. Zaytsev

[Nikita Nemkin]

On Mon, 22 Jul 2013 20:17:58 +0600, Yury V. Zaytsev <yu...@shurup.com>
wrote:

> Hi folks,
>
> Is it possible to conditionally compile Cython code depending on some
> externally defined value?
>
> From reading the documentation, I've got an impression that presently
> Cython doesn't have any kind of preprocessor, and I hope that I'm just
> missing something here:

See
http://docs.cython.org/src/userguide/language_basics.html#conditional-statements
"Conditional Statements", these are Cython "macros".

You can discover the values and set Cython compile constants like this:

cythonize(
[...],
compile_time_env={'PY_VERSION_HEX': sys.hexversion})


Best regards,
Nikita Nemkin

[Yury V. Zaytsev]

On Mon, 2013-07-22 at 20:43 +0600, Nikita Nemkin wrote:
>
> You can discover the values and set Cython compile constants like
> this:

Hi Nikita,

I'm using the 'cython' command line tool to compile my Cython files into
C++ files using an external build system instead of distutils, because
it's much easier to integrate in the build process.

I couldn't find any way of defining macros on the command line, so I
posted a message to the developer list to see if such functionality is
going to be accepted into Cython...

[Nikita Nemkin]

On Mon, 22 Jul 2013 21:29:34 +0600, Yury V. Zaytsev <yu...@shurup.com>
wrote:

cythonize is not a distutils command, it's a standalone function
providing an entry point to the Cython compiler.
Just write a short script that imports and calls cythonize(),
parsing command line arguments however you like.


Best regards,
Nikita Nemkin

[Robert Bradshaw]

There was a lot of talk about pre-processing/macros/etc. for Cython at
Cython days 1, and the conclusion was that there wasn't a clear right
answer and doing something half-way based on textual substitution was
not where we wanted to head. Instead, we should make it really easy to
us whatever pre-processor you want (e.g. gcc -E or m4 or a general
templating engine) before invoking cython itself. Instead of writing
pyx files, write pyx.in (or whatever) files and make conversion from
pyx.in to pyx part of the build process yourself. It would probably be
worth adding this option to the cythonize command (e.g. it would take
an arbitrary command to invoke on each input file before the parsing
phase).

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Yury V. Zaytsev]

On Mon, 2013-07-22 at 21:37 +0600, Nikita Nemkin wrote:
>
> cythonize is not a distutils command, it's a standalone function
> providing an entry point to the Cython compiler. Just write a short
> script that imports and calls cythonize(), parsing command line
> arguments however you like.

Hi Nikita,

After I've got a word from Robert that there is no interest in providing
macro values via the command line, and some faffing around the PXI files
(*), I decided to give your solution a go and came up with the following
unsavory make target:

pynestkernel.cpp: pynestkernel.pyx pynestkernel.pxd
cd $(top_srcdir)/pynest; $(PYTHON) -c \
"from Cython.Build import cythonize; cythonize('pynestkernel.pyx', \
verbose=1, emit_linenums=1, language='c++', cplus=1, output_file='pynestkernel.cpp',\
compiler_directives={ \
'warn.maybe_uninitialized': True, \
'warn.unreachable': True, \
'warn.unused': True, \
}, \
compile_time_env={'HAVE_LIBNEUROSIM': $(DEF_LIBNEUROSIM)}, \
)"

However, no matter what I tried, cythonize from Cython 0.19.1 always
generates pynestkernel.c instead of pynestkernel.cpp as I want it to.

Do you have any ideas on how to tackle this problem?

Also, I wasn't about to find any canonical reference that lists all
arguments that cythonize accepts other than the source code. Do you
happen to know if there is one and, in any case, how do I properly
enable all warnings?

(*) PXI files turned out to be a PITA, because it's not clear which
target they should depend upon to be properly updated, and if you
declare them as .PHONY, dependent targets such as Cython + the library
itself will be always triggered unconditionally.

Many thanks,

[Robert Bradshaw]

Cythonize compiles a list of extensions, some of which may be C and
some C++, so there's no global option. You could try

from Cython.Build import cythonize
from distutils.extension import Extension

cythonize(
[Extension('pynestkernel.pyx', language = 'c++')],
compile_time_env=...)

IMHO, it'd be cleaner to put this in a script rather than inline in a makefile.

[Yury V. Zaytsev]

On Thu, 2013-07-25 at 21:30 -0700, Robert Bradshaw wrote:
>
>
> Cythonize compiles a list of extensions, some of which may be C and
> some C++, so there's no global option. You could try

Hi Robert,

Many thanks, this works!

Additionally, it made me think that I can try putting the "# distutils:
language" pragma directly in the source file, and not only this works
too, but also I'm even happier with this solution.

> IMHO, it'd be cleaner to put this in a script rather than inline in a
> makefile.

Well, please allow me to respectfully disagree.

Proliferation of template and script files with only several lines of
code is a significant problem when you are working on a large project,
of which Cython code is only a small subproject.

This is especially true if the project's build system is based on
autotools, and has to work on all kinds of weird platforms and with all
kinds of strange compilers, support cross-compilation and VPATH builds.

It's much easier and way more manageable to use Cython's built-in macro
expansion features and encode everything in the automake templates, as
compared to using m4 or scripts that you will need to ship with `make
dist`, make sure they are called correctly on VPATH builds, etc.

To me this looks safe, compact and perfectly manageable:

pynestkernel.cpp: pynestkernel.pyx pynestkernel.pxd
cd $(top_srcdir)/pynest; $(PYTHON) -c \
"from Cython.Build import cythonize; \

cythonize('pynestkernel.pyx', verbose=1, emit_linenums=1, \
compile_time_env={'HAVE_LIBNEUROSIM': $(CYTHON_LIBNEUROSIM)}, \
)"

If only there was *one* option equivalent to -Wall, I'd be completely
happy with this.

YMMV, of course...

[Robert Bradshaw]

On Fri, Jul 26, 2013 at 12:49 AM, Yury V. Zaytsev <yu...@shurup.com> wrote:
> On Thu, 2013-07-25 at 21:30 -0700, Robert Bradshaw wrote:
>>
>>
>> Cythonize compiles a list of extensions, some of which may be C and
>> some C++, so there's no global option. You could try
>
> Hi Robert,
>
> Many thanks, this works!
>
> Additionally, it made me think that I can try putting the "# distutils:
> language" pragma directly in the source file, and not only this works
> too, but also I'm even happier with this solution.

Yes, this is even better.

>> IMHO, it'd be cleaner to put this in a script rather than inline in a
>> makefile.
>
> Well, please allow me to respectfully disagree.
>
> Proliferation of template and script files with only several lines of
> code is a significant problem when you are working on a large project,
> of which Cython code is only a small subproject.
>
> This is especially true if the project's build system is based on
> autotools, and has to work on all kinds of weird platforms and with all
> kinds of strange compilers, support cross-compilation and VPATH builds.
>
> It's much easier and way more manageable to use Cython's built-in macro
> expansion features and encode everything in the automake templates, as
> compared to using m4 or scripts that you will need to ship with `make
> dist`, make sure they are called correctly on VPATH builds, etc.
>
> To me this looks safe, compact and perfectly manageable:
>
> pynestkernel.cpp: pynestkernel.pyx pynestkernel.pxd
> cd $(top_srcdir)/pynest; $(PYTHON) -c \
> "from Cython.Build import cythonize; \
> cythonize('pynestkernel.pyx', verbose=1, emit_linenums=1, \
> compile_time_env={'HAVE_LIBNEUROSIM': $(CYTHON_LIBNEUROSIM)}, \
> )"

Perhaps it would be nice to provide a "cythonize" cli command that had
as-close-to-possible semantics as the cythonize(...) function. One
could then write.

pynestkernel.cpp: pynestkernel.pyx pynestkernel.pxd
cythonize --verbose=1 --emit_linenums=True
--compile_time_env="{'HAVE_LIBNEUROSIM': $(CYTHON_LIBNEUROSIM)}"
pynestkernel.pyx

Unfortunately many of the arguments are not simple values.

> If only there was *one* option equivalent to -Wall, I'd be completely
> happy with this.

That certainly seems like a reasonable request.

> YMMV, of course...
>
> --
> Sincerely yours,
> Yury V. Zaytsev
>
>

[Robert Bradshaw]

OK, now foo.all will set foo.* to the given value, assuming foo.all is
not already a directive.

https://github.com/cython/cython/commit/ce36ba4b40a6cdbe241c814c4a44256f1024be8c