Re: [cython-users] Cython and defines

[Robert Bradshaw]

On Tue, Jul 17, 2012 at 7:52 AM, Dan Speed <d...@ccpgames.com> wrote:
> I'm dealing with an issue where I want to compile a cython module for a
> number of different environments, most being pure python, but one or two
> being an embedded python interpreter (32/64)
> In the case where I'm running in the embedded python interpreter, there are
> specific classes / functions that I need to use to do things like access
> files / hook into performance monitoring / custom allocators
>
> I'd like be able to write a single cython file and pass in a compiler define
> to determine how to behave under the hood, but I can't find any mention of
> how to do this.
> Does Cython also need to know about the defines that will be used in
> compilation when reading a header file?

You can do

cdef extern from "header.h":
int MACRO_VALUE
int MAYBE_DEFINED_VALUE "defined(MAYBE_DEFINED_VALUE)"

and then write

if MACRO_VALUE:
...
if MAYBE_DEFINED_VALUE:
...

which any self-respecting optimizing C compiler will optimize away.

It gets trickier to use functions only available on a single platform,
in that case you may need to provide empty dummy declarations for the
case where they are not defined.

- Robert

>
> what I'd like to do is:
> <myModule.pyx>
>
> IF EMBEDDED_ENVIRONMENT:
> # cdef extern ...
> # hook up environment specific stuff
>
> def MyCythonFunction( ... ):
> IF EMBEDDED_ENVIRONMENT:
> OpenFileFromPackedResource( ... )
>
> My module now looks the same in all cases to python but can play nice with
> platform specific requirements.
>
> Thanks in advance!

[Denis Bilenko]

On Tue, Jul 17, 2012 at 6:52 PM, Dan Speed <d...@ccpgames.com> wrote:
> I'm dealing with an issue where I want to compile a cython module for a
> number of different environments, most being pure python, but one or two
> being an embedded python interpreter (32/64)

> My module now looks the same in all cases to python but can play nice with
> platform specific requirements.

I had the same problem in gevent (http://gevent.org/) and ended up
writing a wrapper around Cython that adds C-like preprocessor
capability to it. It's a bit limited but works well for my case.

Here's what it can process:
https://bitbucket.org/denis/gevent/src/8c9722b1fc2a/gevent/core.ppyx#cl-227
And here's the script:
https://bitbucket.org/denis/gevent/src/8c9722b1fc2a/util/cythonpp.py

(Previous versions used to depend on unifdef, this one only needs Cython.)

[Robert Bradshaw]

This works fine if one wants to re-generate the .c file for every
platform (and in that case you can use IF and DEF if need be as well).

- Robert

[mark florisson]

On 17 July 2012 20:04, Robert Bradshaw <robe...@gmail.com> wrote:
> On Tue, Jul 17, 2012 at 7:52 AM, Dan Speed <d...@ccpgames.com> wrote:
>> I'm dealing with an issue where I want to compile a cython module for a
>> number of different environments, most being pure python, but one or two
>> being an embedded python interpreter (32/64)
>> In the case where I'm running in the embedded python interpreter, there are
>> specific classes / functions that I need to use to do things like access
>> files / hook into performance monitoring / custom allocators
>>
>> I'd like be able to write a single cython file and pass in a compiler define
>> to determine how to behave under the hood, but I can't find any mention of
>> how to do this.
>> Does Cython also need to know about the defines that will be used in
>> compilation when reading a header file?
>
> You can do
>
> cdef extern from "header.h":
> int MACRO_VALUE
> int MAYBE_DEFINED_VALUE "defined(MAYBE_DEFINED_VALUE)"
>
> and then write
>
> if MACRO_VALUE:
> ...
> if MAYBE_DEFINED_VALUE:
> ...

Does this actually work? defined() only works in preprocessor
#if/#elif statements.

[Robert Bradshaw]

D'oh, you're right. You'd have to have a separate header file with
something like

#ifdef MAYBE_DEFINED_VALUE
#define MAYBE_DEFINED_VALUE_IS_DEFINED 1
#else
#define MAYBE_DEFINED_VALUE_IS_DEFINED 0
#endif

Unless you did something extremely hackish like knowing it was always
defined to be a non-zero integer and wrote

int MAYBE_DEFINED_VALUE "(MAYBE_DEFINED_VALUE - 0 == 0)"

- Robert

[Denis Bilenko]

On Wed, Jul 18, 2012 at 12:59 AM, Robert Bradshaw <robe...@gmail.com> wrote:
>> Here's what it can process:
>> https://bitbucket.org/denis/gevent/src/8c9722b1fc2a/gevent/core.ppyx#cl-227
>> And here's the script:
>> https://bitbucket.org/denis/gevent/src/8c9722b1fc2a/util/cythonpp.py
>>
>> (Previous versions used to depend on unifdef, this one only needs Cython.)
>
> This works fine if one wants to re-generate the .c file for every
> platform (and in that case you can use IF and DEF if need be as well).

This generates a single .c for all platforms. I'm not sure how you can
use IF and DEF for that.

[Robert Bradshaw]

You can't.

What I would do is write a single, small .c file that abstracts away
all the platform dependent details, as that's typically hard to do at
any time other than C compile time. In your case that might be as
simple as creating some empty methods for the non-embedded case to
"call." Then I'd write the Cython wrapper against this.

- Robert

[Denis Bilenko]

On Thu, Jul 19, 2012 at 9:35 AM, Robert Bradshaw <robe...@gmail.com> wrote:
>> This generates a single .c for all platforms. I'm not sure how you can
>> use IF and DEF for that.
>
> You can't.

Yes, that's the difference. I run cythonpp.py once when making the
release tarball. It produces a .c file (just like regular Cython
does). The users installing gevent package do not need neither
cythonpp.py nor Cython to build gevent.

>
> What I would do is write a single, small .c file that abstracts away
> all the platform dependent details, as that's typically hard to do at
> any time other than C compile time. In your case that might be as
> simple as creating some empty methods for the non-embedded case to
> "call." Then I'd write the Cython wrapper against this.

When it's possible to do so, it's great. Some functionality cannot be
wrapped though. For instance, you cannot provide a wrapper for fork()
on Windows. In such case, you simply remove corresponding
functionality from the Python module. In gevent, some functionality is
not available on Windows and it looks like this in the source:
https://bitbucket.org/denis/gevent/src/8c9722b1fc2a/gevent/core.ppyx#cl-419

Another case where wrappers won't help is when you need Cython classes
have slightly different definition on different platforms. In gevent,
we have

- object fields that is only available on some platforms:
https://bitbucket.org/denis/gevent/src/8c9722b1fc2a/gevent/core.ppyx#cl-227
- function signature slightly different between platforms:
https://bitbucket.org/denis/gevent/src/8c9722b1fc2a/gevent/core.ppyx#cl-666

[Robert Bradshaw]

On Mon, Jul 23, 2012 at 1:09 AM, Denis Bilenko <denis....@gmail.com> wrote:
> On Thu, Jul 19, 2012 at 9:35 AM, Robert Bradshaw <robe...@gmail.com> wrote:
>>> This generates a single .c for all platforms. I'm not sure how you can
>>> use IF and DEF for that.
>>
>> You can't.
>
> Yes, that's the difference. I run cythonpp.py once when making the
> release tarball. It produces a .c file (just like regular Cython
> does). The users installing gevent package do not need neither
> cythonpp.py nor Cython to build gevent.

Yep. This is pretty typical (we try hard to make a single .c file that
can compile in a wide variety of environments) and one of the defects
of using IF/DEF.

So, if I understand correctly, cythonpp.py propagates #ifdef et al. to
the .c file. I can see how that would be handy.

>> What I would do is write a single, small .c file that abstracts away
>> all the platform dependent details, as that's typically hard to do at
>> any time other than C compile time. In your case that might be as
>> simple as creating some empty methods for the non-embedded case to
>> "call." Then I'd write the Cython wrapper against this.
>
> When it's possible to do so, it's great. Some functionality cannot be
> wrapped though. For instance, you cannot provide a wrapper for fork()
> on Windows. In such case, you simply remove corresponding
> functionality from the Python module. In gevent, some functionality is
> not available on Windows and it looks like this in the source:
> https://bitbucket.org/denis/gevent/src/8c9722b1fc2a/gevent/core.ppyx#cl-419

Yes, but defining a child() and install_sigchld() that raise an error
on Windows and guarding against/avoiding it at runtime will allow you
to create a .c file that still compiles everywhere.

> Another case where wrappers won't help is when you need Cython classes
> have slightly different definition on different platforms. In gevent,
> we have
>
> - object fields that is only available on some platforms:
> https://bitbucket.org/denis/gevent/src/8c9722b1fc2a/gevent/core.ppyx#cl-227
> - function signature slightly different between platforms:
> https://bitbucket.org/denis/gevent/src/8c9722b1fc2a/gevent/core.ppyx#cl-666

Personally, I'd rather not expose platform-dependent function
signature differences to the Python API; I'd probably use
__gettattribute__ and optional arguments with runtime checks (and
informative messages) for unsupported features. However, it looks like
you're aiming for a fairly low-level wrapper.

- Robert

[Denis Bilenko]

On Mon, Jul 23, 2012 at 8:01 PM, Robert Bradshaw <robe...@gmail.com> wrote:
> Yep. This is pretty typical (we try hard to make a single .c file that
> can compile in a wide variety of environments) and one of the defects
> of using IF/DEF.
>
> So, if I understand correctly, cythonpp.py propagates #ifdef et al. to
> the .c file. I can see how that would be handy.

Yes, that's what it produces. What it actually does is quite simple:
it generates a Cython source (post-preprocesor) for each possible
configuration and then runs Cython on each of those sources. (which is
slow - as it is exponential to the number of preprocessor variables in
use, but what else you are going to do if you really need that
self-sufficient .c source with platform specific stuff?).

Then it combines all that individual .c files in a single .c file,
generating equivalent of this:

#if ... configuration1 ...
cython output for configuration1
#elif .. configurtation 2 ...
cython output for configuration 2
...
#endif

I said equivalent, because it tries to be smart and excludes common
parts from inside #if.

>> When it's possible to do so, it's great. Some functionality cannot be
>> wrapped though. For instance, you cannot provide a wrapper for fork()
>> on Windows. In such case, you simply remove corresponding
>> functionality from the Python module. In gevent, some functionality is
>> not available on Windows and it looks like this in the source:
>> https://bitbucket.org/denis/gevent/src/8c9722b1fc2a/gevent/core.ppyx#cl-419
>
> Yes, but defining a child() and install_sigchld() that raise an error
> on Windows and guarding against/avoiding it at runtime will allow you
> to create a .c file that still compiles everywhere.

True, but, arguably, not providing the function/class at all is nicer.
For example, os.fork is not available on Windows, as well as many
other platform-specific functions/classes in Python stdlib.

This allows you to find out that certain functionality is not
available earlier rather than later, e.g.

try:
fork = os.fork
except AttributeError:
fork = None

if fork:
...
else:
...

Cheers,
Denis.

[Dan Speed]

For me, there's also just the issue that hiding the platform specific differences in yet anothyer layer of abstraction code can simply obfuscate thins and increase the maintenance burden. If people can see that there's only any memory tracking hooked up on the embedded version when they read the pyx, it's immediately obvious, while if I hide that, then they have to understand 2 bits of code to know why they can't dump a memory report in pure python.

We're providing tools libraries for automated machines and users, or libraries that are precompiled and shipped with our game. If I want to use cython, then I need to hook it into our auto-builder for all those people who aren't going to have gcc/MinGW/MSVC, and as a packaged artifact for our game build process. I simply can't hide the fact that the environment that we run our game in is completely different, because open() doesn't work on packaged game resources, among many other differences. I also want/need to drive the cython compiler in a different way from most people who are trying to distutils compile a module for their particular python environment, because I have to deal very carefully with VC90 manifests, signing, PDB symbol servers, build directories, different include and lib paths for our embedded python, and that doesn't even begin to consider building things for the PS3!

[Robert Bradshaw]

Really, you're just pushing the burden from yourself to your users.
Which may be OK, but the platform incompatibilities will have to be
dealt with at some point. For this example, you could create very
nice, explanatory errors in your wrapper. For open(), you could
provide a higher-level function that does the "right" thing (whatever
that is) in all contexts. It all depends on what you're trying to
provide.

> We're providing tools libraries for automated machines and users, or
> libraries that are precompiled and shipped with our game. If I want to use
> cython, then I need to hook it into our auto-builder for all those people
> who aren't going to have gcc/MinGW/MSVC, and as a packaged artifact for our
> game build process. I simply can't hide the fact that the environment that
> we run our game in is completely different, because open() doesn't work on
> packaged game resources, among many other differences. I also want/need to
> drive the cython compiler in a different way from most people who are trying
> to distutils compile a module for their particular python environment,
> because I have to deal very carefully with VC90 manifests, signing, PDB
> symbol servers, build directories, different include and lib paths for our
> embedded python, and that doesn't even begin to consider building things for
> the PS3!

If things are very different from platform to, you could consider
splitting things out into separate modules, the way os.path is done.
Or you perhaps something like cythonpp.py is right in your case.

- Robert

[Dan Speed]

We don't support writing to the runtime resource packages (which is prevented by the UAC anyway), so the abstraction would either conceal something that's only going to work in some cases (ie. open with mode 'w' suddenly throws exceptions once we make a full build), or would remove functionality that you need during the development process.

If I see someone using open() in client/server code, I can very quickly identify that they're probably doing it wrong. If they use a normal filepath in our resfiles, they're also doing it wrong. Obfuscating it merely helps to conceal errors in assumption about the environment or a reduction to the minimum mutually supported feature set, which are substantive and important. Our programmers are paid a salary to have to deal with some of these things.

While I understand your point that abstracting platform specific peculiarities away is useful, this is what we do by providing a more limited interface that is guaranteed to work correctly on all platforms that is clearly the one that you're supposed to use in a particular context, rather than having an abstraction with parts that work inconsistently in different environments, which you have to wait to have a build to see if it behaves correctly in that particular environment.

[Robert Bradshaw]

On Tue, Jul 24, 2012 at 6:09 AM, Dan Speed <d...@ccpgames.com> wrote:
> We don't support writing to the runtime resource packages (which is
> prevented by the UAC anyway), so the abstraction would either conceal
> something that's only going to work in some cases (ie. open with mode 'w'
> suddenly throws exceptions once we make a full build), or would remove
> functionality that you need during the development process.

I would consider moving such functionality to a clearly marked
"development-only" module.

> If I see someone using open() in client/server code, I can very quickly
> identify that they're probably doing it wrong. If they use a normal filepath
> in our resfiles, they're also doing it wrong. Obfuscating it merely helps to
> conceal errors in assumption about the environment or a reduction to the
> minimum mutually supported feature set, which are substantive and important.
> Our programmers are paid a salary to have to deal with some of these things.

Fair enough, but as a salaried tool producer and consumer I bet they
wouldn't complain if you made it even easier. :-)

> While I understand your point that abstracting platform specific
> peculiarities away is useful, this is what we do by providing a more limited
> interface that is guaranteed to work correctly on all platforms that is
> clearly the one that you're supposed to use in a particular context, rather
> than having an abstraction with parts that work inconsistently in different
> environments, which you have to wait to have a build to see if it behaves
> correctly in that particular environment.

Clearly I can only speak in abstractions, as I only have scant
information about the project you're working on. I was thinking the
problem you were trying to solve was "My module now looks the same in

all cases to python but can play nice with platform specific

requirements." In particular, I thought you wanted the Python-level
API to map to several possible C APIs (providing the same high-level
behavior) depending on the compile-time environment. This is why I
suggested a C-written shim to allow these decisions to be more easily
and explicitly made at C compile time (and conditionally invoke
functions that only exist on certain platforms).

It sounds like you're trying to do this with your limited interface.

I agree that writing an abstraction that works inconsistently is a
horrible idea. It's almost the definition of leaky and broken.

- Robert