Cython .pxd introspection: listing defined constants
W. Trevor King
cdef extern from 'mylib.h'
enum: CONST_A
enum: CONST_B
...
where I declare each constant macro from the library's header `mylib.h`:
#define CONST_A 1
#define CONST_B 2
...
Now I want to expose those constants in Python, so I have `expose.pyx`:
cimport mylib
CONST_A = mylib.CONST_A
CONST_B = mylib.CONST_B
...
But the last part seems pretty silly. I'd like to do something like
cimport mylib
import sys
for name in dir(mylib):
setattr(sys.modules[__name__], name, getattr(mylib, name))
which compiles fine, but fails to import with
$ python -c 'import myproject.expose'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "expose.pyx", line 4, in init myproject.expose (myproject/expose.c:515)
NameError: mylib
The relevant snippet of expose.c is:
__pyx_t_1 = __Pyx_GetName(__pyx_m, __pyx_n_s__mylib); ...
So it looks like it's checking the `expose` module and not finding any
objects named `mylib`.
How can I dynamically generate a list of objects defined in
`mylib.pxd`? Is it even possible? Perhaps there is an easier way to
expose those constants?
Thanks,
Trevor
--
This email may be signed or encrypted with GPG (http://www.gnupg.org).
The GPG signature (if present) will be attached as 'signature.asc'.
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
Robert Bradshaw
You can't dynamically generate this list, especially as these are
statically defined and resolved at compile rather than runtime. What
you can do is "from mylib cimport *" or, to be more selective, "from
mylib cimport CONST_A, CONST_B."
- Robert
W. Trevor King
> You can't dynamically generate this list, especially as these are
> statically defined and resolved at compile rather than runtime.
Then how does `from mylib cimport *` know what to import? I'll start
digging through the Cython source. Maybe that will clear this up for
me.
> What you can do is "from mylib cimport *" or, to be more selective,
> "from mylib cimport CONST_A, CONST_B."
Which exposes the constants inside expose.pyx, but
$ python -c 'import myproject.expose; print pycomedi.expose.CONST_A'
fails with
Traceback (most recent call last):
File "<string>", line 1, in <module>
AttributeError: 'module' object has no attribute 'CONST_A'
unless I explicity add a line like
CONST_A = mylib.CONST_A
to expose.pyx.
Robert Bradshaw
> On Wed, Feb 09, 2011 at 10:06:43AM -0800, Robert Bradshaw wrote:
>> You can't dynamically generate this list, especially as these are
>> statically defined and resolved at compile rather than runtime.
>
> Then how does `from mylib cimport *` know what to import? I'll start
> digging through the Cython source. Maybe that will clear this up for
> me.
It's resolved at compile time--the compiler looks at the source (tree)
and sees what is defined.
>> What you can do is "from mylib cimport *" or, to be more selective,
>> "from mylib cimport CONST_A, CONST_B."
>
> Which exposes the constants inside expose.pyx, but
> $ python -c 'import myproject.expose; print pycomedi.expose.CONST_A'
> fails with
> Traceback (most recent call last):
> File "<string>", line 1, in <module>
> AttributeError: 'module' object has no attribute 'CONST_A'
> unless I explicity add a line like
> CONST_A = mylib.CONST_A
> to expose.pyx.
That is true, these enums are C ints that can be used directly, not
automatically wrapped in Python-level objects. This comes back to the
distinction that Cython is something that lets you call C libraries,
it's not a wrapper-generator. It would be nice to make something like
this easier, but we don't want to automatically pollute the module
namespace with all the C-level constructs (and there's also the issue
of re-definition, what if someone does "pycomedi.expose.CONST_A=100").
- Robert
W. Trevor King
> On Wed, Feb 9, 2011 at 11:30 AM, W. Trevor King <wk...@drexel.edu> wrote:
> > On Wed, Feb 09, 2011 at 10:06:43AM -0800, Robert Bradshaw wrote:
> >> You can't dynamically generate this list, especially as these are
> >> statically defined and resolved at compile rather than runtime.
> >
> > Then how does `from mylib cimport *` know what to import? I'll start
> > digging through the Cython source. Maybe that will clear this up for
> > me.
>
> It's resolved at compile time--the compiler looks at the source (tree)
> and sees what is defined.
If it's just the compiler (e.g. gcc) that cares about what's in the
header, why do we need a .pxd file to tell Cython about it? Is it
just to make sure Cython doesn't try to generate some sort of default
code for those symbols/macros?
> >> What you can do is "from mylib cimport *" or, to be more selective,
> >> "from mylib cimport CONST_A, CONST_B."
> >
> > Which exposes the constants inside expose.pyx, but
> > $ python -c 'import myproject.expose; print pycomedi.expose.CONST_A'
> > fails with
> > Traceback (most recent call last):
> > File "<string>", line 1, in <module>
> > AttributeError: 'module' object has no attribute 'CONST_A'
> > unless I explicity add a line like
> > CONST_A = mylib.CONST_A
> > to expose.pyx.
>
> That is true, these enums are C ints that can be used directly, not
> automatically wrapped in Python-level objects. This comes back to the
> distinction that Cython is something that lets you call C libraries,
> it's not a wrapper-generator.
I don't want it to generate a wrapper for me, I want to write a
wrapper with it. I told Cython about the constants, but then it
forgets about them ;).
> It would be nice to make something like this easier, but we don't
> want to automatically pollute the module namespace with all the
> C-level constructs
Which module are you talking about?
import mylib
would not pollute expose.pyx's namespace (even if it worked like I
think it should). I'm looking for a way to explicity pollute
expose.pyx's namespace ;).
> (and there's also the issue of re-definition, what if someone does
> "pycomedi.expose.CONST_A=100").
I think that would fall under the "consenting adults" clause ;).
Thanks for helping clear this up!
Robert Bradshaw
> On Wed, Feb 09, 2011 at 11:38:11AM -0800, Robert Bradshaw wrote:
>> On Wed, Feb 9, 2011 at 11:30 AM, W. Trevor King <wk...@drexel.edu> wrote:
>> > On Wed, Feb 09, 2011 at 10:06:43AM -0800, Robert Bradshaw wrote:
>> >> You can't dynamically generate this list, especially as these are
>> >> statically defined and resolved at compile rather than runtime.
>> >
>> > Then how does `from mylib cimport *` know what to import? I'll start
>> > digging through the Cython source. Maybe that will clear this up for
>> > me.
>>
>> It's resolved at compile time--the compiler looks at the source (tree)
>> and sees what is defined.
>
> If it's just the compiler (e.g. gcc) that cares about what's in the
> header, why do we need a .pxd file to tell Cython about it? Is it
> just to make sure Cython doesn't try to generate some sort of default
> code for those symbols/macros?
Cython doesn't actually read the header files, just emits #include
statements, so we need to declare things to use them. An automatic .h
-> .pxd translator would be nice, but there's a lot of corner-cases to
take care of here (not to mention issues of platform dependence).
cimport * will pollute expose.pyx's C-level namespace. You can use
them just fine there. If you need to access these constants from
Python, you do have to explicitly expose them.
- Robert
Stefan Behnel
> Cython doesn't actually read the header files, just emits #include
> statements, so we need to declare things to use them. An automatic .h
> -> .pxd translator would be nice, but there's a lot of corner-cases to
> take care of here (not to mention issues of platform dependence).
I think a one-time .pxd generator would be much more useful than automatic
.h file parsing at compile time. I don't think there are so many real world
cases where the definitions in a .h file are usable as they come in (just
think of #ifdef's and friends). Plus, it's often helpful to adapt the
declarations slightly to make them nicer to use from Cython.
Stefan
Stefan Behnel
> On Wed, Feb 09, 2011 at 10:06:43AM -0800, Robert Bradshaw wrote:
>> You can't dynamically generate this list, especially as these are
>> statically defined and resolved at compile rather than runtime.
>
> Then how does `from mylib cimport *` know what to import?
Interesting question. Do we actually support __all__ in .pxd files?
Stefan
W. Trevor King
I just checked the .c files generated by
from mylib cimport *
and
cimport mylib
Test 1:
mylib.pxd only contained `cdef extern from 'mylib.h': ...`
Results:
* Identical .c files (except for the comments listing the lines from
the .pyx file used to generate the .c file, of course).
* mylib.pxd externs lead to an `#include "mylib.h"` call.
* No other content due to mylib.pxd.
Test 2:
mylib.pxd defines a cdef class fleshed out in mylib.pyx
Results:
* Identical .c files (except for the comments listing the lines from
the .pyx file used to generate the .c file, of course).
* mylib.pxd externs lead to an `#include "mylib.h"` call.
* Code generated defining the cdef class from `mylib.pxd`.
I suppose that means that __all__ is not supported.
Robert Bradshaw
That is correct. I don't suppose that __all__ would be that hard to
add (though we'd have to make a special case that an assignment would
be allowed in a pxd file).
- Robert
W. Trevor King
On Wed, Feb 09, 2011 at 12:23:25PM -0500, W. Trevor King wrote:
> Now I want to expose those constants in Python, so I have `expose.pyx`:
>
> cimport mylib
>
> CONST_A = mylib.CONST_A
> CONST_B = mylib.CONST_B
> ...
>
> But the last part seems pretty silly. I'd like to do something like
>
> cimport mylib
> import sys
>
> for name in dir(mylib):
> setattr(sys.modules[__name__], name, getattr(mylib, name))
>
> which compiles fine, but fails to import with...
The discussion eventually led to:
On Wed, Feb 09, 2011 at 12:22:41PM -0800, Robert Bradshaw wrote:
> cimport * will pollute expose.pyx's C-level namespace. You can use
> them just fine there. If you need to access these constants from
> Python, you do have to explicitly expose them.
In an attempt to make it easier to explicitly expose them to other
Python modules at compile time, I've been working through the Cython
source, and found that when cythoning expose.pyx, all the items
declared in mylib.pxd are stored in `Cython.Compiler.Symtab.Entry`s in
mylib.pxd's `Cython.Compiler.Symtab.ModuleScope` instance.
By tweaking
Cython.Compiler.Node.CImportStatNode.generate_function_definitions
to be
def generate_function_definitions(self, env, code):
modules = [m for m in env.cimported_modules if m.module_name == self.module_name]
if len(modules) > 0:
module = modules[0]
for name,entry in module.entries.iteritems():
code.putln('/* %s.%s */' % (self.module_name, name))
I can add a comment to extern.c listing all the stuff I'd declared in
`cdef extern` blocks in mylib.pxd. That's nice, but not very useful
(it does show, however, that Cython is not "forgetting" about the
definitions as I previously thought).
What I'm missing is a way to bind the ModuleScope namespace to a name
in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib,
name)` will work in expose.pyx. If it would be easier to pre-compute
the result of these commands and hard-code them in `extern.c` at
compile time, that's fine.
For example, my comment-generating version of
`generate_function_definitions` is not far from being able to generate
the results of `dir(mylib)`, it would just need to wrap the results in
a list and place references to that list in place of the stock `dir()`
execution code when the dir argument resolved to the cimported module.
Yuck ;).
It seems like it would be easier to generate some kind of wrapper
class (PxdModule?) for mylib when it is cimported (at compile time),
and then further interactions would take care of themselves (at run
time).
Does anyone know how I would go about doing this?
On an administrative level, since this would seem to require altering
the Cython source code, should I move this discussion to cython-devel,
or does splitting the thread across two lists make it too hard to
follow?
Thanks,
Trevor
Robert Bradshaw
Cython modules have two namespaces, the Python level one and the C
level one. This separation is necessary, as the C level one may
contain objects that are unrepresentable in the Python-level layer,
and get statically bound (e.g. one would want to mark them as
read-only at the very least). Many users also like the fact that the
C-level implementation details of their Cython modules do not get
leaked out to their Python namespaces (though use of __all__ could
provide, but not enforce, similar behavior).
You have also hit into the thorny issue that .pxd files are used for
many things. They may be pure C library declarations with no Python
module backing, they may be declarations of (externally implemented)
Python modules (such as numpy.pxd), or they may be declarations for
Cython-implemented modules.
In terms of your specific question, I don't think hijacking the
builtin dir and getattr to blur this line.
> It seems like it would be easier to generate some kind of wrapper
> class (PxdModule?) for mylib when it is cimported (at compile time),
> and then further interactions would take care of themselves (at run
> time).
>
> Does anyone know how I would go about doing this?
Would such an object be created anew for every module that cimports
the declaration file?
I have toyed with the idea of subclassing the module object itself for
better support of C-level attributes from the Python (and Cython)
namespaces.
Here's another idea, what if extern blocks could contain cpdef
declarations, which would automatically generate a Python-level
wrappers for the declared members (if possible, otherwise an error)?
> On an administrative level, since this would seem to require altering
> the Cython source code, should I move this discussion to cython-devel,
> or does splitting the thread across two lists make it too hard to
> follow?
Yes, it would probably be best to move this thread over there. A
succinct summary of what issue you are trying to solve would probably
be helpful now anyways.
- Robert