Extending Python with C or C++
Ryan
that I enjoy using everyday. I'm now interested in getting to know
more about the guts (C/C++) and extending it. But, extending python
still seems like a black art to me. Is there anymore docs or info on
extending it besides the standard sparse ones (http://www.python.org/
doc/2.5.2/ext/intro.html) that may give me more insight? Is there a
class available? How can I learn more about the guts of python? How
would one go about following an interest in contributing to the
development of python.
Thanks,
Ryan
Ralf Schoenian
It is not exactly what you are looking for but nevertheless I am
thinking the article "Automatic C Library Wrapping -- Ctypes from the
Trenches" may be interesting for you. You can find it in the latest
Python Papers issue or simply following the link:
http://ojs.pythonpapers.org/index.php/tpp/article/view/71
Regards,
Ralf
Terry Reedy
> I've been using Python for many years now. It's a wonderful language
> that I enjoy using everyday. I'm now interested in getting to know
> more about the guts
The 'guts' of Python the language include the object model, namespaces
(including modules), and the statement and infix-expression syntax.
> (C/C++) and extending it.
Now you are asking about CPython, the leading computer implementation.
> But, extending python still seems like a black art to me.
> Is there anymore docs or info on
> extending it besides the standard sparse ones (http://www.python.org/
> doc/2.5.2/ext/intro.html) that may give me more insight? Is there a
> class available?
If you want to connect CPython to Python-oblivious code written in C,
Swig (with C code) and Ctypes (with Python code) are the main choices.
If you want to write new Python-aware (and specific) code, you can use
the CPython C-API functions. Extensions in C are written as importable
modules. The interface for such is not difficult; existing examples
should be a good guide.
> How can I learn more about the guts of python?
The 'guts' of an implementation follow from the 'guts' of the language.
There must be a syntax parser and compiler to internal form,
evaluation loop, and implemenations of built-in constants, functions,
classes, and modules. CPython's source tree begins as
http://svn.python.org/view/
You might actually want to start at
http://svn.python.org/view/python/trunk/
Note: if you click a filename, such as 'setup.py', you get the entire
revision history with checkin messages.
If you click the displayed revision number, such as '67978', you get the
latest checkin message and the current version of the file.
> How would one go about following an interest in contributing to the
> development of python.
Read
http://python.org/dev/
and start following the pydev list, mirrored to gmane.comp.python.devel
at news.gmane.org.
Terry Jan Reedy
Ryan
> Ryan wrote:
> > I've been using Python for many years now. It's a wonderful language
> > that I enjoy using everyday. I'm now interested in getting to know
> > more about the guts
>
> The 'guts' of Python the language include the object model, namespaces
> (including modules), and the statement and infix-expression syntax.
>
> > (C/C++) and extending it.
>
> Now you are asking about CPython, the leading computer implementation.
>
> > But, extending python still seems like a black art to me.
>
> > Is there anymore docs or info on
>
> > extending it besides the standard sparse ones (http://www.python.org/
> > doc/2.5.2/ext/intro.html) that may give me more insight? Is there a
> > class available?
>
> If you want to connect CPython to Python-oblivious code written in C,
> Swig (with C code) and Ctypes (with Python code) are the main choices.
> If you want to write new Python-aware (and specific) code, you can use
> the CPython C-API functions. Extensions in C are written as importable
> modules. The interface for such is not difficult; existing examples
> should be a good guide.
>
> > How can I learn more about the guts of python?
>
> The 'guts' of an implementation follow from the 'guts' of the language.
> There must be a syntax parser and compiler to internal form,
> evaluation loop, and implemenations of built-in constants, functions,
> You might actually want to start athttp://svn.python.org/view/python/trunk/
> Note: if you click a filename, such as 'setup.py', you get the entire
> revision history with checkin messages.
> If you click the displayed revision number, such as '67978', you get the
> latest checkin message and the current version of the file.
>
> > How would one go about following an interest in contributing to the
> > development of python.
>
> and start following the pydev list, mirrored to gmane.comp.python.devel
> at news.gmane.org.
>
> Terry Jan Reedy
Thanks Terry! This clarifies many of the concepts that I want to get
started to dive deeper into CPython.
1. The abstract Python Language (not specific to any implementation)
2. The CPython implementation (http://svn.python.org/view/python/
trunk/)
3. Extending CPython by connecting it to Python-oblivious code written
in C with Ctypes (Ralf's suggestion is good for this)
4. Extending CPython by connecting it to Python-aware (and specific)
code using the CPython C-API functions (http://docs.python.org/c-api/)
Stefan Behnel
> 3. Extending CPython by connecting it to Python-oblivious code written
> in C with Ctypes (Ralf's suggestion is good for this)
> 4. Extending CPython by connecting it to Python-aware (and specific)
> code using the CPython C-API functions (http://docs.python.org/c-api/)
For extending CPython (and possibly connecting it to external C
code/libraries) without learning too many new things (Python + a little C
should be enough), you can use ctypes if you can accept the runtime
overhead and the dependency on the ctypes module (which does not exist in
all CPython releases). However, I'd recommend Cython instead, as it gives
you very fast and portable code that works with all CPython versions. And
Cython code is not any harder to write at all.
Stefan
Nick Craig-Wood
> Ryan wrote:
> > I've been using Python for many years now. It's a wonderful language
> > that I enjoy using everyday. I'm now interested in getting to know
> > more about the guts (C/C++) and extending it. But, extending python
> > still seems like a black art to me. Is there anymore docs or info on
> > extending it besides the standard sparse ones (http://www.python.org/
> > doc/2.5.2/ext/intro.html) that may give me more insight? Is there a
> > class available? How can I learn more about the guts of python? How
> > would one go about following an interest in contributing to the
> > development of python.
>
> thinking the article "Automatic C Library Wrapping -- Ctypes from the
> Trenches" may be interesting for you. You can find it in the latest
> Python Papers issue or simply following the link:
> http://ojs.pythonpapers.org/index.php/tpp/article/view/71
Interesting - I didn't know about h2xml and xml2py before and I've
done lots of ctypes wrapping! Something to help with the initial
drudge work of converting the structures would be very helpful.
( http://pypi.python.org/pypi/ctypeslib/ )
I gave it a quick go and it worked fine. I had to edit the XML in one
place to make it acceptable (removing a u from a hex number). The
output of xml2py was an excellent place to start for the conversion,
though I don't think I'd want to use an automated process like in the
paper above as its output needed tweaking.
...
Here are my thoughts on the conversion :-
It converted an interface which looked like this (the inotify interface)
struct inotify_event {
int wd; /* Watch descriptor */
uint32_t mask; /* Mask of events */
uint32_t cookie; /* Unique cookie associating related
events (for rename(2)) */
uint32_t len; /* Size of name field */
char name[]; /* Optional null-terminated name */
};
Into this
class inotify_event(Structure):
pass
inotify_event._fields_ = [
('wd', c_int),
('mask', uint32_t),
('cookie', uint32_t),
('len', uint32_t),
('name', c_char * 0),
]
Which is a very good start. However it defined these which clearly
aren't portable
int32_t = c_int
uint32_t = c_uint
Whereas it should have been using the ctypes inbuilt types
c_int32
c_uint32
Also I don't think c_char * 0 does anything sensible in ctypes,
c_byte * 0 is what is required plus a bit of casting. This is a
non-standard GNU extension to C though.
All that said though, it looks like a great time saver.
--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick
sturlamolden
> I've been using Python for many years now. It's a wonderful language
> that I enjoy using everyday. I'm now interested in getting to know
> more about the guts (C/C++) and extending it. But, extending python
> still seems like a black art to me.
There are several alternatives to using the Python C API by hand,
depending on
your particular need. It will save you a lot of greif:
Pyrex: Python-like language for creating extension modules.
*Cython: clone of Pyrex
*ctypes: call DLLs from python
comtypes: call COM DLLs from Python
*f2py: generate wrappers for Fortran or C libraries. Knowledge of
Fortran required.
pywin32: use or implement ActiveX objects with Python
*scipy.weave: inline C++ code in Python
Swig: generate wrappers for C libraries (some C++ support)
CXX: C++ wrapper for the Python C API
Boost.Python: C++ wrapper for the Python C API
SIP: generate wrappers for C++ libraries
(* Works easily with NumPy arrays.)
Thomas Heller
If you are using a recent version of gccxml, then you should use the
ctypeslib-gccxml-0.9 branch of ctypeslib instead of the trunk. (I should
really merge the gccxml-0.9 branch into the trunk;-)
If you are already using the branch and the XML file is not accepted,
then could you please provide a short C code snippet that reproduces
the problem so that I can fix it?
IMO this would be difficult to achive automatically. There are cases
wher c_int is the correct type, in other cases c_int32 is correct.
> Also I don't think c_char * 0 does anything sensible in ctypes,
> c_byte * 0 is what is required plus a bit of casting. This is a
> non-standard GNU extension to C though.
>
> All that said though, it looks like a great time saver.
>
Thanks,
Thomas
Nick Craig-Wood
> Nick Craig-Wood schrieb:
> > Interesting - I didn't know about h2xml and xml2py before and I've
> > done lots of ctypes wrapping! Something to help with the initial
> > drudge work of converting the structures would be very helpful.
> >
> > ( http://pypi.python.org/pypi/ctypeslib/ )
> >
> > I gave it a quick go and it worked fine. I had to edit the XML in one
> > place to make it acceptable (removing a u from a hex number).
>
> ctypeslib-gccxml-0.9 branch of ctypeslib instead of the trunk. (I should
> really merge the gccxml-0.9 branch into the trunk;-)
>
> If you are already using the branch and the XML file is not accepted,
> then could you please provide a short C code snippet that reproduces
> the problem so that I can fix it?
I'm using gccxml from debian testing 0.9.0+cvs20080525-1 which I guess
doesn't have the code from the branch in.
Yes it is almost impossible difficult to achieve automatically -
exactly how far do you want to unravel the twisty turny mess of
typedefs that make up uint32_t? It is easy to change the generated
output since it defines the type in one place.
uint32_t and friends (stdint.h) are standardised in C99 so might it be
reasonable to put some special cases in for them, expecially since the
corresponding types already exist in ctypes?
Thomas Heller
> Thomas Heller <the...@python.net> wrote:
>> Nick Craig-Wood schrieb:
>> > Interesting - I didn't know about h2xml and xml2py before and I've
>> > done lots of ctypes wrapping! Something to help with the initial
>> > drudge work of converting the structures would be very helpful.
>> >
>> > ( http://pypi.python.org/pypi/ctypeslib/ )
>> >
>> > I gave it a quick go and it worked fine. I had to edit the XML in one
>> > place to make it acceptable (removing a u from a hex number).
>>
>> If you are using a recent version of gccxml, then you should use the
>> ctypeslib-gccxml-0.9 branch of ctypeslib instead of the trunk. (I should
>> really merge the gccxml-0.9 branch into the trunk;-)
>>
>> If you are already using the branch and the XML file is not accepted,
>> then could you please provide a short C code snippet that reproduces
>> the problem so that I can fix it?
>
> I'm using gccxml from debian testing 0.9.0+cvs20080525-1 which I guess
> doesn't have the code from the branch in.
I meant the branch in the repository where ctypeslib lives in.
Anyway, it doesn't matter anymore since I merged that branch
into the ctypeslib trunk.
Now the problem with the 'u' suffix that you mentioned should be fixed, and
I also made a quick change so that the sized integer types from stdint.h are generated
correctly. So, please update your ctypeslib installation and trey again;-)
Thomas
Nick Craig-Wood
I gave it a go and I can report success on both counts!
Well done and thank you.