c++ interface?

[nbecker]

Hi, newb here.  Longtime c++/python/numpy/scipy user.  One thing I make heavy use of is boost::python to allow python to interoperate with my c++ code.  So, for example, c++ classes can be exposed and used from python, as well as c++ functions.

I know julia has a c ffi, which can access C functions, using only basic C types.  Any thoughts of something more like boost::python?

[Stefan Karpinski]

Do you know how Python calls C++? Since there's no C++ ABI and it does lots of compiler-specific name mangling, it's hard to see how to do it in a consistent, sane and portable way. To what extent are the boost libraries actual libraries in the C sense of providing function objects that can be called; to what extent are they C++ libraries in the sense that they're a bunch of header files that don't actually provide executable code, but rather allow you to generate C++ code from templates?

[Dag Sverre Seljebotn]

On 05/02/2012 05:03 PM, Stefan Karpinski wrote:
> Do you know how Python calls C++? Since there's no C++ ABI and it does
> lots of compiler-specific name mangling, it's hard to see how to do it
> in a consistent, sane and portable way. To what extent are the boost
> libraries actual libraries in the C sense of providing function objects
> that can be called; to what extent are they C++ libraries in the sense
> that they're a bunch of header files that don't actually provide
> executable code, but rather allow you to generate C++ code from templates?

There's loads of tools to interface Python and C++ (Cython, SWIG,
boost::python...). They all work by (one way or the other) generate C++
code that implements a Python module using the CPython API, then that
C++ code is compiled (using whatever C++ compiler is appropriate!) to
.so and loaded into the Python interpreter.

A major part of the success of Python as a glue language is that
bindings are (often) created on the API level, not the ABI level.

I actually got rather disapppointed when reading the Julia docs in this
read; from the introductory blurb "Call C functions directly (no
wrappers or special APIs needed)" I expected a lot more than the stock
ABI FFI that everybody *has* to provide (except perhaps irrelevant toy
languages).

Dag

[Dag Sverre Seljebotn]

On 05/02/2012 05:27 PM, Dag Sverre Seljebotn wrote:
> On 05/02/2012 05:03 PM, Stefan Karpinski wrote:
>> Do you know how Python calls C++? Since there's no C++ ABI and it does
>> lots of compiler-specific name mangling, it's hard to see how to do it
>> in a consistent, sane and portable way. To what extent are the boost
>> libraries actual libraries in the C sense of providing function objects
>> that can be called; to what extent are they C++ libraries in the sense
>> that they're a bunch of header files that don't actually provide
>> executable code, but rather allow you to generate C++ code from
>> templates?
>
> There's loads of tools to interface Python and C++ (Cython, SWIG,
> boost::python...). They all work by (one way or the other) generate C++
> code that implements a Python module using the CPython API, then that
> C++ code is compiled (using whatever C++ compiler is appropriate!) to
> .so and loaded into the Python interpreter.
>
> A major part of the success of Python as a glue language is that
> bindings are (often) created on the API level, not the ABI level.

I think this is worth highlighting -- even C libraries really export an
API, not an ABI -- all the #ifdef's that conditionally typedef a type to
be one thing or the other, all the macros. Reimplementing that logic in
the Julia wrapper is of course doable, but it is hardly as robust as
just using a C compiler to build the wrapper and use the same header
files as the library was built with.

Dag

[Stefan Karpinski]

On Wed, May 2, 2012 at 11:27 AM, Dag Sverre Seljebotn <d.s.se...@astro.uio.no> wrote:

There's loads of tools to interface Python and C++ (Cython, SWIG, boost::python...). They all work by (one way or the other) generate C++ code that implements a Python module using the CPython API, then that C++ code is compiled (using whatever C++ compiler is appropriate!) to .so and loaded into the Python interpreter.

A major part of the success of Python as a glue language is that bindings are (often) created on the API level, not the ABI level.

I actually got rather disapppointed when reading the Julia docs in this read; from the introductory blurb "Call C functions directly (no wrappers or special APIs needed)" I expected a lot more than the stock ABI FFI that everybody *has* to provide (except perhaps irrelevant toy languages).

Heh. I in turn find the approach of generating a bunch of C++ code disappointing ;-)

Is there a way to call C functions in Python analogous to ccall without having to compile anything? I wasn't aware of such a thing. Certainly ccall is more limited than writing C++ bindings that implement the language's calling convention — since everything is more limited than writing bindings because you can do anything with that approach given a sufficient amount of work or a sufficiently clever binding generator.

[Neal Becker]

boost::python is a system using c++ templates.  The c++ source (using boost::python templates) is compiled to a shared library, which is then usable from python.

An example of usage would be:

BOOST_PYTHON_MODULE (my_module) {

  def ("my_function", &my_function)

  _class<my_class> ("my_class", ...)

     .def ("some_class_method", &my_class::some_class_method)

...

This shows only a tiny subset of the extensive functionality.  This code will build a python module which incorporates a function, and a class (who also exposes 1 method).  The c++ function will become a python function, and the c++ class will become a python class.

[Tim Holy]

On Wednesday, May 02, 2012 11:03:37 AM Stefan Karpinski wrote:
> Do you know how Python calls C++? Since there's no C++ ABI

I was going to release this after much more serious testing, but why not just
jump the gun?
https://github.com/timholy/julia/blob/cpp/extras/cpp.jl

Personally, I don't see what's so awful about the C++ name mangling, aside
from the (rather big) fact that it's not standardized across compilers. C does
the same thing explicitly, i.e., we have floorf and floorl, etc. In C++ they'd
all be called floor, and then the compiler attaches extra characters to signal
the data types. But likely I don't appreciate the more troublesome aspects.

Someone would have to add in the name mangling table from other compilers, but
this is a nice reference:
http://www.agner.org/optimize/calling_conventions.pdf
As you can see, currently that should involve changing just two lines. For a
general solution, we'd need a scheme for figuring out which platform we're on.

There's still a lot missing, of course, like namespaces, virtual table layout,
etc. Re virtual table layout, since we can't currently ccall with structures,
it's not currently an issue. Though given Patrick's progress, it looks like it
could become one fairly soon!

--Tim

[Neal Becker]

There is something like ccall in python called 'ctypes'.  Frankly, I rarely use it since the functionality is so limited (as it is with ccall, IIUC).

[Stefan Karpinski]

Depends on the library. Libraries like LAPACK that are intended to be called from other programs in a variety of languages do in fact export an ABI. At the very least, the combination of the shared library and a header file should be sufficient to call it correctly. But at that point, yes, you do need something tantamount to a C compiler.

[Dag Sverre Seljebotn]

On 05/02/2012 05:33 PM, Stefan Karpinski wrote:
> On Wed, May 2, 2012 at 11:27 AM, Dag Sverre Seljebotn
> <d.s.se...@astro.uio.no <mailto:d.s.se...@astro.uio.no>> wrote:
>
>
> There's loads of tools to interface Python and C++ (Cython, SWIG,
> boost::python...). They all work by (one way or the other) generate
> C++ code that implements a Python module using the CPython API, then
> that C++ code is compiled (using whatever C++ compiler is
> appropriate!) to .so and loaded into the Python interpreter.
>
> A major part of the success of Python as a glue language is that
> bindings are (often) created on the API level, not the ABI level.
>
> I actually got rather disapppointed when reading the Julia docs in
> this read; from the introductory blurb "Call C functions directly
> (no wrappers or special APIs needed)" I expected a lot more than the
> stock ABI FFI that everybody *has* to provide (except perhaps
> irrelevant toy languages).
>
>
> Heh. I in turn find the approach of generating a bunch of C++ code
> disappointing ;-)

It Works (tm).

Pragmatism is what got scientific Python in a position where people
actually use it.

And I fail to see what's so ugly, but then I'm co-author on a tool that
generates C :-)

> Is there a way to call C functions in Python analogous to ccall without
> having to compile anything? I wasn't aware of such a thing. Certainly
> ccall is more limited than writing C++ bindings that implement the
> language's calling convention — since everything is more limited than
> writing bindings because you can do anything with that approach given a
> sufficient amount of work or a sufficiently clever binding generator.

Yes, the "ctypes" module provides the stock FFI, it is one of the tools
people use to wrap C (perhaps in particular in Windows-centric circles
where relying on a C compiler on the users machine is more of an issue).
It is an optional part of standard Python in the sense that it only
builds wherever libffi builds (so on some very wierd embedded platforms
it would perhaps not be available).

Dag

[Dag Sverre Seljebotn]

Well, the 'ABI' is whatever the de facto Fortran 77 ABI was -- a bit
less specified than C, but not catastrophically so.

At least in SciPy there's some wierd stuff like passing the length of
the character(1) transpose flags on the end of the call stack. That gets
ignored by code compiled by modern Fortran compilers (they don't look at
that part of the call stack), but perhaps somewhere there's a Fortran 77
compiler that uses it, and if you compile reference BLAS with it...

Dag

[Stefan Karpinski]

Oh, very nice! This may give me the necessary kick to flesh base/h2j.cpp out into an actual bindings generator. Being able to generate Julia bindings for C and C++ code from header files would be quite lovely.

The *only* problem I have with name mangling is the lack of standardization.

[Dag Sverre Seljebotn]

Of course, I expect that the Julia ccall is a lot more elegant and
achieves a lot higher performance than ctypes. The poor performance of
ctypes is one of the reasons people may use tools like Cython.

(I believe PyPy JITs away the ctypes module though, much like Julia.)

Dag

[Stefan Karpinski]

An interesting thought. Since we'll likely be generating Julia code that loads and calls C++ object files, it would be interesting to *dynamically* determine the calling convention of the compiler that generated the library. It might be possible just from inspecting the symbol table since how the compiler mangles names probably indicates what compiler it is.

[Stefan Karpinski]

On Wed, May 2, 2012 at 11:53 AM, Dag Sverre Seljebotn <d.s.se...@astro.uio.no> wrote:

Of course, I expect that the Julia ccall is a lot more elegant and achieves a lot higher performance than ctypes. The poor performance of ctypes is one of the reasons people may use tools like Cython.

 Yes, ccall turns into just a function call, so it's as fast as the function call would be in C.

[Matthew Emmett]

On Wed, May 2, 2012 at 11:36 AM, Neal Becker <ndbe...@gmail.com> wrote:
> There is something like ccall in python called 'ctypes'.  Frankly, I rarely
> use it since the functionality is so limited (as it is with ccall, IIUC).

I have a slightly different experience with ctypes: I find it quite
flexible and find myself using it a lot these days. I use it to call
C, Fortran 90, and (with a small C shim) C++. After telling ctypes a
little bit more about my C callable functions (ie, their call
signatures), I can make calling them in Python very straight forward.
I can even define ctypes wrappers around C structs and F90 derived
types (if they are C bound), and set their contents directly.

I used to use Cython and/or write my own C extension modules for
calling C code, but now I just stick with ctypes. Similarly, I use
ctypes instead of f2py for calling F90 now too.

Anyway, it seems to me that ccall can be equally flexible.
Futhermore, if Julia is easily callable from C, then it can be easily
callable from Python through ctypes.

Matt

PS, great work on Julia!

[Tim Holy]

On Wednesday, May 02, 2012 11:49:40 AM Stefan Karpinski wrote:
> Oh, very nice! This may give me the necessary kick to flesh base/h2j.cpp
> out into an actual bindings generator. Being able to generate Julia
> bindings for C and C++ code from header files would be quite lovely.

Shall I push into extras? (That was just my branch.)

--Tim

[Tim Holy]

On Wednesday, May 02, 2012 11:55:11 AM Stefan Karpinski wrote:
> An interesting thought. Since we'll likely be generating Julia code that
> loads and calls C++ object files, it would be interesting to *dynamically*
> determine the calling convention of the compiler that generated the
> library. It might be possible just from inspecting the symbol table since
> how the compiler mangles names probably indicates what compiler it is.

That is a good idea!

--Tim

[Stefan Karpinski]

Yes, please :-)

[Stefan Karpinski]

I'm picturing running a Julia binding generator on a C++ .h file and generating a single Julia file that will work for C++ object files generated by a variety of different compilers, so that the .jl bindings file isn't tied to a particular compiler.

On Wed, May 2, 2012 at 12:14 PM, Tim Holy <tim....@gmail.com> wrote:

[Tim Holy]

On Wednesday, May 02, 2012 12:20:53 PM Stefan Karpinski wrote:
> Yes, please

It's in.

--Tim

[Tim Holy]

On Wednesday, May 02, 2012 11:36:29 AM Neal Becker wrote:
> There is something like ccall in python called 'ctypes'. Frankly, I rarely
> use it since the functionality is so limited (as it is with ccall, IIUC).

Is the limitation similar to Julia's current situation? I.e., the most
important one being no support for structs?

There's a serious effort underway (by Patrick O'Leary) towards supporting
structs in Julia's ccall. I'd guess that it might be extensible to passing C++
objects back and forth, too, with a little more work on that cpp macro I just
committed.

--Tim

[Matthew Emmett]

On Wed, May 2, 2012 at 12:28 PM, Tim Holy <tim....@gmail.com> wrote:
> On Wednesday, May 02, 2012 11:36:29 AM Neal Becker wrote:
>> There is something like ccall in python called 'ctypes'.  Frankly, I rarely
>> use it since the functionality is so limited (as it is with ccall, IIUC).
>
> Is the limitation similar to Julia's current situation? I.e., the most
> important one being no support for structs?

You can wrap C structs (and F90 derived types that are C bound) with
ctypes. See http://docs.python.org/library/ctypes.html#structures-and-unions

Matt

[Patrick O'Leary]

Reminder: that's a side benefit, not the primary goal. To make this efficient in general we might need to find a way to do struct interfaces with zero copies, which I'm not even touching right now.

[Tim Holy]

On Wednesday, May 02, 2012 09:54:37 AM Patrick O'Leary wrote:
> Reminder: that's a side benefit, not the primary goal. To make this
> efficient in general we might need to find a way to do struct interfaces
> with zero copies, which I'm not even touching right now.

For zero-copy, would we "just" :-) need pointer_to_compositekind and a
corresponding Julia type defined with the same byte packing used in the native
C struct?

--Tim

[Stefan Karpinski]

This is part of the plan for immutable composites — they will have C-compatible layout. Won't address every possible C data structure you might want to call, but will cover 95% of cases.

[Kevin Squire]

I'm just curious if it actually turned out that immutable structs have a C-compatible packing, and if so, if zero-copy packing/parsing is closer to reality?

Kevin

[Jameson Nash]

yes, types where isbits(type) == true (immutable or not) are now C-compatible, for the most part

[Kevin Squire]

Great!  

So, let's say I have a chunk of bytes (Uin8) that I read from a file, and the first chunk of those can be mapped to a set of bitstypes (the rest consists of variable length fields).  The fields themselves are required to be little endian, according to a spec, and I'm on a little endian machine.  Can I just point to that chunk of bytes and say, this is a frob, where frob is a composite type?

Right now, I'm using StrPack.  This isn't a bad option, since it could handle byte swapping on a big endian machine, but I'd love to do be able handle this without copying (the files I'm working with are 10s to 100s of Gb, and in some simple cases, all I want to do is filter on a particular field).

Thanks,

   Kevin