Calling into Rust (i.e., a static/dynamic lib with C-compatible ABI)

[Spencer Wilson]

Hi all,

First, thanks all for the hard work on this project!

I have Rust code, which when compiled (either statically or as a shared lib) can expose a C-compatible ABI. I am looking to build an extension module with CFFI (in the API-level, out-of-line mode, as I understand that is generally the most reliable and performant) which calls to this. My objective is to statically link everything (again, for performance), but it's not clear to me if this is feasible or what the correct CFFI syntax is to accomplish this.

My questions:

1. Compiled Rust exposes a C-compatible ABI; I do not have C source code. Does this preclude any API-level use of CFFI? Does API-level require having C source code?
2. Given that one has only C-compatible object code (static or shared; Not sure if this makes a difference, see above), are the only options an ABI-level, in/out-of-line mode?
3. The CFFI docs make no mention of how, in the scenario where one only has access to object code with a C-compatible ABI, having that code as a static or shared library impacts the usage of CFFI. Does a choice of one or the other limit or restrict which modes of CFFI may be used?
   
4. What is the most performant way to do FFI in Python if one has access only to the compiled native code? How do Cython and ctypes compare in this scenario?

Thank you very much in advance!

Cheers,

Spencer Wilson

[Armin Rigo]

Hi,

On 25 May 2017 at 19:32, Spencer Wilson <spencer...@optimizely.com> wrote:
> objective is to statically link everything (again, for performance), but
> it's not clear to me if this is feasible or what the correct CFFI syntax is
> to accomplish this.

In the API mode, CFFI builds an extension module. If you're focusing
on CPython and not PyPy, then you can think about this extension
module as just the same as a CPython C extension module. It is made
as one ``.c`` file produced by CFFI, which gets compiled into a
``.so`` file (``.dll`` or ``.pyd`` on Windows). This compilation step
is not really restricted: just like CPython extension module, it can
be compiled to also include many other ``.c`` sources or ``.o`` files
and/or link to one or several other libraries.

All your questions can be answered in the same general context as "I
want to make a .so file from Rust code". I don't know the details
about Rust, so I hope I won't get something wrong.

> Compiled Rust exposes a C-compatible ABI; I do not have C source code. Does
> this preclude any API-level use of CFFI? Does API-level require having C
> source code?

If Rust produces ``.a`` or ``.o`` files, for example, which you are
supposed to link together to make an application or library, then you
can link them together with the ``.c`` file produced by CFFI to make
the CFFI extension module.

> Given that one has only C-compatible object code (static or shared; Not sure
> if this makes a difference, see above), are the only options an ABI-level,
> in/out-of-line mode?

No.

> The CFFI docs make no mention of how, in the scenario where one only has
> access to object code with a C-compatible ABI, having that code as a static
> or shared library impacts the usage of CFFI. Does a choice of one or the
> other limit or restrict which modes of CFFI may be used?

If you're asking if you can use a static library, the answer is yes
but that static library's code needs to be position-independent code
(PIC). Otherwise, the static library can only be linked into a
program, and not into a ``.so`` file. (If you ask me, non-PIC
compiled libraries come with such a strong restrictions that they are
basically useless in general.)

> What is the most performant way to do FFI in Python if one has access only
> to the compiled native code? How do Cython and ctypes compare in this
> scenario?

The fastest is most probably using PyPy, and then using CFFI is
obvious because the alternatives are comparatively much slower there.


A bientôt,

Armin.