Design for CFFI 1.0

[Armin Rigo]

Hi all,

I finally have a concrete suggestion about the long-planned redesign
of CFFI 1.0. The goal is to change the way we call verify() and move
towards an explicit separate compilation step, rather than doing too
many implicit magic to know if and when we have to recompile. The
verify() API will still exist but be deprecated. The dlopen() API
won't change.

The proposal is to write it this way: first, a separate module
"foomake.py" that contains the template for making the C library:

import cffi
lib = """#include <foo.h>"""
cdef = """int foo(int x);"""
builder = cffi.Builder("fooffi", lib=lib, cdef=cdef)
if __name__ == '__main__':
builder.build()

This file can be run directly with "python foomake.py", in which case
the build() call works like an unconditional verify() and remakes the
C library. The library is called "fooffi" rather than
"_cffi_x94879...".

Then you use it by saying in the normal program:

from fooffi import ffi, lib

(and no explicit "import cffi" is needed there.) This imports an ffi
object that is already loaded with the type declarations (pre-parsed,
thus cutting down import time), and a lib object to access all C
functions like now. It's kept as two separate objects because you
don't want conflicts between the names of C functions and (current or
future) methods on ffi.

Finally, you distribute your project by writing this in setup.py:

from foomake import builder
setup(..., ext_modules=[builder.get_ext_module()])

The point of the "if __name__=='__main__'" condition in foomake.py is
only to make it nicer for this case: here, we want to get the full
definition of "fooffi" but don't need it to be built right at import
time. Instead, we want to get a distutils Extension instance that
will describe to distutils how it should build this extension (in some
subdirectory of "build") and copy it into the final installation
directory.

The 'templates' written in foomake.py can of course be built in more
advanced ways than just as single strings. For example, you can still
do things like loading the library with dlopen() to check if some
function really exists, and add it to the cdef only if it does. You
should also have an easy way to check if something would compile/link
or not, e.g. by making another builder and calling methods like
check_compile() or check_link().

Does this approach make sense to you?


A bientôt,

Armin.

[Maciej Fijalkowski]

+1 from me

--
-- python-cffi: To unsubscribe from this group, send email to python-cffi...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/python-cffi?hl=en
---
You received this message because you are subscribed to the Google Groups "python-cffi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-cffi...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Matti Picus]

+1 for more clarity in what actually happens "under the hood"

What would the path toward this take, where would the place to start coding it be?

Matti

On Wednesday, 27 November 2013 11:33:45 UTC+2, Armin Rigo wrote:

Hi all,

I finally have a concrete suggestion about the long-planned redesign
of CFFI 1.0.  The goal is to change the way we call verify() and move
towards an explicit separate compilation step, rather than doing too
many implicit magic to know if and when we have to recompile.  The
verify() API will still exist but be deprecated.  The dlopen() API
won't change.

.....
A bientôt,

Armin.

[Armin Rigo]

Hi Matti,

On Sun, Dec 15, 2013 at 1:03 PM, Matti Picus <matti...@gmail.com> wrote:
> +1 for more clarity in what actually happens "under the hood"
> What would the path toward this take, where would the place to start coding
> it be?

The first step would be to review the work done during the Cape Town
sprint on the "split-verify" branch. I did not review it in detail
myself, but I know it was done in a style that I would consider
suitable only for experimentation; for real usage I would go for a
rewrite. The approach would be to start from the _cffi_backend C
module (which would probably not change at all), and build a different
'cffi1' Python package around it --- from scratch, but of course
reusing some parts of the current 'cffi' Python package.


A bientôt,

Armin.

[Sarvi Shanmugham]

Is there a patch for this that I could try.

"This imports an ffi object that is already loaded with the type declarations (pre-parsed, 

thus cutting down import time),"

I am hearing on the thread that just avoiding the parsing of the C header files by 

cPickling the outputput during compile and reading it from the pickle and thereby avoiding parsing 

all the C header files does not get us much improvement in performance.

Has this been tried? How does these import performance numbers compare? 

I ask because, because Import time of modules are very important for low-power hardware and some classes of applications.

And we are hitting the later in ours that uses CFFI extensively. 

And am looking for solutions.

Sarvi