python-cffi Google Group

Re: [python-cffi] Callback issue

2013-05-19T22:16:28Z

Hi Geert,

Yes, there are workarounds to do that. I opened a few days ago an
issue about identifying the "best" way and giving it some official
support: [link]
.

A bientôt,

Armin.

Re: [python-cffi] Callback issue

2013-05-18T20:40:00Z

Hi Armin,
On Sat, May 18, 2013 at 9:23 PM, Geert Jansen <gee...@gmail.com<javascript:> >
Thanks! Indeed this was the issue.

The reason I made it a method and not a global function is that I need to
close the callback over some request specific data. Unfortunately it seems
that cffi.callback() as a decorator doesn't work for methods. I have

Re: [python-cffi] Callback issue

2013-05-18T19:35:45Z

Hi Geert,

callbacks.on_message_begin = ffi.callback('http_cb', on_message_begin)

This line is wrong. ffi.callback() returns a cffi pointer that has
ownership of the callback metadata. You then copy the raw pointer
into a struct, and immediately forget the ffi.callback() object
itself, which frees the metadata. You need to keep it alive as long

Callback issue

2013-05-18T19:23:11Z

Hi,
I'm wrapping the Joyent/node.js HTTP parser with CFFI and I'm having some
issues with callbacks. The parser takes a pointer to a structure containing
different callbacks as an argument. It will then call the callbacks during
parsing.
I've isolated the issue here:
[link]

Re: [python-cffi] Zero-copy

2013-05-18T18:02:37Z

Hi Geert,

Yes.

Armin

Re: [python-cffi] Zero-copy

2013-05-18T17:47:52Z

Hi,
On Fri, May 17, 2013 at 12:42 AM, Geert Jansen <gee...@gmail.com<javascript:> >
Thanks Armin. Will cffi use this hack automatically if it detects it is on
CPython?
To really avoid copies you need to find a way to rewrite what you need
The problem here is that I get the string from a framework outside my

Re: [python-cffi] multiple ffi modules

2013-05-18T11:19:45Z

lritter had this problem on IRC.

Maybe he can show you :)

Cheers,
fijal

Re: [python-cffi] multiple ffi modules

2013-05-18T11:17:13Z

Hi Fijal,

I understand the problem in theory. What I fail to get is an example
that clearly shows that it's a good idea to use ffi.include(). In
your "module a module b" senario, we could as well say that you can
wrap them both in a single ffi.cdef()/ffi.verify(): modules a and b
may be sufficiently close for that. The opposite extreme is that

Re: [python-cffi] multiple ffi modules

2013-05-18T11:05:36Z

The problem is as folllows:

module b is an extension of module a. Now you receive from module a
some struct and you want it to be passed to module's b function (or
the other way around). You must then reuse the original ffi, which
maybe makes some sense, but I think ffi include is cleaner, especially

Re: [python-cffi] multiple ffi modules

2013-05-18T10:57:34Z

Hi Fijal,

Two use cases I can think of:

1. in the normal case each library has its own interface and so, in
Python, its own FFI instance and its own Pythonic layer to manipulate
it.

2. in the case of libraries that are "extensions" of other libraries,
adding some features but keeping the same basic functions and types,

Re: [python-cffi] multiple ffi modules

2013-05-18T10:22:35Z

I guess so, but what if you want to use a library from a library? C
also has just one FFI instance that's global :-P

Re: [python-cffi] Zero-copy

2013-05-18T06:56:23Z

Hi Geert,

Not reliably. For example on PyPy string objects are not
null-terminated and can move around memory, so they are not suitable
for taking a raw "char *". On CPython only, there is a hack that will
take a pointer inside the string if passed as argument to a function.

To really avoid copies you need to find a way to rewrite what you need

Re: [python-cffi] multiple ffi modules

2013-05-18T06:39:26Z

Hi Fijal,

Ok. Please submit an issue so that we can track it:
[link]

As far as I can tell this is already implemented. (But I'm a bit
skeptical about the need anyway. If it was C you'd just stick it all
in one namespace. The equivalent is to use a single FFI() instance.)

multiple ffi modules

2013-05-17T14:33:09Z

There was a request that i perfectly understand for a case of mixing
types between two libraries. There are two issues:

a) we need better error reports (those two things come from different
FFI instances and not <cdata x> is not <cdata x>)

b) ffi.include(other_ffi) for making types available?

Zero-copy

2013-05-16T22:42:15Z

Hi,
is there a way to pass a Python string to a C function without the string
data being copied?
Regards,
Geert

Pointer equality and hashes

2013-05-16T15:16:19Z

Hi all,

Warning: I just fixed the following a bug where, given two cffi
pointer objects of different types that point to the same location,
you have:

x = ffi.cast("char *", p)
y = ffi.cast("int *", p)
assert x == y # this is true
assert hash(x) == hash(y) # but this fails!

Re: [python-cffi] cffi performance on CPython and PyPy

2013-05-11T20:42:31Z

I think this test is a red herring, in that I think gcc simply saw the
loop and thought "return 100000000;".

Andreas

Re: [python-cffi] Re: CFFI performance vs Python C extension

2013-05-14T09:05:43Z

Hi Armin,

I'm unsure, but I suspect that this is the problem:

OK, this was the problem, together with the problem that _kfj identified
where I am allocating two 'int *' cdata objects per call. Creating even a
small cdata object appears expensive. I've updated the code in my code. It
now pre-allocates the int arrays, and does not create the buffer but

Re: [python-cffi] Re: CFFI performance vs Python C extension

2013-05-13T09:08:16Z

to malloc, even if you only allocate 5 bytes for an int, and when the
object is no longer referenced, the memory is freed by a call to free. To
really remove these effects, you should avoid all calls to ffi.new in your
inner loop. Removing the other two calls to ffi.new, judging from what you
report about having removed one of them, might therefore save you another

Re: cffi performance on CPython and PyPy

2013-05-12T09:04:40Z

Hi,
thanks for clarification :)
Regards,
Dimitri
Am Samstag, 11. Mai 2013 21:03:20 UTC+2 schrieb Dimitri Vorona:

Cpython/Pypy cffi wrapper for libeinspline

2013-05-12T08:56:18Z

Hi group!
As I've mentioned previously on this list, I've been working on wrapping
libeinspline, a comprehensive C library for creating and evaluating cubic
B-splines in 1-3 dimensions and real and complex data types. I've now put
an alpha version online at
[link]

Re: [python-cffi] cffi performance on CPython and PyPy

2013-05-12T06:48:30Z

Hi again,

Fixed in 9ffefaf25ca3: distutils/sysconfig_pypy.py didn't pass any -O
option to the C compiler. Now going with -O2, which is the standard
(by default) on CPython, as far as I know. Your super-fast CPython
results might have been obtained with -O3 or some specific aggressive
flags, but honestly I don't think there is a point in specifying

Re: [python-cffi] cffi performance on CPython and PyPy

2013-05-12T06:27:48Z

Hi Dimitri,

At this point the loop in C is long enough that you're benchmarking
the C compiler only. The indicates different optimization options
passed to the C compiler. From the results, the CPython version calls
the compiler with good optimizations flags, and the C compiler figures
out that the loop is nonsense and removes it completely (hence it's

Re: [python-cffi] Re: CFFI performance vs Python C extension

2013-05-12T06:23:33Z

Hi Geert,

I'm unsure, but I suspect that this is the problem:

It creates a char[] buffer and copies the string "buf" in it. If I
see it correctly, the CPython extension module doesn't do any copying.
I guess the benchmark takes a big string and calls split_json() a lot of times
to get various positions in this string. Then it would be a much

Re: [python-cffi] cffi performance on CPython and PyPy

2013-05-11T19:57:47Z

Hi Alex,
see [link] .
Loaded the gist on IPython both with PyPy and CPython run timeit lib.test()
and got the results mentioned above. Is it some IPython artifact, maybe?
Regards,
Dimitri.
Am Samstag, 11. Mai 2013 21:04:57 UTC+2 schrieb Alex Gaynor:

Re: [python-cffi] Re: CFFI performance vs Python C extension

2013-05-11T19:52:25Z

Hi,

[...]
Thanks Kay. I actually did check this by caching c_buf, and it made a bit
of difference (roughly 30%). But as-is the code is 8x slower when called
via CFFI. So even after this modest speedup there is an almost
order-of-magnitude difference in performance.

Regards,
Geert

Re: [python-cffi] CFFI performance vs Python C extension

2013-05-11T19:49:41Z

This is already on PyPy 2.0. Regarding b), I am looking for a way to use an
already optimized C function in an efficient way from PyPy and CPython. I
hoped CFFI would offer me that. Why should re-coding it in Python help? The
JIT can do a good job but why should it be faster than compiled and hand

Re: cffi performance on CPython and PyPy

2013-05-11T19:05:57Z

Make it six orders of magnitude :)

Re: [python-cffi] cffi performance on CPython and PyPy

2013-05-11T19:04:57Z

Hi Dimitri,

I'm a bit confused about what you're benchmarking, can you include the full
script you ran?

Alex

--
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero

cffi performance on CPython and PyPy

2013-05-11T19:03:20Z

Hi,
while playing around with CFFI I decided to make a silly speed test of with
10 ^ 8 iteration of a simple increment operation, i.e.:
long test() {
long i, s = 0;
for (i = 0; i < 100000000; i++) {
s++;
}
return s;
The results with time it are:
PyPy: 211ms
CPython 74.1ns(!)

Re: CFFI performance vs Python C extension

2013-05-11T18:17:52Z

def cffi_split(buf, offset=0, state=0):
c_buf = ffi.new('char[]', buf)
c_offset = ffi.new('int *', offset)
c_state = ffi.new('int *', state)
ret = lib.split_json(c_buf, len(buf), c_offset, c_state)
return (ret, c_offset[0], c_state[0])
every invocation of this function allocates three buffers, while three

Re: [python-cffi] CFFI performance vs Python C extension

2013-05-11T15:37:09Z

hey

Two things:

a) you can try newer pypy
b) on pypy you can try writing this function in Python, should be faster

Cheers,
fijal

CFFI performance vs Python C extension

2013-05-11T06:28:19Z

Hi,
I have a C function for framing JSON objects from a stream. This function
is written in C for speed. Originally this function was exposed via a
python extension module. I have been doing an experiment where I exposed
this function via CFFI. My goal is to have one high performance function
that I can share between CPython and PyPy. The results are below:

Re: [python-cffi] docstrings for cffi-wrapped C entities

2013-05-10T10:06:38Z

Nice indeed...

Gulp... I'm just right inside the innards of the fat C library I'm
wrapping. I'm a cffi *user*, really; I haven't even looked into the code
(yet). Okay, I'll put it on the to-do-list and once I've recovered from my
current project I may give it a shot ;-)
Kay

Re: [python-cffi] docstrings for cffi-wrapped C entities

2013-05-07T08:35:59Z

Hi Kay,

Good point. Right now you only have ffi.typeof(c_function), which
gives the signature as a string reconstructed from the internal
representation, without the names of formal parameters. It would be
nice if these c_function's also had a __doc__ that contains, say, the
original declaration. Patches welcome, and bonus points if the

Re: [python-cffi] struggling with importing and packaging cffi-generated modules

2013-05-07T08:05:06Z

Hi Kay,

Sorry for the delay in answering this mail:

What's passed to verify is *never* parsed: only gcc sees it. It is
hashed though, to check if something changed.

No, that's fine.

Sorry for not being clear. This is not something that works now.
It's only an idea about the future.

docstrings for cffi-wrapped C entities

2013-05-04T09:42:24Z

Hi group!
The cffi-wrapped C code has a good notion of the number and type of
arguments functions accept, the values needed to initialize a structured C
data type etc. - and it should even know the names of formal parameters.
I would like to somehow extract this information and ideally have it appear

Re: [python-cffi] cffi to unittest a c module

2013-05-02T17:01:46Z

Hi Alexander,

On Thu, May 2, 2013 at 4:57 PM, Alexander Eisenhuth

Interestingly, I did just that myself :-) It's slightly cumbersome:
as you say you have to repeat ffi.cdef("...") by copy-pasting a large
portion of the .h file, editing it as necessary to replace a few
things with ellipsis ("...", i.e. dot-dot-dot). But overall it seems

cffi to unittest a c module

2013-05-02T14:57:48Z

Hello,
I want to figure out if it is easy to unittest a C-module with python-ffi.
For that reason I wrote a little C-Project( RefTypes.h, Device.c, Device.h)
and test_device.py that I've attached.
Question to test_device.py:
- I tried to implement a Callback with a Uin32* as Parameter, but I failed.

Re: using hierarchical shared libraries, best practices

2013-05-01T15:45:16Z

libtest1.so, but to have the glue code in a python sting, cdef() the
prototypes and feed the code itself it into verify(). Anyway, judging from
my (limited) experience, the expensive step in the whole process is
definitely not the call to verify, but the cdefs. This surprised me when I
looked into the matter, but AFAICT it's due to the fact that the cdef

Re: [python-cffi] pypy cffi callback crashes when using multithread

2013-04-28T03:43:17Z

Thanks, we've been using CPython with this kind of callbacks nearly a year,
and it works fine for us.
Recently we want to try Pypy.
Since it does not work on Pypy, we've just implemented a Python thread and
wait for the C thread to complete, and do the callback stuff, so that the
system emulate the callback function.

Re: [python-cffi] Re: Using verify(source, libraries=[mylib]) and tmpdir parameter

2013-04-27T19:37:34Z

Thanks, 'include_dirs' variable solved the problem very elegantly.

Re: [python-cffi] pypy cffi callback crashes when using multithread

2013-04-27T18:53:37Z

Hi,

I'm just saying, I'm not sure it works correctly on top of CPython
too. It might work by luck only in the particular case you're using.

A bientôt,

Armin.

Re: [python-cffi] Re: Using verify(source, libraries=[mylib]) and tmpdir parameter

2013-04-27T18:51:42Z

Hi,

The official answer is to use ffi.verify(...., include_dirs=['/path/to/x']).

A bientôt,

Armin.

Re: Using verify(source, libraries=[mylib]) and tmpdir parameter

2013-04-27T17:11:27Z

"/home/you/yourdir/test1.h". Then the C compiler can find it. And if you
put the header into a standard place, you can use #include <test1.h> syntax.
Kay

Re: [python-cffi] using hierarchical shared libraries, best practices

2013-04-27T15:38:18Z

Actually, I figured it out. I need to call dlopen in test1.cc to open
shared library,
because when I tried to set ffi.RTLD_LAZY in lib =
ffi.dlopen('libtest1.so', ffi.RTLD_LAZY)
the response is:

test1 # it works
python: symbol lookup error: libtest1.so: undefined symbol: test2

PS: Sorry for spamming so many answers, but at least I figured it out.

Re: [python-cffi] pypy cffi callback crashes when using multithread

2013-04-27T15:36:40Z

thanks. I think I have to find another way to solve my problem when using
Pypy instead waiting for them to fix this.

Armin Rigo於 2013年4月27日星期六UTC+8下午4時16分54秒寫道：

Re: [python-cffi] using hierarchical shared libraries, best practices

2013-04-27T15:24:43Z

Very interesting is that
*lib = ffi.dlopen('libtest2.so')*
*lib.test2()*
*
*
*works fine!*
*
*

Re: [python-cffi] using hierarchical shared libraries, best practices

2013-04-27T15:20:47Z

See the example code which fails int the attached test4.zip

Re: [python-cffi] using hierarchical shared libraries, best practices

2013-04-27T14:49:43Z

*ffi.dlopen('libtest1.so')*
throws an error
*library not found: 'libtest1.so*

I tried setting LD_LIBRARY_PATH to the directory with *libtest1.so*,
but that also did not helped, so I tried to use *ffi.verify*

Ondra