Type of a c function is "int(*)(int)" in ABI mode vs "built-in method" in API mode

[Василий Повереннов]

First of all, thank you for such a great library!

I am wrapping a DLL on windows (don't care about other platforms now) and I do have the source code for the DLL, so I decided to switch from ABI mode (out of line) to API mode (and actually want to use sources= to avoid the original DLL entirely)

Here's my example:

from cffi import FFI
common_cdef = """
int call(int (*operation)(int), int value);
extern int (add1)(int);
"""

ffi_abi_in = FFI()
ffi_abi_in.cdef(common_cdef)
lib_abi_in = ffi_abi_in.dlopen('testlib')
print(f'{lib_abi_in.add1=}')
print(f'{ffi_abi_in.typeof(lib_abi_in.add1)=}')
print(f'{lib_abi_in.add1(2)=}')
try:
print(f'{lib_abi_in.call(lib_abi_in.add1, 3)=}')
except TypeError as e:
print(f"can't call {e}")
print()

ffi_api_in = FFI()
ffi_api_in.cdef(common_cdef)
lib_api_in = ffi_api_in.verify('''
int add1(int x) {
return x + 1;
}
int call(int (*operation)(int), int value) {
return operation(value);
}
''')
print(f'{lib_api_in.add1=}')
print(f'{ffi_api_in.typeof(lib_api_in.add1)=}')
print(f'{lib_api_in.add1(2)=}')
try:
print(f'{lib_api_in.call(lib_api_in.add1, 3)=}')
except TypeError as e:
print(f"can't call {e}")
print()

results in

> lib_abi_in.add1=<cdata 'int(*)(int)' 0x00007FFA97D21000>

> ffi_abi_in.typeof(lib_abi_in.add1)=<ctype 'int(*)(int)'>
> lib_abi_in.add1(2)=3
> lib_abi_in.call(lib_abi_in.add1, 3)=4

> lib_api_in.add1=<built-in function add1>
> ffi_api_in.typeof(lib_api_in.add1)=<ctype 'int(*)(int)'>
> lib_api_in.add1(2)=3
> can't call initializer for ctype 'int(*)(int)' must be a cdata pointer, not 

> builtin_function_or_method

so the python type for add1 seems to be different, I can call the function in both cases, but can only pass it as a callback to C in the ABI mode. In-line / out-of-line does not matter. 

How can I make API mode behave like the ABI mode here so I can pass the function back into C?

Hope that's enough information, I can put a complete repro on github if needed, but I'm assuming I'm missing something obvious

[Armin Rigo]

Hi,

On Mon, 18 Dec 2023 at 09:29, Василий Повереннов <bazoo...@gmail.com> wrote:
> > can't call initializer for ctype 'int(*)(int)' must be a cdata pointer, not
> > builtin_function_or_method
>
> so the python type for add1 seems to be different, I can call the function in both cases, but can only pass it as a callback to C in the ABI mode. In-line / out-of-line does not matter.

Indeed. I'll answer, but first a quick note that you are using
`ffi.verify()` in your example---I assume it's only for this example.
The `verify()` method is very much deprecated; you should run
`ffi.set_source()` and `ffi.compile()` in a separate script to run the
compiler ahead of time.

The problem you're running into is that indeed `lib.myfunc` does not
return an object that can be directly passed as a C callable object,
if you're using the API mode. For performance reasons `lib.myfunc` is
a "built-in function object". The trick is
https://cffi.readthedocs.io/en/stable/ref.html#ffi-offsetof-ffi-addressof:
you need to write `ffi.addressof(lib, 'myfunc')` to get a C data
object that contains a raw pointer to the C function.


A bientôt,

Armin Rigo

[Vasiliy Poverennov]

Thanks for the swift reply!

I am using out-of line mode (with .compile and a separate script), that's handled by setuptools when building the wheel, I wanted to go from using ABI mode to API mode, unfortunately that changes the interface (I'm exposing ffi and lib directly from my package)

I guess I'll have to expose a wrapper that replaces those functions with ffi.addressof(lib, 'myfunc') or break the backwards compatibility

Unless there a way to change the cdef / ask cffi nicely so it exposes the function as CData instead of a "built-in function object"?

[Armin Rigo]

Hi,

On Mon, 18 Dec 2023 at 20:28, Vasiliy Poverennov <bazoo...@gmail.com> wrote:
> Unless there a way to change the cdef / ask cffi nicely so it exposes the function as CData instead of a "built-in function object"?

Yes! You can declare them in the cdef as "constant function pointers"
instead of regular functions:

ffi_builder.cdef("""
int (*const add1)(int);
""")

Then, when you read `lib.add1`, you get a `<cdata 'int(*)(int)'>`.
This is the same as you get in ABI mode with the `int add1(int);`
declaration, but in API mode it bypasses the optimization.


A bientôt,

Armin.

[Vasiliy Poverennov]

That's perfect, thanks

For my use-case I am only passing the callback back to C, not calling from python, so I don't care if I lose the optimization for this method. And I'm assuming in this case it won't be worse than manually doing ffi.addressof.
Overall I believe it would still be better to switch from ABI to API mode.

One interesting thing I noted when testing is that in ABI in-line mode this definition gives an error, and in out-of-line mode it crashes when I try to call it direct / pass into lib.call. Just curious - is this by design?

> NotImplementedError("non-integer constant 'add1' cannot be accessed from a dlopen() library")

> lib_abi_out.add1=<cdata 'int(*)(int)' 0x24244C8928EC8348>

> Process finished with exit code -1073741819 (0xC0000005)

from cffi import FFI

common_cdef = """
int call(int (*operation)(int), int value);

int (*const add1)(int);

"""

ffi_abi_in = FFI()
ffi_abi_in.cdef(common_cdef)
lib_abi_in = ffi_abi_in.dlopen('testlib')

try:
    print(lib_abi_in.add1)
except NotImplementedError as e:
    print(e)
print()

ffi_abi_out = FFI()
ffi_abi_out.cdef(common_cdef)
ffi_abi_out.set_source('_ffi_abi_out', None)
ffi_abi_out.compile()
import _ffi_abi_out

lib_abi_out = _ffi_abi_out.ffi.dlopen('testlib.dll')
print(f'{lib_abi_out.add1=}')
print(f'{lib_abi_out.add1(3)=}')
print()

[Armin Rigo]

Hi,

On Tue, 19 Dec 2023 at 11:00, Vasiliy Poverennov <bazoo...@gmail.com> wrote:
> One interesting thing I noted when testing is that in ABI in-line mode this definition gives an error, and in out-of-line mode it crashes when I try to call it direct / pass into lib.call. Just curious - is this by design?

Ah. OK, my mistake, then it won't work in any ABI mode. You get an
error in in-line ABI mode, and in the slightly less tested out-of-line
ABI mode you get a crash, which (I think now) is because cffi takes
the address of the function, and then reads the first 8 bytes there
and assumes it's a function pointer---but it is not, it's the start of
the code for the function. Sorry. In my test I get `<cdata
'int(*)(int)' 0xe5894855fa1e0ff3>`; that number is definitely not a
valid pointer and looks more like 8 bytes of executable code.

For now you'll have to use the "const function pointer" as a
workaround for the API mode only. A possibility in the future would
be in some cases to allow the "builtin function" objects from API libs
to be passed at places that expect a cdata object, like you can pass,
say, an integer where in theory you need a cdata object containing an
integer, and there is automatic conversion.


A bientôt,

Armin Rigo

[Armin Rigo]

Hi again,

On Tue, 19 Dec 2023 at 13:46, Armin Rigo <armin...@gmail.com> wrote:
> For now you'll have to use the "const function pointer" as a
> workaround for the API mode only. A possibility in the future would
> be in some cases to allow the "builtin function" objects from API libs
> to be passed at places that expect a cdata object, like you can pass,
> say, an integer where in theory you need a cdata object containing an
> integer, and there is automatic conversion.

I've written a pull request for CFFI to add exactly that. You can
follow it at https://github.com/python-cffi/cffi/pull/42 .


A bientôt,
Armin

[Vasiliy Poverennov]

Thank you so much!

This is way beyond my expectations