Module initialization on PyPy

[Alex Gaynor]

Hi all,

I'm one of the developers of PyCA cryptography. We've been using cffi
for many many years at this point.

Right now we're looking at doing something novel: rather than having
cffi emit a module that's normally imported, instead have it emit an
object file which is linked into another module, and that other module
is responsible for calling `PyInit_{module}` and adding it to the
module hierarchy. By putting both modules in a single .so, we ensure
that they link the same version of external libraries (in our case,
OpenSSL).

You can read about the motivation for this
https://github.com/pyca/cryptography/issues/6634 and see the PoC here
https://github.com/pyca/cryptography/pull/6695

Amazingly, this all seems to work fine, but only on CPython. On PyPy,
we encounter linker errors, because cffi modules on PyPy do not
actually emit `PyInit_{module}` symbols! They have an entirely
different API for module initialization
(https://foss.heptapod.net/pypy/pypy/-/blob/branch/default/pypy/module/_cffi_backend/cffi1_module.py#L20).

What's the best way to accomplish this? Essentially we need some API
that can find a function ptr inside the same .so as the calling code
and return the freshly created module for it.

I suspect this requires a new API from cffi, but I'd appreciate any thoughts.

Alex

--
All that is necessary for evil to succeed is for good people to do nothing.

[Matti Picus]

Perhaps the PyPy-cffi backend could emit a `PyInit_{module}` function that would encapsulate the code you pointed to. I don't think this needs any changes in cffi itself, rather the PyPy-specific _cffi_backend could be adapted to emit this code. Of course this is all just an idea. I opened https://foss.heptapod.net/pypy/pypy/-/issues/3619 to discuss this further.

[Alex Gaynor]

Thanks Matti!

I don't quite follow how cffi_start_python works so I'm not sure it's
sufficient, but if it works I'm happy with it :-)

If we'd like something even simpler, I think exposing
load_cffi1_module via cpyext (e.g. PyObject *PyPyLoadCFFIModule(char
*name, INITFUNCPTR initptr)) would be enough for what we'd like to do!

Thanks,
Alex

> --
> -- python-cffi: To unsubscribe from this group, send email to python-cffi...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/python-cffi?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "python-cffi" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to python-cffi...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/python-cffi/f0f3871e-5aa6-4f18-b71b-c7e1e4544efen%40googlegroups.com.