Best practive for packaging a cffi project in ABI mode

[Victor Blomqvist]

Hi!

I have a project that Im porting over to use cffi (pymunk), and now Ive started looking into how to best package the project.

I have the source code of the c library I use. Sometimes I have had to patch this library myself, or use a specific GIT version of it instead of its released version for bug fixes for features. Its likely that my pyhon code will only work with a specific version of the library. At the moment I have its source included in a subfolder.

In cffi I use ABI mode to wrap it since its a pain to compile the c library on windows otherwise (with ABI mode I can use GCC everywhere)

Currently and before when I used ctypes I used to compile the c-library myself and then put the compiled files inside my package folder. Is this the best way?

My current file structure is:
root/
....chipmunk_src/
........c library source file goes in this folder (and subfolders)
....pymunk/
........__init__.py and other python code files
........chipmunk.dll
........chipmunk64.dll
........libchipmunk.so
........libchipmunk64.so
....setup.py
 
Then in setup.py
1. I have overridden build_clib to build the source in chipmunk_src and output the dll/so/dylib file.
2. Told it to include the dll/so/dylib files when doing source and binary distributions.

On windows I definitely have to ship the dll files, must users cant compile themselves. On OSX/linux Im not even sure if its possible, but so far I have not had any incompatibilities by compiling the .so files myself and including them in the dist.

My questions
1. Is this the proper way to do it or can you suggest me a better way?
2. How to handle wheels? When I built a wheel for windows it specifies that its specific to a python version. However, since the dll itself works on any windows python (I think?) and the python code itself is not bound to a specific python version.

Thanks!
/Victor

[Armin Rigo]

Hi,

On 8 June 2016 at 04:26, Victor Blomqvist <v...@viblo.se> wrote:
> Currently and before when I used ctypes I used to compile the c-library
> myself and then put the compiled files inside my package folder. Is this the
> best way?

There is no "known best way", but I can only repeat the way I
recommend---though it is not better than yours in all aspects. This
is using the modern API mode:

root/
chipmunk_src/
c library source files
pymunk_build.py
pymunk/
python code files

The pymunk_build.py is a build file like
http://cffi.readthedocs.io/en/latest/overview.html#real-example-api-level-out-of-line
. It specifies the C library source files as an extra
``sources=[files]`` argument to set_source(). In this way, by running
pymunk_build.py once, you compile both the C library and the CFFI
bindings together, and produce a single .dll/.so that can be used
directly from Python ("import ..."). Just like your solution, you get
a single .dll/.so. The advantage is that you don't have to use the
ABI mode (the API mode is safer and faster). The disadvantage is that
the .dll/.so is specific to a Python version.

This disadvantage could also be considered an "advantage", as this is
a standard case for Python's wheel packaging system. (In your
solution, I don't know if you can hack wheels to get a binary wheel
that is still independent on the Python version. Maybe you should not
use wheels at all for that, but somehow consider the .dll as a data
file that happens to be distributed with your package.)


A bientôt,

Armin.

[Victor Blomqvist]

At the moment the c library cant be compiled directly with MSVC, only
with GCC, and there are no (that I know of) distributions of GCC that can
do that for most python versions used. Therefor it would be quite difficult
to compile it as an extension. Another downside is the need to compile
with a number of different versions of MSVC, which is quite big to install.
Otherwise I can see why that would be a good option.

I have started to use mingwpy which has as its goal to support building
python extensions on windows with gcc, and when it can support most
versions I will consider it again.
 
According to the spec of the wheel it should be able to support
"pure dlls" if I understand it correctly, but it will require an ugly rename
after building. And so far I havent tested it :)

Another interesting finding I made after writing my first email was that
setup.py build_clib which can build c libraries cant include them in the
actual distribution, so for now I use setup.py build_ext instead even
thou its not an extension. Reading around I found that there are
discussions about extending the packaging of pure dlls, but nothing
concrete yet.

Anyway, thanks for the input!

/Victor