CFFI distribution questions

[Israel Brewster]

I originally posted this at pytho...@python.org, but after some consideration (and getting no responses there after 24 hours), I thought it might be more appropriate here.

I have a python module that I have written which uses CFFI to link against a C library I have compiled. Specifically, it is a Database driver for the 4th dimension database, using an open-source C library distributed by the 4D company. I have tested the module and C code on a couple of different platforms, but I have a few questions regarding where to go from here.

1) Currently, I am manually compiling the C library, and placing it in a subfolder of the module. So the top level of the module directory contains the python code and the __init__.py file, and then there is a sub-directory (lib4d_sql) containing the C code and compiled C library. I point CFFI to the library using a construct like this:

_CWD = os.path.dirname(os.path.realpath(__file__))
ffi.verifier(..., Library_dirs=["{}/lib4d_sql/].format(_CWD), ...)

Obviously I have left out a lot of code there. Is this sort of construct considered kosher? Or is there a better way to say "this directory relative to your location"? I can't just use relative paths, because that would be relative to the execution directory (I think).

2) What is the proper way to compile and distribute the C library with the python module? 

The examples I've found about distributing a CFFI module all assume you are are using some brain-dead-simple built-in C command where you don't have to worry about compiling or installing a library. I found some documentation related to building C and C++ extensions with distutils, and following that managed to get the library to compile, but I can't figure out what it does with the compiled library, or how to get it into the proper location relative to my module. I also found some more "raw" distutil code here: https://coderwall.com/p/mjrepq/easy-static-shared-libraries-with-distutils that I managed to use by overriding the finalize_options function of the setup tools install class (using the cmdclass option of setup), and this allowed me to build the library in the proper location in the tmp install directory, but it doesn't seem to keep the library when installing the module in the final location. So far, the only way I have found to work around that is by including a "dummy" copy of the library in the package_data option to setup, such that when the library is built it replaces this dummy file. Is there a better/more canonical way to do this?

3) The majority of the setup/packing procedure I pulled from here: https://caremad.io/2014/11/distributing-a-cffi-project/ and it seems to work - mostly. The one problem I am running into is with the implicit compile that CFFI does. Some of the code (most, really) on that page is designed to work around this by doing the compile on install so it doesn't have to be done at runtime, however this doesn't appear to be working for me. I see signs that it is doing the compile at install time, however when I try to run the module it still tries to compile at that time, an operation that fails due to lack of permissions. Might anyone have some thoughts on how I can fix this?

Finally, let me ask you guys this: is distributing this as a CFFI module even the right approach? Or should I be looking at something else entirely? C code isn't needed for speed - the database itself will be way slower than the interface code. It's just that the driver is distributed as C code and I didn't want to have to reverse engineer their code and re-write it all. I've read over https://docs.python.org/2/extending/extending.html, but I think I'm missing something when it comes to actually interfacing with python, and dealing with the non-python types (pointers to a struct, for example) that the C library uses. Would I have to put 100% of my code, including things like the cursor and connection classes, in C code? When I call the connect function from python (which should return an instance of a connection class), what exactly should my C function return? All the examples on that page show basic C types, not pointers to class instances or the like. Maybe I just need to read over the documentation a few more times :-)

Thanks for any help anyone can provide on any of these questions! :-) Sorry for being so long-winded.

-----------------------------------------------

Israel Brewster

Systems Analyst II

Ravn Alaska

5245 Airport Industrial Rd

Fairbanks, AK 99709

(907) 450-7293

-----------------------------------------------

[Armin Rigo]

Hi,

On 26 November 2014 at 19:37, Israel Brewster <macav...@gmail.com> wrote:
> 1) Currently, I am manually compiling the C library, and placing it in a
> subfolder of the module.

That might work, but the CFFI installation instructions generally
assume that you're in one of two cases:

- either you are interfacing with a separate library that is installed
normally into the system (not distributed together with CFFI);

- or you are compiling everything with distutils in "ffi.verify()".

So in your case, try to see if it's possible to have the complete C
library compiled together with the rest of the interfacing code in the
call to ffi.verify(). Something like

ffi.verify("#include <stuff.h>", sources=["lib4d_sql/foo.c",
"lib4d_sql/bar.c"])

...which brings me to your second question: the problem with CFFI
recompiling things after installation is because you specify absolute
paths. You must call ffi.verify() with exactly the same arguments all
the time. It's a bit annoying. One way is to give relative paths
like I did above, and make sure you're running with the correct
current working directory *when not installed*. The point is that
after installation, it should not actually need the .c files any more,
because the compiler should (really this time) not be called any more.

> Finally, let me ask you guys this: is distributing this as a CFFI module
> even the right approach? Or should I be looking at something else entirely?

I think it is absolutely the right approach, yes, but in this list
you're likely to get a slightly biased answer :-)


A bientôt,

Armin.

[Antonio Cuni]

On Thu, Nov 27, 2014 at 9:39 AM, Armin Rigo <ar...@tunes.org> wrote:

 

...which brings me to your second question: the problem with CFFI
recompiling things after installation is because you specify absolute
paths.  You must call ffi.verify() with exactly the same arguments all
the time.  It's a bit annoying.  One way is to give relative paths
like I did above, and make sure you're running with the correct
current working directory *when not installed*.  The point is that
after installation, it should not actually need the .c files any more,
because the compiler should (really this time) not be called any more.

​yes, I confirm that this method works. It's a bit ugly because you have to change and restore the current working directory around the call to verify(), but it solves the problem. A more proper solution would be if cffi supported something like $ORIGIN when specifying paths.

See this code for an example:

https://bitbucket.org/antocuni/cffi-shm/src/fdf8fc4755cc435bdc61fe2ba60fec1f269ac0c9/shm/libcfu.py?at=master​

 

[Armin Rigo]

Hi,

On 28 November 2014 at 10:31, Antonio Cuni <anto...@gmail.com> wrote:
> A more proper solution would be if cffi
> supported something like $ORIGIN when specifying paths.

I think it may be a good idea, but I'm not 100% sure. So here is a
pull request I created in order to get comments from you all, before
possibly merging it:

https://bitbucket.org/cffi/cffi/pull-request/53/allow-specifying-here-in-the-paths-given/diff


A bientôt,

Armin.

[Israel Brewster]

On Wednesday, November 26, 2014 11:40:29 PM UTC-9, Armin Rigo wrote:

Hi,

On 26 November 2014 at 19:37, Israel Brewster <macav...@gmail.com> wrote:
> 1) Currently, I am manually compiling the C library, and placing it in a
> subfolder of the module.

That might work, but the CFFI installation instructions generally
assume that you're in one of two cases:

- either you are interfacing with a separate library that is installed
normally into the system (not distributed together with CFFI);

- or you are compiling everything with distutils in "ffi.verify()".

So in your case, try to see if it's possible to have the complete C
library compiled together with the rest of the interfacing code in the
call to ffi.verify().  Something like

    ffi.verify("#include <stuff.h>", sources=["lib4d_sql/foo.c",
"lib4d_sql/bar.c"])

Indeed it was. Which actually solved another problem as well, which was getting the library to compile at the right time in the install and figuring out where to stick it so that the module could access it at runtime. Having everything compiled in the ffi.verify() command simply removed both of those issues from consideration.

 

...which brings me to your second question: the problem with CFFI
recompiling things after installation is because you specify absolute
paths.  You must call ffi.verify() with exactly the same arguments all
the time.  It's a bit annoying.  One way is to give relative paths
like I did above, and make sure you're running with the correct
current working directory *when not installed*.  The point is that
after installation, it should not actually need the .c files any more,
because the compiler should (really this time) not be called any more.

And now I can also confirm that this works - I was able to make an installation package, move it to a clean machine, install it with easy_install, and use the module, with no fuss. Sweet!

 

> Finally, let me ask you guys this: is distributing this as a CFFI module
> even the right approach? Or should I be looking at something else entirely?

I think it is absolutely the right approach, yes, but in this list
you're likely to get a slightly biased answer :-)

Heh. Perhaps, but then it's also possible that CFFI was only intended for "quick prototyping" or the like, and not for "production/distribution" use :-). Good to know that is not the case, and I can move forward with the current approach.

Thanks much for the responses - I think I have everything up and running now!

A bientôt,

Armin.

[Ghislain Vaillant]

Le vendredi 28 novembre 2014 21:26:42 UTC, Israel Brewster a écrit :

Thanks much for the responses - I think I have everything up and running now!

Could you summarize your solution please ?

I too recently used cffi to wrap a C-library and would be interested to know how to properly distribute it.

Cheers
- Ghis

[Israel Brewster]

On Tuesday, January 13, 2015 at 5:20:22 AM UTC-9, Ghislain Vaillant wrote:

Le vendredi 28 novembre 2014 21:26:42 UTC, Israel Brewster a écrit :

Thanks much for the responses - I think I have everything up and running now!

Could you summarize your solution please ?

For me, the key was letting verify() compile everything. The library I was using was available as easily-compiled source files with no external library dependancies, so this worked out easily. Once I had added all the .c files to the sources argument of verify(),  and gotten the module working properly, I was able to follow the directions here:  https://caremad.io/2014/11/distributing-a-cffi-project/ to create the actual package.

The only "special" things I had to do once I let verify() handle compiling the library (they may be mentioned in the above link, I don't recall) were:

1) Add a manifest file with the line "include <lib_source_directory>/*.h. Otherwise the various header files included with the library weren't included in the package, and compilation failed.

2) Add any extra files needed in the package_data argument of setup() in the setup.py file. In my case, I stored the text for the cdef call in a separate file that I then loaded, since it got rather long by the time I defined all the functions I needed. So I had to add this file to the package_data argument.

 

I too recently used cffi to wrap a C-library and would be interested to know how to properly distribute it.

Well, I can't say if following the method in that blog post is *proper*, but it worked for me :-)

 

Cheers
- Ghis

[Will Spearman]

As someone who also struggled through this process in the last year, I would have also appreciated some more documentation on the distribution process. Then again, I was so delighted by the general ease of use of the CFFI project that I assumed it was my fault for not understanding it.

I'd be happy to contribute the documentation if Mr Rigo sees value in it.

[anatoly techtonik]

On Monday, January 26, 2015 at 5:08:59 AM UTC+3, Will Spearman wrote:

As someone who also struggled through this process in the last year, I would have also appreciated some more documentation on the distribution process. Then again, I was so delighted by the general ease of use of the CFFI project that I assumed it was my fault for not understanding it.

I'd be happy to contribute the documentation if Mr Rigo sees value in it.

Will there be pictures? 

[Armin Rigo]

Hi Will,

On 26 January 2015 at 03:08, Will Spearman <willsp...@gmail.com> wrote:
> As someone who also struggled through this process in the last year, I would
> have also appreciated some more documentation on the distribution process.
> Then again, I was so delighted by the general ease of use of the CFFI
> project that I assumed it was my fault for not understanding it.
>
> I'd be happy to contribute the documentation if Mr Rigo sees value in it.

Sorry for the delay. Yes, any documentation would be nice to have.
This is particularly true if it is done as a rewrite and extension of
(some parts of) the existing documentation, i.e. a patch to
``doc/source/index.rst``. But you can also come up with independent
text that we try to integrate later. Thanks!


A bientôt,

Armin.

[Will Spearman]

I'll see what I can do to create a patch in my spare time.