Distribute including a (pre-)compiled library
335 views
Skip to first unread message
ecastro...@gmail.com
Apr 14, 2016, 2:58:13 PM4/14/16
to python-cffi
Hi!
I'm trying to distribute a package that includes a compiled library. Most project I've seen out there link the CFFI wrapper to some library that's already installed on the system, but in my case I need to somehow link to it after installing the package or with a relative path.
The thing is setuptools will run the ffi.compile within a temporary directory (AFAIK) and then copy all files to site-package (or whatever the destiny is), so if I pass an absolute path (dinamically constructed at runtime, ie: os.path.abspath(...)) to ffi.set_source libraries then later it will try to find the library at the non-existent tmp location, and if I pass a relative url, then (I think) it will try to find it at a relative path to the current working directory or LD_LIBRARY_PATH.
The solutions I can think of are:
* Add a post-install command to run ffi.compile and passing an absolute path.
* Distribute the (pre-)compiled library as a separate package, then add it as a dependency to requirements.txt.
Is there a better way?
I'm trying to distribute a package that includes a compiled library. Most project I've seen out there link the CFFI wrapper to some library that's already installed on the system, but in my case I need to somehow link to it after installing the package or with a relative path.
The thing is setuptools will run the ffi.compile within a temporary directory (AFAIK) and then copy all files to site-package (or whatever the destiny is), so if I pass an absolute path (dinamically constructed at runtime, ie: os.path.abspath(...)) to ffi.set_source libraries then later it will try to find the library at the non-existent tmp location, and if I pass a relative url, then (I think) it will try to find it at a relative path to the current working directory or LD_LIBRARY_PATH.
The solutions I can think of are:
* Add a post-install command to run ffi.compile and passing an absolute path.
* Distribute the (pre-)compiled library as a separate package, then add it as a dependency to requirements.txt.
Is there a better way?
Armin Rigo
Apr 14, 2016, 5:20:54 PM4/14/16
to pytho...@googlegroups.com
Hi,
On 14 April 2016 at 20:58, <ecastro...@gmail.com> wrote:
> I'm trying to distribute a package that includes a compiled library. Most
> project I've seen out there link the CFFI wrapper to some library that's
> already installed on the system, but in my case I need to somehow link to it
> after installing the package or with a relative path.
Note that, at least on Linux and OS/X, it is more messy: a library or
program that depends on another library normally does not contain any
path, and the system will look for that library in the standard paths
(or in LD_LIBRARY_PATH). The fact that the library was in a temporary
location when you compiled doesn't mean that the path to that location
is included in the binary---it is not. You need semi-independently to
make sure that gcc finds the library when compiling, *and* that the
system finds the library when running. For the runtime, there are
ways based on "rpath" to specify the path to the library, but they are
very, very, very much platform-specific (and hackish).
On Windows, at least, you can place a DLL inside the same directory as
the DLL/EXE that uses it, and it will be found.
The only clean solution I can recommend is that if you want to
distribute some library together with its cffi wrapper, you should
arrange to compile everything as a single library, instead of two (the
original library + the Python cffi wrapper library). Then there is no
problem. There are two ways:
* something like ``ffi.set_source(..., ...,
sources=[list-of-C-files])``, if you can convert the original
library's Makefile specificities to extra set_source() arguments
* or, use ``ffi.emit_c_code()`` instead of ``ffi.compile()``, and then
compile the produced C file together with the rest of the library,
e.g. by adding it manually to the library's own Makefile. (Might be a
bit messy because you might also need to set up paths like
libpython's, which would be done for you by ``ffi.compile()``.)
A bientôt,
Armin.
On 14 April 2016 at 20:58, <ecastro...@gmail.com> wrote:
> I'm trying to distribute a package that includes a compiled library. Most
> project I've seen out there link the CFFI wrapper to some library that's
> already installed on the system, but in my case I need to somehow link to it
> after installing the package or with a relative path.
program that depends on another library normally does not contain any
path, and the system will look for that library in the standard paths
(or in LD_LIBRARY_PATH). The fact that the library was in a temporary
location when you compiled doesn't mean that the path to that location
is included in the binary---it is not. You need semi-independently to
make sure that gcc finds the library when compiling, *and* that the
system finds the library when running. For the runtime, there are
ways based on "rpath" to specify the path to the library, but they are
very, very, very much platform-specific (and hackish).
On Windows, at least, you can place a DLL inside the same directory as
the DLL/EXE that uses it, and it will be found.
The only clean solution I can recommend is that if you want to
distribute some library together with its cffi wrapper, you should
arrange to compile everything as a single library, instead of two (the
original library + the Python cffi wrapper library). Then there is no
problem. There are two ways:
* something like ``ffi.set_source(..., ...,
sources=[list-of-C-files])``, if you can convert the original
library's Makefile specificities to extra set_source() arguments
* or, use ``ffi.emit_c_code()`` instead of ``ffi.compile()``, and then
compile the produced C file together with the rest of the library,
e.g. by adding it manually to the library's own Makefile. (Might be a
bit messy because you might also need to set up paths like
libpython's, which would be done for you by ``ffi.compile()``.)
A bientôt,
Armin.
ecastro...@gmail.com
Apr 14, 2016, 5:53:22 PM4/14/16
to python-cffi, ar...@tunes.org
Hi Armin, thanks for the reply.
I see... then the first way is likely the way to go. Sadly the library is in C++ and it seems there are some unresolved issues with that[0], but I will try anyway.
One other question if you don't mind, is it better (as in good practice) to distribute the compiled library+cffi wrapper? I mean for every CPython and Pypy version, instead of compile on install.
I ask coz I have not seen this in the wild but I remember someone recommended it a while ago.
[0] https://bitbucket.org/pypy/pypy/issues/1763/not-using-proper-c-compilers-linker-while
I see... then the first way is likely the way to go. Sadly the library is in C++ and it seems there are some unresolved issues with that[0], but I will try anyway.
One other question if you don't mind, is it better (as in good practice) to distribute the compiled library+cffi wrapper? I mean for every CPython and Pypy version, instead of compile on install.
I ask coz I have not seen this in the wild but I remember someone recommended it a while ago.
[0] https://bitbucket.org/pypy/pypy/issues/1763/not-using-proper-c-compilers-linker-while
Armin Rigo
Apr 15, 2016, 10:47:21 AM4/15/16
to pytho...@googlegroups.com
Hi,
On 14 April 2016 at 23:53, <ecastro...@gmail.com> wrote:
> I see... then the first way is likely the way to go. Sadly the library is in
> C++ and it seems there are some unresolved issues with that[0], but I will
> try anyway.
I have no experience with that myself, but that "should" work. At
least the .c file produced by cffi can be compiled as C++; there is a
test to that extent.
> One other question if you don't mind, is it better (as in good practice) to
> distribute the compiled library+cffi wrapper? I mean for every CPython and
> Pypy version, instead of compile on install.
I cannot comment on that. You would have the same question about
distributing a standard CPython C extension module, and I don't know
the answer to this either. I don't really see the point of
distributing binaries when the time to redo compilation is short.
(Windows is an exception.) But that is only my own point of view.
A bientôt,
Armin.
On 14 April 2016 at 23:53, <ecastro...@gmail.com> wrote:
> I see... then the first way is likely the way to go. Sadly the library is in
> C++ and it seems there are some unresolved issues with that[0], but I will
> try anyway.
least the .c file produced by cffi can be compiled as C++; there is a
test to that extent.
> One other question if you don't mind, is it better (as in good practice) to
> distribute the compiled library+cffi wrapper? I mean for every CPython and
> Pypy version, instead of compile on install.
distributing a standard CPython C extension module, and I don't know
the answer to this either. I don't really see the point of
distributing binaries when the time to redo compilation is short.
(Windows is an exception.) But that is only my own point of view.
A bientôt,
Armin.
ecastro...@gmail.com
Apr 15, 2016, 11:33:36 AM4/15/16
to python-cffi, ar...@tunes.org
El viernes, 15 de abril de 2016, 11:47:21 (UTC-3), Armin Rigo escribió:
Hi,
On 14 April 2016 at 23:53, <ecastro...@gmail.com> wrote:
> I see... then the first way is likely the way to go. Sadly the library is in
> C++ and it seems there are some unresolved issues with that[0], but I will
> try anyway.
I have no experience with that myself, but that "should" work. At
least the .c file produced by cffi can be compiled as C++; there is a
test to that extent.
It works indeed.
I was about to figure out what the makefile is doing to pass to set_source ad try to compile everything with cffi (distutils), but that would be nightmare-ish to reproduce for every platform and then maintain it to keep it up with every new release of the library (google's V8). The makefile will produce static files so I decide it to try to link agains them, it seems to work well so far.
Here is the set_source in case someone else find it useful:
ffi.set_source(
"_myfoo",
"""
#include "foo.h"
"""
language='c++',
source_extension='.cpp',
extra_compile_args=['-std=c++'],
extra_links_args=['lrt', '-ldt', '-std=c++'],
include_dirs=['foo'],
sources=['foo/mybar.cpp'],
extra_objects=['-Wl,--start-group', 'bar/mybaz.a', 'bar/myfoz.a', '-Wl,--end-group'])
Apparently it's possible to pass args to extra_objects although the docs does not say so.
ecastro...@gmail.com
Apr 15, 2016, 3:48:57 PM4/15/16
to python-cffi
One last observation, the above works but only if everything gets compiled by the same compiler version. Compiling the static libraries with g++4.8 will throw an error when later compiling the cffi wrapper with g++5.0, so it's not a good idea to distribute the static libraries.
I wish there was a way to run the makefile before the cffi.compile.
I wish there was a way to run the makefile before the cffi.compile.
ecastro...@gmail.com
Apr 15, 2016, 5:23:27 PM4/15/16
to python-cffi
Oops, it seems the issue was not related to the g++ version but to the fact the generated archives (.a) are "thin", converting them into normal arhives[0] solve the issue.
PS. Sorry about the flood of posts.
[0] http://stackoverflow.com/questions/25554621/turn-thin-archive-into-normal-one
PS. Sorry about the flood of posts.
[0] http://stackoverflow.com/questions/25554621/turn-thin-archive-into-normal-one
Reply all
Reply to author
Forward
0 new messages