Wrap C static library

[Yang Wang]

Is it possible to wrap a static C library that was compiled withOUT -fPIC flag?

For an example, jsmn (https://github.com/zserge/jsmn) is a minimal JSON parser in C. When compile, the make command creates a static library called "libjsmn.a" (without -fPIC flag).

I tried to wrap this little library with following cython file called "cjsmn.pyx"

cdef extern from 'jsmn.h':
    ctypedef struct jsmn_parser:
        pass

    void jsmn_init(jsmn_parser *parser)

def init():
    cdef jsmn_parser parser
    jsmn_init(&parser)


and the simple "setup.py" file is as the follows:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

ext = Extension('cjsmn',
        sources=['cjsmn.pyx'],
        include_dirs=['../jsmn'],
        library_dirs=['../jsmn'],
        libraries=['jsmn'],
        )

setup(ext_modules=cythonize(ext))


But it does NOT compile (python setup.py build_ext -i) and the error message is:

/usr/bin/ld: ../jsmn/libjsmn.a(jsmn.o): relocation R_X86_64_32S against `a local symbol' can not be used when making a shared object; recompile with -fPIC
../jsmn/libjsmn.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

If I re-compile the static library with "-fPIC" as suggested by the error message, everything works. But is it possible to fix this problem withOUT recompile the static library?? Because I need to wrap some other static C libraries that I do NOT have the source to recompile ..

Any help is appreciated!!

[Yury V. Zaytsev]

On Wed, 9 Mar 2016, Yang Wang wrote:

> If I re-compile the static library with "-fPIC" as suggested by the
> error message, everything works. But is it possible to fix this problem
> withOUT recompile the static library?? Because I need to wrap some other
> static C libraries that I do NOT have the source to recompile ..

"In computing, position-independent code (PIC) or position-independent
executable (PIE) is a body of machine code that, being placed somewhere in
the primary memory, executes properly regardless of its absolute address.
PIC is commonly used for shared libraries, so that the same library code
can be loaded in a location in each program address space where it will
not overlap any other uses of memory (for example, other shared
libraries)."

https://en.wikipedia.org/wiki/Position-independent_code

On some architectures like x86_64 it is *required* that shared libraries
(*.so) are PIC-enabled in order for them to work on Linux.

Since you are not just linking the objects from static library in question
into an executable, but trying to make a shared library which should be
dynamically loadable by Python out of it, you have no choice but to enable
PIC on x86_64. So, I'm sorry, but the answer is "no". If you don't have
the source for other libraries, you'll have to get the authors to
recompile them with -fPIC so that you can use them for your purposes.

--
Sincerely yours,
Yury V. Zaytsev

[Yang Wang]

Thanks for the reply. 
It is very helpful to get the answer even it is a "no". Now I can stop wasting time on trying something impossible. Instead, I'll try get the author to recompile it. Thanks again!

[Chris Barker]

hmm -- I have built wrappers around static libs by including the lib with the source for the linker to add in, rather than having the dynamic linker do it -- is it possible that that is a different story?

I did this with the "extra_objects" setup keyword argument:

                        extra_objects=["./static_libs/lib/libgd.a",

                                                 "./static_libs/lib/libpng.a"],

worth a shot.

-CHB

 

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Ian Henriksen]

On Tuesday, March 15, 2016 at 10:55:06 AM UTC-6, Chris Barker wrote:

hmm -- I have built wrappers around static libs by including the lib with the source for the linker to add in, rather than having the dynamic linker do it -- is it possible that that is a different story?

I did this with the "extra_objects" setup keyword argument:

                        extra_objects=["./static_libs/lib/libgd.a",

                                                 "./static_libs/lib/libpng.a"],

I'd be really surprised if that worked with the default settings if those objects weren't build with the `-fPIC` flag set.

That said, supposedly you can build a shared object with objects built without `-fPIC` if you specify `-mimpure-text`

when building the shared library. See http://stackoverflow.com/a/1590050/1935144 for more details.

If you have any say in the matter I'd still recommend going with `-fPIC` though.

Best,

-Ian Henriksen

[Yury V. Zaytsev]

On Tue, 15 Mar 2016, Ian Henriksen wrote:

> I'd be really surprised if that worked with the default settings if
> those objects weren't build with the `-fPIC` flag set. That said,
> supposedly you can build a shared object with objects built without
> `-fPIC` if you specify `-mimpure-text` when building the shared library.
> See http://stackoverflow.com/a/1590050/1935144 for more details.

Good point, theoretically you can shut up gcc with `-mimpure-text`, but
this causes "the necessary relocations to trigger copy-on-write, and the
shared object is not actually shared across processes", which kind of
defeats the purpose, not even speaking of the security implications, etc.

Speaking of evil hacks, one could also build the static library with
`-mcmodel=large`, but then one could just as well rebuild it with `-fPIC`:

http://eli.thegreenplace.net/2011/11/11/position-independent-code-pic-in-shared-libraries-on-x64

So the blanket answer is still "it's not possible, forget anything you've
read here and go ask to recompile with `-fPIC`" ;-)