How do I list a dependent shared library in the Extension (setup.py)?

[Czarek Tomczak]

Hi there,

Making my first steps on Linux with Cython and I have this issue.

When importing a Cython module I'm getting this error:

ImportError: /home/czarek/cefpython/cefpython/cef1/linux/binaries/cefpython_py27.so: undefined symbol: cef_quit_message_loop

I have a libcef.so shared library. I also have a static library for it

called "libcef_dll_wrapper.a" that I include in setup.py Extension

through "libraries=['cef_dll_wrapper']". How do I explicitily

say that there is "libcef.so" and that is where the symbols should

be searched? Right now before importing module I've added this

hack to make it work:

ctypes.CDLL(os.path.join(os.path.dirname(os.path.abspath(__file__)), 

'libcef.so'), ctypes.RTLD_GLOBAL) 

What are the alternatives?

Regards,

Czarek

[Chris Barker - NOAA Federal]

I'm confused:

On Thu, May 2, 2013 at 11:28 PM, Czarek Tomczak <czarek....@gmail.com> wrote:

 

When importing a Cython module I'm getting this error:

ImportError: /home/czarek/cefpython/cefpython/cef1/linux/binaries/cefpython_py27.so: undefined symbol: cef_quit_message_loop

I have a libcef.so shared library. I also have a static library for it

called "libcef_dll_wrapper.a" that I include in setup.py Extension

through "libraries=['cef_dll_wrapper']". How do I explicitily

say that there is "libcef.so" and that is where the symbols should

be searched?

I'm confused -- usually one uses either the .a (static) or the .so (dynamic) lib -- why would you use both?

or is the .a lib a different lib -- i.e. something that calls into the *.so? If so, I think you can just list the shared lib in libraries when setting up the Extension -- just like you do any library required.

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Czarek Tomczak]

The libcef_dll_wrapper.a wraps the C++ api of libcef.so, it just has to be

the way it is.

I've tried adding "libcef.so" to "libraries" in Extension, but getting this error:

Traceback (most recent call last):
  File "./pygtk_.py", line 11, in <module>
    import cefpython_py27 as cefpython
ImportError: libcef.so: cannot open shared object file: No such file or directory

The libcef.so is in the same directory as cefpython_py27 module that is

being imported. I've tried putting it to "lib.target" directory (saw it somewhere),

but no luck.

Cheers,

Czarek

[Bradley Froehle]

Czarek:

For dynamic libraries you need only specify the name of the library --- i.e, the filename but without the leading 'lib' and the trailing '.so'.  If the library isn't in a standard library directory you may need to set the 'library_dirs' argument in your setup.py file.  See http://docs.python.org/2/distutils/setupscript.html#library-options.  Lastly, you may need to set LD_LIBRARY_PATH or a similar environment variable when you run Python so that the dynamic linker knows where to find the cef library.

-Brad

[Chris Barker - NOAA Federal]

On Fri, May 3, 2013 at 10:29 AM, Czarek Tomczak <czarek....@gmail.com> wrote:

The libcef_dll_wrapper.a wraps the C++ api of libcef.so, it just has to be

the way it is.

Is it an option to compile and use libcef as a static lib? That might make things easier. And it appears you are not trying to use a system-installed libdef anyway.

-Chris

[Czarek Tomczak]

Hi Bradley,

Yes, I did as you say, if I did it the other way the error message would

probably look like:

ImportError: liblibcef.so.so: cannot open shared object file: No such file or directory

The library_dirs was also set.

Regards,

Czarek

[Bradley Froehle]

On Friday, May 3, 2013 11:43:00 AM UTC-7, Czarek Tomczak wrote:

Hi Bradley,

Yes, I did as you say, if I did it the other way the error message would

probably look like:

ImportError: liblibcef.so.so: cannot open shared object file: No such file or directory

The library_dirs was also set.

Right.  I mis-read your original e-mail, sorry.

See what this returns: ldd /home/czarek/cefpython/cefpython/cef1/linux/binaries/cefpython_py27.so

Can it find find your libcef.so?  If not, try setting and exporting LD_LIBRARY_PATH and repeating.

-Brad

[Czarek Tomczak]

Chris,

Yes, I could probably compile libcef as a static lib, but I'm not sure this

is a good idea, as I will be creating two cython modules named "cefpython_py27.so"

and "cefpython_py32.so" for different python versions, and this approach

would double the size of my application, wouldn't it? I'm not so sure if this

would even work, as there is a helper application that also uses libcef.so.

No, I'm not using a system-installed libcef, I'm not even sure on how to do that,

but I do not like this idea, I want app to be portable, I don't want any DLL hell.

Regards,

Czarek

[Czarek Tomczak]

Thanks for reply,

I've tried the command, here is the output:

ldd /home/czarek/cefpython/cefpython/cef1/linux/binaries/cefpython_py27.so
linux-gate.so.1 =>  (0xb7706000)
libcef.so => not found
libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb7540000)
libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb7521000)
libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb7506000)
libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb735d000)
libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xb7331000)
/lib/ld-linux.so.2 (0xb7707000)

After setting LD_LIBRARY_PATH it was found:

linux-gate.so.1 =>  (0xb770f000)
libcef.so => /home/czarek/cefpython/cefpython/cef1/linux/binaries/libcef.so (0xb434b000)
libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb4251000)
libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb4232000)
libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb4217000)
libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb406e000)
libX11.so.6 => /usr/lib/i386-linux-gnu/libX11.so.6 (0xb3f3a000)
libXrandr.so.2 => /usr/lib/i386-linux-gnu/libXrandr.so.2 (0xb3f31000)
librt.so.1 => /lib/i386-linux-gnu/librt.so.1 (0xb3f27000)
.... many many more

 

But I want the app to be portable and just work out of the box (directory),

I'm not convinced to the idea of messing with system variables (LD_LIBRARY_PATH).

Regards,

Czarek

[Daniele Nicolodi]

On 03/05/2013 21:08, Czarek Tomczak wrote:
> But I want the app to be portable and just work out of the box (directory),
> I'm not convinced to the idea of messing with system variables
> (LD_LIBRARY_PATH).

What do you mean by portability here?

If you want to use a dynamically loaded library, you need to tell you
dynamic linker where to find the library. Either you install it in the
system library path, you add the library location to the system library
path, or you add the library location to LD_LIBRARY_PATH.

In another mail you say that you don't want to install the library
system wide. There aren't many options left.

Cheers,
Daniele

[Czarek Tomczak]

Hi Daniele,

A portable app is one that can be extracted from a zip file

and ran instantly without modifying anything on the system.

I'm from a Windows world and searching for a library in current

directory is nothing unusual, looks like Linux can't do that?

I think I'll just stay with the ctypes solution.

ctypes.CDLL(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'libcef.so'), ctypes.RTLD_GLOBAL) 

Anything wrong with that I should be aware of?

Regards,

Czarek

[Chris Barker - NOAA Federal]

On Fri, May 3, 2013 at 12:26 PM, Czarek Tomczak <czarek....@gmail.com> wrote:

A portable app is one that can be extracted from a zip file

and ran instantly without modifying anything on the system.

That is something that is rarely done in Linux....the whole point of shared libs is that they are shared. In your case, you want to share them among only your apps, which makes sense ,but isn't a common use case in the Linux world.

Usually, you would define ceff as a dependency, and count on the package manager to install it for you in the right place. dll hell is much less of an issue on Linux than Windows, because:

shared libs are usually versioned on linux -- you wouldn't have libdef.so, you'd have something like libcef.2.3.0 -- so if some other app needed a different version, they would not get confused.

By the way, is this the Chrome Embedded Framework? Whether it is or not, if it is a general purpose lib, then there are likely packages for it for major Linux distros.

I'm from a Windows world and searching for a library in current

directory is nothing unusual, looks like Linux can't do that?

dll hell is much worse on Windows -- trust me on this -- and the dll search path is a good part of that reason. (and the fact that many dlls don't have version numbers -- what is up with that???)

 

I think I'll just stay with the ctypes solution.

ctypes.CDLL(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'libcef.so'), ctypes.RTLD_GLOBAL) 

Anything wrong with that I should be aware of?

no, but...

 

what other platforms do you want to support? building/loading dlls is not the same on teh 3 major platforms...

You may want to look inot PyINstaller -- it is designed to build stand-alone executables, adn supports Windows Linux and OS-X -- that way, you could do that natural thing with loading *.so libs and let PyInstaller handle it for you.

oh, and you may be able to edit LD_LIBRARY_PATH from within your app, so it would only be changed for that process....though I suppose that's not much different than the ctypes trick.

-Chris

-- 

[Czarek Tomczak]

Thank you Chris for this in-depth insight on the subject, that clears things up.

Regards,

Czarek

[Bradley Froehle]

On Friday, May 3, 2013 12:26:57 PM UTC-7, Czarek Tomczak wrote:

A portable app is one that can be extracted from a zip file

and ran instantly without modifying anything on the system.

You might be able to specify the RPATH in the shared object, so that linux-gate.so.1 knows where to find libcef.so.  I've never done this with Python's setuptools but you might be able to find a workable example out there.

I think I'll just stay with the ctypes solution.

ctypes.CDLL(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'libcef.so'), ctypes.RTLD_GLOBAL) 

Anything wrong with that I should be aware of?

It's not a very elegant solution but it should probably be okay.  I guess your uses will tell you if they run into any problems.

-Brad

[Czarek Tomczak]

Bradley, you're the best :-) Thank you, the RPATH solution worked, I've added

this to my setup.py:

    runtime_library_dirs=[
        './'
    ],

Ye ye ye :)

I've realized that either way I would have to use LD_LIBRARY_PATH as I was

going to release two versions of app, a portable one, and one as a python package,

the packaged one couldn't use the ctypes solution as the path to libcef.so was not 

known from user's script. But now with the RPATH it's all solved.

Regards,

Czarek

[Czarek Tomczak]

Bad news, it only works when running script from within the same

directory that libcef.so resides in, when I put my scripts in some

other directory I'm still getting "cannot open shared object file"

error. Ehhhhhh.

-Czarek

On Saturday, May 4, 2013 2:08:39 AM UTC+2, Bradley Froehle wrote:

[Chris Barker - NOAA Federal]

I suspect that' because you added "./", or the current working dir.

with a bit of __file__ trickery you should be able to find the right path to add.

-Chris

--
 
---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Chris Barker - NOAA Federal]

On Mon, May 6, 2013 at 8:34 PM, Chris Barker - NOAA Federal <chris....@noaa.gov> wrote:

I suspect that' because you added "./", or the current working dir.

with a bit of __file__ trickery you should be able to find the right path to add.

oops, sorry, I wasn't thinking -- it looks like RPATH is set at build time, so you'd need to know where it was going to get installed -- that doesn't do much good (though it's possible that distutls is smart about this???) Even so, it wouldn't be re-locateable.

setting LD_LIBRARY_PATH at run time withe the __file__ trickery could work, though.

-Chris