Monkeypatching ctypes.util.find_library and distributing AVbin

["Juan J. Martínez"]

Hello,

I've been experimenting with the idea of distributing a local copy of
ABvin (both 32 and 64 bit) with my games in Linux, in the same way I do
with py2exe.

With py2exe I include the AVbin version I know works best with my game
(well, it is actually the latest alpha of version 11), and there's no
need for the user to install anything.

So I got libavbin.so.11 both 32 and 64 bit version and I put the in a
lib directory in my game, in this way:

$ tree lib
lib
|-- 32bit
| |-- COPYING.LESSER
| `-- libavbin.so.11
`-- 64bit
|-- COPYING.LESSER
`-- libavbin.so.11

And I monkeypatch ctypes.util.find_library, so pyglet will load my local
AVbin version instead of the one on the system (if any!).

def find_local_library(path, fn):
"""Load libraries for a local path"""
LOCAL = { "avbin": r"%s/lib/%s/libavbin.so.11" }
bits = "64bit" if sys.maxsize > 2**32 else "32bit"
if path in LOCAL:
return LOCAL[path] % (pyglet.resource.get_script_home(), bits)
return fn(path)

_find_library = ctypes.util.find_library
ctypes.util.find_library = lambda path: find_local_library(path,
_find_library)

Note the forward slashes for the path. This code is only used in Linux
so I didn't bother doing it portable. Mind if you use this idea in other
platforms!

I know it's a simple hack, but it works like a charm. I've used this
idea to include AVbin in my last two PyWeek games* when building
Debian/Ubuntu packages and it's great.

Potentially you can include any python-only dependency with your game
(ie. pyglet) and add to LOCAL dict any library you're distributing with
the game. Then it would be unzip and run!

This could be used in Mac too, but I don't have a Mac to test to any
comments would be welcome!

Regards,

Juan

* http://www.usebox.net/jjm/lunar/
* http://www.usebox.net/jjm/for-science/

--
jjm's home: http://www.usebox.net/jjm/
blackshell: http://blackshell.usebox.net/

[Paul Colomiets]

Hi Juan,

On Sat, Sep 28, 2013 at 1:56 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
> And I monkeypatch ctypes.util.find_library, so pyglet will load my local
> AVbin version instead of the one on the system (if any!).
>

I had this problem too on OS X too. But I ended up with setting
os.environ['DYLD_LIBRARY_PATH'] and renaming libavbin.10.dylib to
libavbin.dylib. I think you could set "LD_LIBRARY_PATH" on linux too.

I believe that it should be fixed in pyglet rather that relying
changing environment or monkey patching.

Thoughts?

--
Paul

["Juan J. Martínez"]

On 28/09/13 18:04, Paul Colomiets wrote:
> Hi Juan,
>

> On Sat, Sep 28, 2013 at 1:56 PM, "Juan J. Mart�nez" <j...@usebox.net> wrote:
>> And I monkeypatch ctypes.util.find_library, so pyglet will load my local
>> AVbin version instead of the one on the system (if any!).
>>
>
> I had this problem too on OS X too. But I ended up with setting
> os.environ['DYLD_LIBRARY_PATH'] and renaming libavbin.10.dylib to
> libavbin.dylib. I think you could set "LD_LIBRARY_PATH" on linux too.

IMHO pyglet does the right thing that is to rely on ctypes.utils, and
that one uses system tools such as ldconfig, gcc, objdump, etc.

I didn't like fiddling with LD_* because some Linux distributions may
not allow using them for security reasons (ie. Fedora with selinux
enabled), restricting what ldconfig will do.

BUT since we're not going to use the dynamic linker but ctypes'
LoadLibrary, I though monkeypatching was the cleaner solution to avoid
using the system library even if it exists and at the same time avoid
any security restriction.

(not 100% sure this was needed though)

> I believe that it should be fixed in pyglet rather that relying
> changing environment or monkey patching.
>
> Thoughts?

I'm not sure that there's anything "to fix". The libraries *should* be
installed on the system, but AVbin is not very well supported by Linux
distributions (Ubuntu Precise packages AVbin 7!).

I don't think there's a solution that covers all the cases (in my
example I was supporting both 32 and 64 bits, good luck doing that with
ldconfig hehe). This is perhaps "too advanced" for a general use case.

If I had to support this, I would use a pyglet.options entry with a
callable:

find_local_library
By default None. It is expected to be a callable with one parameter
`name` that will contain the name of the library to load. This callable
must return the path to the platform specific library or None if there's
no local library and ctypes.util.find_library must be used (default
behaviour).
Example:

def find_local_library(name):
import sys
if sys.platform.startswith('linux') and name == "avbin":
return "./lib/libavbin.so"
return None
pyglet.options['find_local_library'] = find_local_library

What do you think? May be you can propose something better :)

Regards,

Juan

[Nathan]

Sounds like a good technical solution would be to add a cross-platform standard way to customize Pyglet's library loading preferences.  It'd certainly simplify testing Libav (I currently install it and then use it via Pyglet to test it).

FYI, AVbin is licensed under the LGPL 2.1 (since Libav upstream is), so you should be aware of legal stuff if you're doing any distributing. http://libav.org/legal.html

~ Nathan

--
You received this message because you are subscribed to the Google Groups "pyglet-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyglet-users...@googlegroups.com.
To post to this group, send email to pyglet...@googlegroups.com.
Visit this group at http://groups.google.com/group/pyglet-users.
For more options, visit https://groups.google.com/groups/opt_out.

["Juan J. Martínez"]

On 28/09/13 18:47, Nathan wrote:
> [...]

> FYI, AVbin is licensed under the LGPL 2.1 (since Libav upstream is), so
> you should be aware of legal stuff if you're doing any
> distributing. http://libav.org/legal.html

No problem with that, I include the license and I'm not modifying the
library so I don't have changes to distribute.

Regards,

Juan

[Paul Colomiets]

Hi Juan,

On Sat, Sep 28, 2013 at 8:40 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
> I didn't like fiddling with LD_* because some Linux distributions may
> not allow using them for security reasons (ie. Fedora with selinux
> enabled), restricting what ldconfig will do.
>

AFAIK, they restrict if LD_* used by ldconfig. But not how we use it
with ctypes (dlopen). So:

directories.extend(os.environ['LD_LIBRARY_PATH'].split(':'))

.. still work (that's at
https://code.google.com/p/pyglet/source/browse/pyglet/lib.py#268)

> If I had to support this, I would use a pyglet.options entry with a callable:

I would say we should support loading libraries from project directory
out of the box, i.e. without any configuration.

The complexity is to find out what "project directory" is. It may be
directory where sys.executable is, in case of running py2exe or
cx_freeze, or it may be a directory of the main script, or a workdir.
Maybe just search all of the above and their "lib", "lib/x86",
"lib/x64" subdirectories. Probably we could also use some py2exe or
cx_freeze hooks.

[Nathan]

Sounds like you're covered.

~ Nathan

["Juan J. Martínez"]

On 28/09/13 19:03, Paul Colomiets wrote:
> Hi Juan,
>

> On Sat, Sep 28, 2013 at 8:40 PM, "Juan J. Mart�nez" <j...@usebox.net> wrote:
>> I didn't like fiddling with LD_* because some Linux distributions may
>> not allow using them for security reasons (ie. Fedora with selinux
>> enabled), restricting what ldconfig will do.
>>
>
> AFAIK, they restrict if LD_* used by ldconfig. But not how we use it
> with ctypes (dlopen). So:

I think you're right.

> directories.extend(os.environ['LD_LIBRARY_PATH'].split(':'))
>
> .. still work (that's at
> https://code.google.com/p/pyglet/source/browse/pyglet/lib.py#268)
>
>> If I had to support this, I would use a pyglet.options entry with a callable:
>
> I would say we should support loading libraries from project directory
> out of the box, i.e. without any configuration.
>
> The complexity is to find out what "project directory" is. It may be
> directory where sys.executable is, in case of running py2exe or
> cx_freeze, or it may be a directory of the main script, or a workdir.
> Maybe just search all of the above and their "lib", "lib/x86",
> "lib/x64" subdirectories. Probably we could also use some py2exe or
> cx_freeze hooks.
>

We can use pyglet.resource.get_script_home() as it will take into
account all the py2exe and cx_freeze cases. I like searching in current
script dir and lib directories (lib/x86 and lib/x64 doesn't make sense
because... ARM etc hehe).

Still like the idea of at least disabling this behaviour from
pyglet.options so I'll add it.

I can implement this. I'll submit a bug report with a patch so anybody
can review it before gets into the repo.

Thanks for your help Paul!

[Paul Colomiets]

Hi Juan,

On Sat, Sep 28, 2013 at 9:10 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
> We can use pyglet.resource.get_script_home() as it will take into
> account all the py2exe and cx_freeze cases.

Nice.

> I like searching in current
> script dir and lib directories (lib/x86 and lib/x64 doesn't make sense
> because... ARM etc hehe).
>

How about "lib/" + platform.machine() ?

--
Paul

["Juan J. Martínez"]

On 28/09/13 19:17, Paul Colomiets wrote:
> Hi Juan,
>

> On Sat, Sep 28, 2013 at 9:10 PM, "Juan J. Mart�nez" <j...@usebox.net> wrote:
>> We can use pyglet.resource.get_script_home() as it will take into
>> account all the py2exe and cx_freeze cases.
>
> Nice.
>
>> I like searching in current
>> script dir and lib directories (lib/x86 and lib/x64 doesn't make sense
>> because... ARM etc hehe).
>>
>
> How about "lib/" + platform.machine() ?
>

Fair enough. I can do that!

["Juan J. Martínez"]

On 28/09/13 19:19, "Juan J. Mart�nez" wrote:
> [...]

>>
>>> I like searching in current
>>> script dir and lib directories (lib/x86 and lib/x64 doesn't make sense
>>> because... ARM etc hehe).
>>>
>>
>> How about "lib/" + platform.machine() ?
>>
>
> Fair enough. I can do that!

Eh, not really... I get i686 here and that's not what we are looking for :D

See what I was doing for the "bits" in my original code:

bits = "64bit" if sys.maxsize > 2**32 else "32bit"

...but it is because I was assuming Intel/AMD only. I think we may have
to go with the "lib" version only. After all it doesn't make sense to
distribute a single package for Linux, Mac, Windows x n-architectures.

[Paul Colomiets]

Hi Juan,

On Sat, Sep 28, 2013 at 9:23 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
> On 28/09/13 19:19, "Juan J. Martínez" wrote:
>> [...]
>>>
>>>> I like searching in current
>>>> script dir and lib directories (lib/x86 and lib/x64 doesn't make sense
>>>> because... ARM etc hehe).
>>>>
>>>
>>> How about "lib/" + platform.machine() ?
>>>
>>
>> Fair enough. I can do that!
>
> Eh, not really... I get i686 here and that's not what we are looking for :D
>

I wouldn't care if the would be called i686/x86_64. However there are
rumors that it seems to be non-reliable on OS X.

> See what I was doing for the "bits" in my original code:
>
> bits = "64bit" if sys.maxsize > 2**32 else "32bit"
>
> ...but it is because I was assuming Intel/AMD only. I think we may have
> to go with the "lib" version only. After all it doesn't make sense to
> distribute a single package for Linux, Mac, Windows x n-architectures.
>

Mac usually uses universal binaries. For Windows I doubt I will build
64bit version soon, because that needs VS license. So only system
affected is linux, outlined in your use case. I don't distribute
pyglet apps for Linux (yet), so I don't care much. (For development
system avbin is just OK).

--
Paul

["Juan J. Martínez"]

On 28/09/13 19:03, Paul Colomiets wrote:

> [...]

> AFAIK, they restrict if LD_* used by ldconfig. But not how we use it
> with ctypes (dlopen). So:
>
> directories.extend(os.environ['LD_LIBRARY_PATH'].split(':'))
>
> .. still work (that's at
> https://code.google.com/p/pyglet/source/browse/pyglet/lib.py#268)
>

Ah, that's going to be harder than I though. That code is only used in
Linux as a fallback mechanism and the default ctypes.util.find_library
won't check LD_LIBRARY_PATH (at least not in my system).

I'm certain it will work in Mac, but it won't work in Linux if library
is installed in the system.

[Paul Colomiets]

Hi Juan,

On Sat, Sep 28, 2013 at 9:57 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
> On 28/09/13 19:03, Paul Colomiets wrote:
>> [...]
>> AFAIK, they restrict if LD_* used by ldconfig. But not how we use it
>> with ctypes (dlopen). So:
>>
>> directories.extend(os.environ['LD_LIBRARY_PATH'].split(':'))
>>
>> .. still work (that's at
>> https://code.google.com/p/pyglet/source/browse/pyglet/lib.py#268)
>>
>
> Ah, that's going to be harder than I though. That code is only used in
> Linux as a fallback mechanism and the default ctypes.util.find_library
> won't check LD_LIBRARY_PATH (at least not in my system).
>
> I'm certain it will work in Mac, but it won't work in Linux if library
> is installed in the system.
>

AFAICS, The whole point of the find_library is to find system library
and find out the exact version of it (e.g. libm.so.6). Given we don't
want to have multiple versions and given that the whole intention is
to *override* system libraries, I think it's safe to just use CDLL(dir
+ 'libname.so') before trying to find_library.

--
Paul

["Juan J. Martínez"]

On 28/09/13 20:14, Paul Colomiets wrote:
> [...]

>
> AFAICS, The whole point of the find_library is to find system library
> and find out the exact version of it (e.g. libm.so.6). Given we don't
> want to have multiple versions and given that the whole intention is
> to *override* system libraries, I think it's safe to just use CDLL(dir
> + 'libname.so') before trying to find_library.

You're right but I think I've done it slightly better.

Please see the following issue (has the patch):

http://code.google.com/p/pyglet/issues/detail?id=671

It's explained there. Let me know if I've done anything stupid but this
should allow us to load libraries from the same directory the script is
run *or* a subdirectory called "lib".

[Paul Colomiets]

Hi Juan,

On Sat, Sep 28, 2013 at 10:40 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
> Please see the following issue (has the patch):
>
> http://code.google.com/p/pyglet/issues/detail?id=671
>
> It's explained there. Let me know if I've done anything stupid but this
> should allow us to load libraries from the same directory the script is
> run *or* a subdirectory called "lib".
>

The patch is probably ok. There are two small issues though:

1. The original code style:

a) `".so" in name` may catch ".sock" which is a bad thing, I'd use
`".so." in name or name.endswith(".so")` or a regexp

b) similarly regexp "lib(.*)\.so" matches "librarian.sock", should
probably use "lib(.*)\.so(?:$|\.)"

c) keeping same dictionary for cache by library name and by file
name seems ugly

2. Libraries that has dependencies on other libraries can use
dependencies from system paths. So LD_LIBRARY_PATH should probably be
populated anyway.

Even if #2 won't work with selinux, it will work for most cases I
think. Does Fedora use selinux by default on the desktop? Selinux
problem can probably be fixed by preloading all the dependencies with
CDLL using RTLD_GLOBAL flag, but I'm not sure that it works and how to
do that automatically.

However, for me the patch would be helpful as is.

--
Paul

["Juan J. Martínez"]

On 28/09/13 21:19, Paul Colomiets wrote:
> Hi Juan,
>

> On Sat, Sep 28, 2013 at 10:40 PM, "Juan J. Mart�nez" <j...@usebox.net> wrote:
>> Please see the following issue (has the patch):
>>
>> http://code.google.com/p/pyglet/issues/detail?id=671
>>
>> It's explained there. Let me know if I've done anything stupid but this
>> should allow us to load libraries from the same directory the script is
>> run *or* a subdirectory called "lib".
>>
>
> The patch is probably ok. There are two small issues though:
>
> 1. The original code style:
>
> a) `".so" in name` may catch ".sock" which is a bad thing, I'd use
> `".so." in name or name.endswith(".so")` or a regexp
>
> b) similarly regexp "lib(.*)\.so" matches "librarian.sock", should
> probably use "lib(.*)\.so(?:$|\.)"
>
> c) keeping same dictionary for cache by library name and by file
> name seems ugly

OK, I can fix a and b. c is not a big deal, but I'll look at it too.

> 2. Libraries that has dependencies on other libraries can use
> dependencies from system paths. So LD_LIBRARY_PATH should probably be
> populated anyway.
>
> Even if #2 won't work with selinux, it will work for most cases I
> think. Does Fedora use selinux by default on the desktop? Selinux
> problem can probably be fixed by preloading all the dependencies with
> CDLL using RTLD_GLOBAL flag, but I'm not sure that it works and how to
> do that automatically.

selinux is enabled by default in Fedora (even in desktop), but I don't
think that's the problem.

The problem is that in Linux ctypes.util.find_library won't check
LD_LIBRARY_PATH at all and ldconfig is ignoring it also (AFAIK is
supposed to be a bad idea, the trend is to not support it anymore).

I'll fix the regex and I think the patch is a good and will enable us to
do what we want without changing the code too much.

Thanks again for your help.

[Paul Colomiets]

Hi Juan,

On Sat, Sep 28, 2013 at 11:57 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
> On 28/09/13 21:19, Paul Colomiets wrote:
> The problem is that in Linux ctypes.util.find_library won't check
> LD_LIBRARY_PATH at all and ldconfig is ignoring it also (AFAIK is
> supposed to be a bad idea, the trend is to not support it anymore).

I'll try to explain it once more. Let's imagine you want to include
libpulse into your app. Even if you load "lib/libpulse.so", there are
the following system libraries loaded:
ldd libpulse.so
0:12
linux-vdso.so.1 (0x00007fff16e55000)
libjson-c.so.2 => /usr/lib/libjson-c.so.2 (0x00007f3426770000)
libpulsecommon-4.0.so =>
/usr/lib/pulseaudio/libpulsecommon-4.0.so (0x00007f3426503000)
libdbus-1.so.3 => /usr/lib/libdbus-1.so.3 (0x00007f34262bc000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f342609d000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f3425d9a000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f34259ef000)
libxcb.so.1 => /usr/lib/libxcb.so.1 (0x00007f34257cf000)
libsndfile.so.1 => /usr/lib/libsndfile.so.1 (0x00007f3425569000)
libasyncns.so.0 => /usr/lib/libasyncns.so.0 (0x00007f3425363000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f342515a000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007f3426bf4000)
libXau.so.6 => /usr/lib/libXau.so.6 (0x00007f3424f56000)
libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0x00007f3424d50000)
libFLAC.so.8 => /usr/lib/libFLAC.so.8 (0x00007f3424b1d000)
libvorbisenc.so.2 => /usr/lib/libvorbisenc.so.2 (0x00007f342464e000)
libvorbis.so.0 => /usr/lib/libvorbis.so.0 (0x00007f3424421000)
libogg.so.0 => /usr/lib/libogg.so.0 (0x00007f3424219000)
libnsl.so.1 => /usr/lib/libnsl.so.1 (0x00007f3424001000)
libresolv.so.2 => /usr/lib/libresolv.so.2 (0x00007f3423dea000)

I'm not sure embedding libpulse is a good idea at all, but there might
be other libraries that matter. (openal and avbin seem to modest on
dependencies, though)

--
Paul

["Juan J. Martínez"]

On 28/09/13 22:15, Paul Colomiets wrote:
> [...]

>
> I'm not sure embedding libpulse is a good idea at all, but there might
> be other libraries that matter. (openal and avbin seem to modest on
> dependencies, though)

I understand, but LD_LIBRARY_PATH doesn't work :)

AVbin requires libz.so.1, and it must be installed on the system (very
likely).

The point of loading local libraries is just for almost completely
self-contained libraries.

["Juan J. Martínez"]

On 28/09/13 22:15, Paul Colomiets wrote:
> [...]
>

> I'm not sure embedding libpulse is a good idea at all, but there might
> be other libraries that matter. (openal and avbin seem to modest on
> dependencies, though)
>

Unless I've made a mistake, this should do it:

http://code.google.com/p/pyglet/source/detail?r=da26d7e1c3e7

I've used the regex when looking for libs in Linux and I've included the
local path in LD_LIBRARY_PATH in Mac, although I don't know if it will
make any difference.

In Linux anything that is not in the local path will be handled by
ctypes, so it should be all right if the libs are installed in the system.

Please give it a go and let me know if there's any problem.

[Paul Colomiets]

Hi Juan,

On Sun, Sep 29, 2013 at 12:47 AM, "Juan J. Martínez" <j...@usebox.net> wrote:
> On 28/09/13 22:15, Paul Colomiets wrote:
>> [...]
>>
>> I'm not sure embedding libpulse is a good idea at all, but there might
>> be other libraries that matter. (openal and avbin seem to modest on
>> dependencies, though)
>>
>
> Unless I've made a mistake, this should do it:
>
> http://code.google.com/p/pyglet/source/detail?r=da26d7e1c3e7
>
> I've used the regex when looking for libs in Linux and I've included the
> local path in LD_LIBRARY_PATH in Mac, although I don't know if it will
> make any difference.
>
> In Linux anything that is not in the local path will be handled by
> ctypes, so it should be all right if the libs are installed in the system.
>
> Please give it a go and let me know if there's any problem.
>

Looks ok. May test on OS X on weekdays.

--
Paul

[Nathan]

On Sat, Sep 28, 2013 at 3:30 PM, "Juan J. Martínez" <j...@usebox.net> wrote:

AVbin requires libz.so.1, and it must be installed on the system (very
likely).

Not as of the master branch after 25 Nov 2012.  At that point I made sure libz was being statically included on all OS's.  (Before that it was dynamic in some and static in others).  Of course, I haven't made an official release after that date...

~ Nathan 

["Juan J. Martínez"]

On 28/09/13 23:25, Nathan wrote:

> On Sat, Sep 28, 2013 at 3:30 PM, "Juan J. Mart�nez" <j...@usebox.net
> <mailto:j...@usebox.net>> wrote:
>
> AVbin requires libz.so.1, and it must be installed on the system (very
> likely).
>
>
> Not as of the master branch after 25 Nov 2012. At that point I made
> sure libz was being statically included on all OS's. (Before that it
> was dynamic in some and static in others). Of course, I haven't made an
> official release after that date...

OK :) Good to know! In fact I realized when I was building packages for
Debian... the Debian tools added the dependency for me ;)

I'm using AVbin 11, but I don't really remember which alpha version I'm
using!

[Nathan]

Linux 64-bit was the last OS that was using libz dynamically, so that makes sense.  :)

I don't think I ever made an alpha release after that point.

~ Nathan

On Sat, Sep 28, 2013 at 4:28 PM, "Juan J. Martínez" <j...@usebox.net> wrote:

On 28/09/13 23:25, Nathan wrote:

> On Sat, Sep 28, 2013 at 3:30 PM, "Juan J. Martínez" <j...@usebox.net

> <mailto:j...@usebox.net>> wrote:
>
>     AVbin requires libz.so.1, and it must be installed on the system (very
>     likely).
>
>
> Not as of the master branch after 25 Nov 2012.  At that point I made
> sure libz was being statically included on all OS's.  (Before that it
> was dynamic in some and static in others).  Of course, I haven't made an
> official release after that date...

OK :) Good to know! In fact I realized when I was building packages for
Debian... the Debian tools added the dependency for me ;)

I'm  using AVbin 11, but I don't really remember which alpha version I'm
using!

Regards,

Juan

--
jjm's home: http://www.usebox.net/jjm/
blackshell: http://blackshell.usebox.net/

[Paul Colomiets]

Hi Juan,

On Sun, Sep 29, 2013 at 12:47 AM, "Juan J. Martínez" <j...@usebox.net> wrote:
> Please give it a go and let me know if there's any problem.
>

It seems script_home is wrong with cx_Freeze for OS X:

{'LD_LIBRARY_PATH': ':lib', 'script_home': '', 'workdir':
'/Users/pc/dev/skyward', '__file__':
'/Users/pc/dev/skyward/build/Skyward_LM.app/Contents/MacOS/library.zip/house/main.pyc'}

With the above I would like LD_LIBRARY_PATH be:

/Users/pc/dev/skyward/build/Skyward_LM.app/Contents/MacOS

--
Paul

["Juan J. Martínez"]

On 30/09/13 10:36, Paul Colomiets wrote:
> Hi Juan,
>

> On Sun, Sep 29, 2013 at 12:47 AM, "Juan J. Mart�nez" <j...@usebox.net> wrote:
>> Please give it a go and let me know if there's any problem.
>>
>
> It seems script_home is wrong with cx_Freeze for OS X:
>
> {'LD_LIBRARY_PATH': ':lib', 'script_home': '', 'workdir':
> '/Users/pc/dev/skyward', '__file__':
> '/Users/pc/dev/skyward/build/Skyward_LM.app/Contents/MacOS/library.zip/house/main.pyc'}
>
> With the above I would like LD_LIBRARY_PATH be:
>
> /Users/pc/dev/skyward/build/Skyward_LM.app/Contents/MacOS
>

I'll take a look when I'm home, but your help with Mac part would be
very appreciated.

The code is:

http://code.google.com/p/pyglet/source/browse/pyglet/resource.py#105

script_home may need some love, the comments don't refer to cx_Freeze;
or do you think the local lib patch is wrong?

Regards,

Juan

[Paul Colomiets]

Hi Juan,

I'm not sure. I would do a patch if I know. If we change script_home
it may influence some users. But since it's plain wrong, maybe it
doesn't matter? Also looking at the code it seems it should return
"Contents/Resources" dir for some reason (is it how py2app works?) but
I need it return "Contents/MacOS" since it's where cx_Freeze puts
everything.

--
Paul

["Juan J. Martínez"]

On 30/09/13 10:49, Paul Colomiets wrote:
> Hi Juan,
>

Well, looks like we need to add support for cx_Freeze if the default of
returning '' doesn't work.

See this:

http://cx-freeze.readthedocs.org/en/latest/faq.html#using-data-files

What is the value of getattr(sys, 'frozen', None)?

Looks like using os.path.dirname(sys.executable) is the right thing to
do for cx_Freeze.

Does os.environ['RESOURCEPATH'] exist?

script_home is really useful, I'd like it to work properly in most cases.

Regards,

Juan

[Paul Colomiets]

Hi Juan,

On Mon, Sep 30, 2013 at 12:57 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
> On 30/09/13 10:49, Paul Colomiets wrote:
>> Hi Juan,
>>

>> On Mon, Sep 30, 2013 at 12:42 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
>>> On 30/09/13 10:36, Paul Colomiets wrote:
>>> I'll take a look when I'm home, but your help with Mac part would be
>>> very appreciated.
>>>
>>> The code is:
>>>
>>> http://code.google.com/p/pyglet/source/browse/pyglet/resource.py#105
>>>
>>> script_home may need some love, the comments don't refer to cx_Freeze;
>>> or do you think the local lib patch is wrong?
>>
>> I'm not sure. I would do a patch if I know. If we change script_home
>> it may influence some users. But since it's plain wrong, maybe it
>> doesn't matter? Also looking at the code it seems it should return
>> "Contents/Resources" dir for some reason (is it how py2app works?) but
>> I need it return "Contents/MacOS" since it's where cx_Freeze puts
>> everything.
>>
>
> Well, looks like we need to add support for cx_Freeze if the default of
> returning '' doesn't work.
>
> See this:
>
> http://cx-freeze.readthedocs.org/en/latest/faq.html#using-data-files
>
> What is the value of getattr(sys, 'frozen', None)?
>
> Looks like using os.path.dirname(sys.executable) is the right thing to
> do for cx_Freeze.
>
> Does os.environ['RESOURCEPATH'] exist?
>

Will check. (I'm usually work in Linux, so rebooting to OS X takes time)

> script_home is really useful, I'd like it to work properly in most cases.
>

Where should it point to? According to the docs py2app points to
"Contents/Resources" dir (does py2app puts *.py files there?). But for
cx_Freeze it's useless. But it would be strange that py2app and
cx_Freeze use different dirs on the same platform. Also according to
the docs:

Get the directory containing the program entry module.

And this "dir" is "Contents/MacOS/library.zip". Is it consistent to
strip the "library.zip" prefix? I.e. this is *script* home not a
*data* home, right?

--
Paul

["Juan J. Martínez"]

On 30/09/13 11:13, Paul Colomiets wrote:
> [...]

> Will check. (I'm usually work in Linux, so rebooting to OS X takes time)

OK, thanks.

>> script_home is really useful, I'd like it to work properly in most cases.
>>
>
> Where should it point to? According to the docs py2app points to
> "Contents/Resources" dir (does py2app puts *.py files there?). But for
> cx_Freeze it's useless. But it would be strange that py2app and
> cx_Freeze use different dirs on the same platform. Also according to
> the docs:
>
> Get the directory containing the program entry module.
>
> And this "dir" is "Contents/MacOS/library.zip". Is it consistent to
> strip the "library.zip" prefix? I.e. this is *script* home not a
> *data* home, right?

Well I'd expect, whatever the system is, to have a binary (with all
packaged inside) that will be probably unpacked somewhere. script_home
should return the directory where that binary is (not the unpacked
stuff!), and then you'll have there a "lib" directory with the libraries.

When I'm back home today I'll play a little bit with cx_Freeze because I
assume the behavioue should be consistent in all the supported platforms.

So what I want is:

- detect if we're running a cx_Freeze app
- find where the binary is run

With that in place I think we'll get to the local libs with no problem.

Regards,

Juan

[Paul Colomiets]

Hi Juan,

On Mon, Sep 30, 2013 at 1:19 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
> Well I'd expect, whatever the system is, to have a binary (with all
> packaged inside) that will be probably unpacked somewhere. script_home
> should return the directory where that binary is (not the unpacked
> stuff!), and then you'll have there a "lib" directory with the libraries.
>

Ah, ok, so the FAQ entry you've mentioned has correct solution.

> When I'm back home today I'll play a little bit with cx_Freeze because I
> assume the behavioue should be consistent in all the supported platforms.
>

Should also check consistency with py2app (my app has not worked with
py2app for some reason, I don't remember exactly, still there are many
users of it).

Could you also check AVbin issue? (just apply the patch and play the movie)

--
Paul

["Juan J. Martínez"]

On 30/09/13 11:42, Paul Colomiets wrote:
> Hi Juan,
>

> On Mon, Sep 30, 2013 at 1:19 PM, "Juan J. Mart�nez" <j...@usebox.net> wrote:
>> Well I'd expect, whatever the system is, to have a binary (with all
>> packaged inside) that will be probably unpacked somewhere. script_home
>> should return the directory where that binary is (not the unpacked
>> stuff!), and then you'll have there a "lib" directory with the libraries.
>>
>
> Ah, ok, so the FAQ entry you've mentioned has correct solution.
>
>> When I'm back home today I'll play a little bit with cx_Freeze because I
>> assume the behavioue should be consistent in all the supported platforms.
>>
>
> Should also check consistency with py2app (my app has not worked with
> py2app for some reason, I don't remember exactly, still there are many
> users of it).

Yes, that would be great.

I think our local lib patch is OK, is just that we need to improve
script_home.

Can you open an issue about this including our findings so far? Also
attach there any patch you try!

> Could you also check AVbin issue? (just apply the patch and play the movie)
>

Yes, I'll do when I get back home (stupid job that pays my bills!) ;)

Regards,

Juan

["Juan J. Martínez"]

On 30/09/13 11:46, "Juan J. Mart�nez" wrote:
> [...]
>>

>> Should also check consistency with py2app (my app has not worked with
>> py2app for some reason, I don't remember exactly, still there are many
>> users of it).
>
> Yes, that would be great.
>
> I think our local lib patch is OK, is just that we need to improve
> script_home.

OK, I've tested this with cx_Freeze in Linux and it is... confusing:
sys.forzen is just True.

I've check py2exe, py2app and cx_Freeze source code and these are my
findings:

- sys.freeze can be windows_exe or console_exe in py2exe
- sys.freeze can be macosx_app in py2app; I've checked the os.environ
thing and seems OK!
- sys.freeze is True in cx_Freeze and __main__.__file__ doesn't exist

So to properly support cx_Freeze we should add an alternative using the
documented method: os.path.dirname(sys.executable).

It works perfect in Linux, so I'm going to assume it will be the same in
Mac.

Try it and let me know if it works in Mac:

http://code.google.com/p/pyglet/source/detail?r=f697f60235f30e9662d7f07195524e43e7a06520

>> Could you also check AVbin issue? (just apply the patch and play the movie)
>>

The mp3 plays fine with and without the patch, I can't tell if there's
an improvement.

The video shows a glitch at the beginning and the patch fixes it. Well done!

[Paul Colomiets]

Hi Juan,

On Mon, Sep 30, 2013 at 7:50 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
> Try it and let me know if it works in Mac:
>

Here is what search paths I get:

SEARCH PATH ['/usr/local/lib/libavbin.dylib',
'/Users/pc/lib/libavbin.dylib', '/usr/local/lib/libavbin.dylib',
'/usr/lib/libavbin.dylib']
SEARCH PATH ['/Users/pc/dev/skyward/build/Skyward_LM.app/Contents/MacOS/avbin',
'/Users/pc/dev/skyward/build/Skyward_LM.app/Contents/MacOS/lib/avbin',
'avbin', '/Users/pc/lib/avbin', '/usr/local/lib/avbin',
'/usr/lib/avbin']

The bug is that paths starting with slash are searched in
"DYLD_LIBRARY_PATH" and relative paths are searched in
"LD_LIBRARY_PATH". Not sure what specific fix to apply.

Also is it intended that bare name, without lib*.dylib is searched
for? I believe it's same crappy problem of using same function call
for name and libname.dylib that is wrong. In fact there are dynamic
libraries in OS X without extension but they are loaded with
load_framework.

--
Paul

["Juan J. Martínez"]

On 01/10/13 10:23, Paul Colomiets wrote:
> Hi Juan,
>

> On Mon, Sep 30, 2013 at 7:50 PM, "Juan J. Mart�nez" <j...@usebox.net> wrote:
>> Try it and let me know if it works in Mac:
>>
>
> Here is what search paths I get:
>
> SEARCH PATH ['/usr/local/lib/libavbin.dylib',
> '/Users/pc/lib/libavbin.dylib', '/usr/local/lib/libavbin.dylib',
> '/usr/lib/libavbin.dylib']
> SEARCH PATH ['/Users/pc/dev/skyward/build/Skyward_LM.app/Contents/MacOS/avbin',
> '/Users/pc/dev/skyward/build/Skyward_LM.app/Contents/MacOS/lib/avbin',
> 'avbin', '/Users/pc/lib/avbin', '/usr/local/lib/avbin',
> '/usr/lib/avbin']
>
> The bug is that paths starting with slash are searched in
> "DYLD_LIBRARY_PATH" and relative paths are searched in
> "LD_LIBRARY_PATH". Not sure what specific fix to apply.

Uff, I'm not sure how it works. I added it to LD_LIBRARY_PATH because I
thought that was the right place. I'll check the code this evening, may
be I missed something.

See the patch:

http://code.google.com/p/pyglet/source/detail?r=da26d7e1c3e71dfcc415e3cdec85a4f9a3add387

Should we add the same in DYLD_LIBRARY_PATH part?

Try the attached patch (you may need to change it, it was a quick shot
in the dark!).

> Also is it intended that bare name, without lib*.dylib is searched
> for? I believe it's same crappy problem of using same function call
> for name and libname.dylib that is wrong. In fact there are dynamic
> libraries in OS X without extension but they are loaded with
> load_framework.

ctypes uses the bare name (you just look for "avbin").

I've seen the load_framework part but I don't really know how it works.

I'm going to need your help here because I'm changing thing blindly. The
Linux part it's OK, Windows still OK, only Mac needs tweaking.

Regards,

Juan

[Paul Colomiets]

Hi Juan,

On Tue, Oct 1, 2013 at 12:37 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
> On 01/10/13 10:23, Paul Colomiets wrote:
>> Hi Juan,
>>

>> On Mon, Sep 30, 2013 at 7:50 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
>>> Try it and let me know if it works in Mac:
>>>
>>
>> Here is what search paths I get:
>>
>> SEARCH PATH ['/usr/local/lib/libavbin.dylib',
>> '/Users/pc/lib/libavbin.dylib', '/usr/local/lib/libavbin.dylib',
>> '/usr/lib/libavbin.dylib']
>> SEARCH PATH ['/Users/pc/dev/skyward/build/Skyward_LM.app/Contents/MacOS/avbin',
>> '/Users/pc/dev/skyward/build/Skyward_LM.app/Contents/MacOS/lib/avbin',
>> 'avbin', '/Users/pc/lib/avbin', '/usr/local/lib/avbin',
>> '/usr/lib/avbin']
>>
>> The bug is that paths starting with slash are searched in
>> "DYLD_LIBRARY_PATH" and relative paths are searched in
>> "LD_LIBRARY_PATH". Not sure what specific fix to apply.
>
> Uff, I'm not sure how it works. I added it to LD_LIBRARY_PATH because I
> thought that was the right place. I'll check the code this evening, may
> be I missed something.
>
> See the patch:
>
> http://code.google.com/p/pyglet/source/detail?r=da26d7e1c3e71dfcc415e3cdec85a4f9a3add387
>
> Should we add the same in DYLD_LIBRARY_PATH part?
>

According to the docs, LD_LIBRARY_PATH is not used when library name
contains at least one directory entry. So the actual question is
probably why we don't add "lib*.dylib" for searching?

> Try the attached patch (you may need to change it, it was a quick shot
> in the dark!).
>
>> Also is it intended that bare name, without lib*.dylib is searched
>> for? I believe it's same crappy problem of using same function call
>> for name and libname.dylib that is wrong. In fact there are dynamic
>> libraries in OS X without extension but they are loaded with
>> load_framework.
>
> ctypes uses the bare name (you just look for "avbin").
>

Ah. Ok. So what's the point of "darwin='/usr/local/lib/avbin.dylib'"
isn't it easier to add /usr/local/lib dir to search path? Actually in
docs it's in DYLD_FALLBACK_LIBRARY_PATH, so should be searched for
anyway.

> I've seen the load_framework part but I don't really know how it works.
>

For me it just works, so no need to change it.

> I'm going to need your help here because I'm changing thing blindly. The
> Linux part it's OK, Windows still OK, only Mac needs tweaking.
>

Sure. I'm just trying to understand how it should work before patching anything.

--
Paul

[Paul Colomiets]

Hi Juan,

On Tue, Oct 1, 2013 at 12:37 PM, "Juan J. Martínez" <j...@usebox.net> wrote:
> I'm going to need your help here because I'm changing thing blindly. The
> Linux part it's OK, Windows still OK, only Mac needs tweaking.
>

The patch attached. It satisfies me, not sure it works for anyone.
I've removed "frozen" flag checking from find_library since I don't
think it's needed if we look in script_home.


--
Paul

["Juan J. Martínez"]

On 02/10/13 10:37, Paul Colomiets wrote:
> Hi Juan,
>

> On Tue, Oct 1, 2013 at 12:37 PM, "Juan J. Mart�nez" <j...@usebox.net> wrote:
>> I'm going to need your help here because I'm changing thing blindly. The
>> Linux part it's OK, Windows still OK, only Mac needs tweaking.
>>
>
> The patch attached. It satisfies me, not sure it works for anyone.
> I've removed "frozen" flag checking from find_library since I don't
> think it's needed if we look in script_home.
>
>

OK, thanks. I'll review it this evening!

Regards,

Juan

["Juan J. Martínez"]

On 02/10/13 10:37, Paul Colomiets wrote:

> Hi Juan,

>
> On Tue, Oct 1, 2013 at 12:37 PM, "Juan J. Mart�nez" <j...@usebox.net> wrote:
>> I'm going to need your help here because I'm changing thing blindly. The
>> Linux part it's OK, Windows still OK, only Mac needs tweaking.
>>
>
> The patch attached. It satisfies me, not sure it works for anyone.
> I've removed "frozen" flag checking from find_library since I don't
> think it's needed if we look in script_home.
>
>

OK, looks good.

According to Apple's docs [1], our patch adding the local lib paths to
LD_LIBRARY_PATH does the trick when loading the library, and if it
fails, then the regular find_library code kicks in.

Thanks for the patch!

Regards,

Juan


[1]:
https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/DynamicLibraryUsageGuidelines.html

["Juan J. Martínez"]

On 02/10/13 18:07, "Juan J. Mart�nez" wrote:
> [...]

>> The patch attached. It satisfies me, not sure it works for anyone.
>> I've removed "frozen" flag checking from find_library since I don't
>> think it's needed if we look in script_home.
>>
>>
>
> OK, looks good.
>

I finally applied the patch with some minor changes. Basically I don't
want to drop py2app and pyinstaller support in case it works (I can't
test it really). Because it just add paths to the search path list I
think the best option is to leave it there.

Check out the repo code and give it a go although it shouldn't be any
difference.

Regards,

Juan