[Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app

[Stuart Mentzer]

Hello,

I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows with PyInstaller. The frozen app fails at this new find_library call in __init__.py:

if not ctypes.util.find_library('hdf5dll.dll'):
raise ImportError('Could not load "hdf5dll.dll", please ensure' +
' that it can be found in the system path')

PyInstaller correctly places this DLL in the same directory as the application .exe where standard Windows DLL search logic will find it. Apparently the find_library doesn't do that in a frozen application. That is a big problem. I had to comment this code out to get a working frozen app.

That code was added in revision e9f6919.

This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both PyInstaller 1.5.1 and 2.0.

Should I file a bug report? Any easy work-around?

Thanks,
Stuart

[Antonio Valentino]

Hi Stuart,

Il 27/08/2012 17:43, Stuart Mentzer ha scritto:
> Hello,
>
> I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows with PyInstaller. The frozen app fails at this new find_library call in __init__.py:
>
> if not ctypes.util.find_library('hdf5dll.dll'):
> raise ImportError('Could not load "hdf5dll.dll", please ensure' +
> ' that it can be found in the system path')
>
> PyInstaller correctly places this DLL in the same directory as the application .exe where standard Windows DLL search logic will find it. Apparently the find_library doesn't do that in a frozen application. That is a big problem. I had to comment this code out to get a working frozen app.
>
> That code was added in revision e9f6919.
>

It is mainly a sanity check added under request of one of our users:
https://github.com/PyTables/PyTables/pull/146

> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both PyInstaller 1.5.1 and 2.0.
>
> Should I file a bug report? Any easy work-around?
>
> Thanks,
> Stuart
>

Yes please file a pull request with your patch.
It would be nice to preserve the sanity check in standard case so,
maybe, a good solution could be adding some check on sys.frozen or
something like that.

Thank you

--
Antonio Valentino

[Christoph Gohlke]

Hello,

As a workaround for frozen distributions, try to add the sys.executable
directory to os.environ['PATH'] before importing tables.

Ctypes only tries to find a library in the os.environ['PATH']
directories, not the current directory or the sys.executable directory
as one could expect.
http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48

As a workaround, for distributions that place the HDF5 and other DLLs in
the tables package directory, tables.__init__.py adds the tables package
directory to os.environ['PATH']. This also makes sure that the DLLs are
found when loading the hdf5Extension.pyd and other C extension modules
(another common problem). The use of __file__ to get the tables
directory should better be wrapped in a try..except statement.
https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24

Christoph

[Stuart Mentzer]

Antonio,

Thanks for the quick response.

To save time I did the report as a simple Issue (https://github.com/PyTables/PyTables/issues/177) showing the patch code that preserves and with a reference to #146. I hope that suffices.

Stuart

[Antonio Valentino]

Hi Christoph,

thank you very much for your hints.

Christoph, I suppose it can also be done at this point too:
https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24
Isn't it?

Please Stuart, can you try this fix as well.

> Ctypes only tries to find a library in the os.environ['PATH']
> directories, not the current directory or the sys.executable directory
> as one could expect.
> http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48
>
> As a workaround, for distributions that place the HDF5 and other DLLs in
> the tables package directory, tables.__init__.py adds the tables package
> directory to os.environ['PATH']. This also makes sure that the DLLs are
> found when loading the hdf5Extension.pyd and other C extension modules
> (another common problem). The use of __file__ to get the tables
> directory should better be wrapped in a try..except statement.
> https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24
>
> Christoph

Thanks, I'll fix it ASAP.

P.S.: please Christoph, do you have some hint for gh-175 [1]?
There is something we can do in PyTables?

[1] https://github.com/PyTables/PyTables/issues/175


--
Antonio Valentino

[Stuart Mentzer]

Hi Christoph,

Thanks for the info/suggestions. It might be nice to add your comments to the
Issue #177 I created.

I was aware that altering the PATH is a work-around. Patching PyTables is cleaner
and seems like the proper fix, and I think we agree that there is a problem here
that should be addressed. Maybe the PyTables test suite should even include a
frozen application test.

Thanks,
Stuart

[Stuart Mentzer]

Can't try it right now but I can't see why adding sys.excutable to the PATH at that point
wouldn't "work" as an alternative to skipping the sanity check when frozen. My broader
concern is whether fiddling with the PATH is the best or really the correct fix. If you
need to exactly replicate Windows DLL search rules (which, just to make life fun, vary
depending on the Windows version and the state of the SafeDllSearchMode*) then you should
do that rather than hacking the PATH. For example, what if there is another hdf5dll.dll
already on the PATH? In that case you'd better put sys.executable in the front of the PATH,
but that might break something else. Maybe 99.9% of the time the PATH hack works but
it can also fail.

I think the only proper fix is to make find_library follow the Windows rules exactly.
Short of that, it is not safe to use it when the library might be in different places,
such as frozen vs non-frozen applications.

[*] http://msdn.microsoft.com/en-us/library/windows/desktop/ms682586%28v=vs.85%29.aspx

>> Ctypes only tries to find a library in the os.environ['PATH']
>> directories, not the current directory or the sys.executable directory
>> as one could expect.
>> http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48
>>
>> As a workaround, for distributions that place the HDF5 and other DLLs in
>> the tables package directory, tables.__init__.py adds the tables package
>> directory to os.environ['PATH']. This also makes sure that the DLLs are
>> found when loading the hdf5Extension.pyd and other C extension modules
>> (another common problem). The use of __file__ to get the tables
>> directory should better be wrapped in a try..except statement.
>> https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24
>>
>> Christoph
> Thanks, I'll fix it ASAP.
>
> P.S.: please Christoph, do you have some hint for gh-175 [1]?
> There is something we can do in PyTables?
>
> [1] https://github.com/PyTables/PyTables/issues/175

Stuart

[Christoph Gohlke]

Hi Stuart,

I'll try to work on a patch tonight. It's probably better to use Ctypes
LoadLibrary instead of find_library because that makes sure all HDF5
dependencies are found, it (supposedly) searches the Windows DLL search
path not just os.environ['PATH'], and (supposedly) takes into account
libraries already loaded into the process.

Christoph

[Stuart Mentzer]

Great. I'll be happy to test a patch in the frozen context.

Stuart