Binary hd5 on mac os x: how to install compatibly with h5py

[Lewis Levin]

Just trying to get h5py to install with binary distribution of hdf5.

On one machine brew successfully compiled and built and installed the hdf5 library with brew install /home/brew/hdf5.  Then pip install h5py worked.

On another Mac (both running 10.9.1 and python 2.7.6), brew utterly failed with some kind of recursive compile error.  So, I am not going to debug brew formulas or the make scripts of hdf5.

I decided to try to use the binary libraries for hdf5.  (Yes, that means I am an incompetent noob.  I want to USE h5py, not do the sysadmin or development of it.)

I downloaded:  hdf5-1.8.7-mac-intel-x86_64-shared.tar

Untarred it and copied all of the directories intact under usr/local/hdf5

I added this line to .bash_profile:  export HDF5_DIR=/usr/local/hdf5

I tried both pip install h5py and easy_install h5py.  Each reported completing the installation (though with hundreds of warnings).  pip list reports that h5py (2.2.1) is installed in both cases.

Then, I tried to import h5py and run the tests within Python.   for both easy_install and pip attempts, I get the following error trace within Python immediately after import h5py.  So something is missing, but it is beyond me to understand the dozens of files in h5py and its dependencies to know what is missing, why, where to find it, and what install process to follow.

As mentioned, here is the traceback from the easy_install'ed version (note reference to egg):

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "build/bdist.macosx-10.6-intel/egg/h5py/__init__.py", line 10, in <module>

  File "build/bdist.macosx-10.6-intel/egg/h5py/_errors.py", line 7, in <module>

  File "build/bdist.macosx-10.6-intel/egg/h5py/_errors.py", line 6, in __bootstrap__

ImportError: dlopen(/Users/lewislevinmbr/.python-eggs/h5py-2.2.1-py2.7-macosx-10.6-intel.egg-tmp/h5py/_errors.so, 2): Library not loaded: /mnt/scr1/release-binary/hdf5/v187/builds/fred/lib/libhdf5.7.dylib

  Referenced from: /Users/lewislevinmbr/.python-eggs/h5py-2.2.1-py2.7-macosx-10.6-intel.egg-tmp/h5py/_errors.so

  Reason: image not found

Here is the traceback from the pip'ed version:

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/__init__.py", line 10, in <module>

    from h5py import _errors

ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/_errors.so, 2): Library not loaded: /mnt/scr1/release-binary/hdf5/v187/builds/fred/lib/libhdf5.7.dylib

  Referenced from: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/_errors.so

  Reason: image not found

Something is supposed to be included with the site-packages install of h5py.  This seems to be a problem with the pip and easy_install script (setup.py) and probably has nothing to do with hdf5.   But, I noted the the other information because on the machine on which brew built hdf5, pip install h5py worked.

I'd be happy using brew for this but it seems to be highly fragile black magic.  Not asking anyone here to help with brew.

Just trying to get h5py to install with binary distribution of hdf5.

What seems clear is that h5py is looking in an utterly crazy place for the core dylib.   It turns out that libhd5f.7.dylib is right where it should be at:

/usr/local/hdf5/lib

Pretty much where you'd expect it to be.  So, why is h5py looking in a non-existent place?  There is no /mnt/scr1...     This seems like some holdover from a build script that mounted a drive to install something.

Any thoughts?

Thanks,

Lewis

[Lewis Levin]

I tried the install from git (which is a really nice way to go).

Basically the same problem as this output from a unittest shows (all the unit tests naturally fail in the same way as the basic library for hdf5 cannot be found):

ImportError: dlopen(/Users/lewislevinmbr/Dropbox/Code Mac/Clean Mac Python Install/h5py/build/lib.macosx-10.6-intel-2.7/h5py/_errors.so, 2): Library not loaded: /mnt/scr1/release-binary/hdf5/v187/builds/fred/lib/libhdf5.7.dylib

So, how do I get h5py to look in a sensible place?  Where on earth is it getting this crazy path (unlikely to exist on anyone's system)?

[Lewis Levin]

So, further information.  I got the brew install of hdf5 libraries (version 1.8.12) to work.

But, pip install h5py still looks to insane non-existent directory for core hdf5 files:

import h5py

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/__init__.py", line 10, in <module>

    from h5py import _errors

ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/_errors.so, 2): Library not loaded: /mnt/scr1/release-binary/hdf5/v187/builds/fred/lib/libhdf5.7.dylib

  Referenced from: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/_errors.so

  Reason: image not found

I am tired.  I am giving up.

[Andrew Collette]

Hi Lewis,

> So, further information. I got the brew install of hdf5 libraries (version
> 1.8.12) to work.

I personally use the Homebrew python and libhdf5 for my work (my main
development machine is OS 10.9). You've posted a lot of information,
more than I can parse at the moment, but here's what worked for me:

1. Use the Homebrew Python (not the system Python)
2. Install libhdf5 via homebrew (you'll have to `brew tap
homebrew/science` first.)
3. Install via pip or via the h5py tarball.

Check to make sure no environment variables are set ($HDF5_DIR in
particular) and clang should find your Homebrew install of libhdf5.
Unless $HDF5_DIR is set to something crazy I don't know where that
/mnt/scr1 path could have come from; it's certainly not coded into
h5py anywhere.

Other solutions, including binary installs of HDF5 or installations
involving the system Python, are likely to have mixed results, as
you've found. Unfortunately on the Mac there are a million ways
things could go wrong, so I would try to stick to the above steps if
at all possible.

Andrew

[Lewis Levin]

Thanks for the reply.

I have tried from the tar ball of h5py with the same exact problem.

There is a reason I don't want to use brew python.  I am trying to build a very clean installer, along the lines of winpython, but for the Mac.  In fact, I'll make any dependencies requiring brew optional as its dependencies, Xcode with clt, may be too challenging.  Hence, the interest in the binary dist of hdf5.  It doesn't really seem that hdf5 is the culprit.  It really seems to be that h5py or its install process couldn't find hdf5.  I did set hdf5_dir because that seemed the way to force the issue.  But, I will eliminate that.  I also tried referencing the library location in the command line for setup.py as suggested, to no avail.  If these are truly no-ops, why not eliminate them from the documentation and from the code?

I am not using the system Python.   I am using python resulting from the binary installer for the mac at python.org.  This is the "official" python.  

I actually got exactly this config to work on one machine--pretty effortlessly actually.   I can't really tell what's different except the order in which things were done and if there is detritus left from previous attempts.

[Lewis Levin]

It seems that one must have some way to tell the installer script where to find hdf5.  Unless there is some secret sauce deposited in the file system by the installation of the hdf5 libraries, it would appear that auto-discovery (grep?) is really broken:

There are many indications that auto detect easily fails except under magic (undocumented and optimistically non-deterministic) conditions:

In file included from h5py/defs.c:259:

h5py/api_compat.h:22:10: fatal error: 'hdf5.h' file not found

#include "hdf5.h"

         ^

1 warning and 1 error generated.

error: command '/usr/bin/clang' failed with exit status 1

AND:

 dlopen(libhdf5.so, 6): image not found

    HDF5 autodetection failed; building for 1.8.4+

When I am actually building against 1.8.7 (well, I guess > 1.8.4 should work...)

[Andrew Collette]

Hi Lewis,

> It seems that one must have some way to tell the installer script where to
> find hdf5.

There is. Set $HDF5_DIR, or use --hdf5=/path/to/hdf5 when calling setup.py:

http://docs.h5py.org/en/latest/build.html#custom-installation

All this does is add the path you provide to the standard compiler
search paths, analogous to adding "-I/path/to/include" for gcc. We
also automatically provide some common include paths in setup.py, for
both macports and homebrew.

> Unless there is some secret sauce deposited in the file system
> by the installation of the hdf5 libraries, it would appear that
> auto-discovery (grep?) is really broken:

Auto-detection refers only to *version* auto-detection; if you have
HDF5 installed somewhere ctypes can find it, we can figure out what
version it is and have a chance to enable features more recent than
HDF5 1.8.4. Otherwise, we'll assume the earliest version h5py
supports (1.8.4). You can also override this with $HDF5_VERSION or
--hdf5-version=1.8.X option to setup.py.

HTH,
Andrew

[Lewis Levin]

Well, thanks again.

You are off the hook.  

It seems that the hdf5 binary distribution for os x has a hard coded mount point in a settings file that refers to a specific developer's machine, which probably got carried forward into various header files and compiled files.  This was reported on their forum in 2011!  They claimed to fix it but they seem not to have done so.  I have sent a message to their list for their help.   It doesn't seem they particularly care about these distributions as no one seems to test them or use them beyond the single maintainer who creates them.

[Lewis Levin]

Oh, and I had used that environment variable--while it seems essential for pip or easy_install to work, it didn't solve the problem once h5py was built.  I confrmed this because pytables has the same problem.

[Lewis Levin]

On a finally note, I have cried "uncle".  I just build the thing like anyone else and without brew.   The HDF5 installation documentation leaves much to be desired, especially platform specific (and quite simple) issues.  But, in the end it is not a big deal to build the thing.

I wish OSS maintainers built binaries for major platforms.  I understand the movement ideology, but there is also reliability and convenience for users to consider.

[Thomas Caswell]

For the most part it has nothing to do with ideology and all to do
with time/resources.

As you have discovered building distributable binaries is hard. The
typical dev does not have the expertise or resources to build,
maintain, distribute, and support binaries for every 'major' system
(which is setting aside the problem of what is a 'major' platform, if
you ask me the major platforms are debian derived distributions, RHEL
derived distributions, and BSD flavors).

Tom

> --
> You received this message because you are subscribed to the Google Groups
> "h5py" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to h5py+uns...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Thomas Caswell
tcas...@gmail.com