HDF5 1.10.0 and Anaconda install

[Jt Sauls]

Hello all, 

HDF5 1.10.0 is released as of 2016.03.30 which is required to get at SWMR functionality in h5py 2.5.0 (and h5py 2.6.0). 

If you want to custom install h5py against the new library I think all is peachy with from source or with pip. However I was wondering how to do this with a conda install. 

> conda install h5py

Will give me version 2.6.0 of h5py but only HDF5 1.8.16. It seems so other people have had some success making a conda recipe to build against the parallel version of hdf5. I am not familiar with making recipes but I can give that a stab if that is the way to go. 

What is the best way to go about using h5py with anaconda but built against the hdf5 so be able to use SWMR mode? I installed hdf5 1.10.0 from source and tried to use Anaconda's pip to install h5py, but it doesn't load all the libraries, specifically:

ImportError: dlopen(//anaconda/lib/python2.7/site-packages/h5py/_errors.so, 2): Library not loaded: libhdf5.10.dylib
  Referenced from: //anaconda/lib/python2.7/site-packages/h5py/_errors.so
  Reason: image not found

Any help greatly appreciated, 

jt

[Toby Shearman]

Did you ever find a solution?

I'm trying to do the same thing but haven't found a way to get my anaconda installation to talk to an updated version of HDF5 (1.10.0).

Thanks!

ts

[Jt Sauls]

No, unfortunately not. I couldn't force conda to install HDF5 1.10.0, even in a separate environment and using my own conda recipe. I'm sure it's possible, but I couldn't figure it out. 

After installing HDF5 1.10.0 from source I ended up using another python environment outside of anaconda (using pip and virtualenv) for this project. 

Please let me know if you have any more success. 

[Jt Sauls]

Okay so I solved this and it wasn't any fault of conda or h5py. The problem was my GCC compiler was messed up upon upgrading to OSX El Capitan. Command line tools (which is needed for GCC) seems to be lost from Xcode when upgrading, so even if you had it before you don't have it after. You need GCC to compile h5py against HDF5 1.10. 

You can fix the Xcode command line tools pretty simply by putting this into your terminal:

> xcode-select --install

Then you should be fine. Install HDF5 1.10 from source into somewhere you want to. The .tar is here:

https://www.hdfgroup.org/HDF5/release/obtainsrc5110.html

Follow the install readme but basically you just need to give it a directory with:

> ./configure --prefix=/usr/local

before you make. 

Now install with you anaconda version of python. You may want to make a separate environment using conda but that's your call. 

Remove the h5py you have with anaconda using 

> conda uninstall h5py 

or

> pip uninstall h5py

Then use pip to reinstall h5py but pointing to the HDF5 library you made from source. From here: http://docs.h5py.org/en/latest/build.html

> HDF5_DIR=/usr/local/hdf5 pip install h5py

Then you should be good. Open up a python terminal and test if you can use SWMR mode:

> import h5py

> f = h5py.File("./swmr.h5", 'a', libver='latest', swmr=True)