Error: Blosc library does not have support for the 'snappy' compressor, but only for: blosclz,lz4,lz4hc,zlib,zstd

[Sal Abbasi]

Hi,

I have a 30 GB hdf5 file with datasets compressed using the snappy algorithm.  I recently upgraded anaconda, h5py and pytables and now I can no longer read this file.  When I try to read a dataset using the following code:

import tables

import h5py

f = h5py.File('/tmp/options.hdf5', 'r')

f['prices/E1AF8/C:2830/timestamp'][()]

I get the stack trace below.  Strangely enough, if I copy one small group from the file using h5copy, I can read the new file, but if I copy the parent group, I get the same error when reading data from the new file.

I am running on Mac OS 10.15.15 with python 3.7.6, hdf5 1.10.6 ,  h5py 2.10.0 and pytables 3.6.1.  I tried downgrading all these files to their original versions, but still get the same error.

Best,

Sal

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-3-d7e57ae31f9d> in <module>
      1 f = h5py.File('/tmp/options.hdf5', 'r')
----> 2 f['prices/E1AF8/C:2830/timestamp'][()]

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

~/anaconda3/envs/py37/lib/python3.7/site-packages/h5py/_hl/dataset.py in __getitem__(self, args)
    571         mspace = h5s.create_simple(mshape)
    572         fspace = selection.id
--> 573         self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
    574 
    575         # Patch up the output for NumPy

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5d.pyx in h5py.h5d.DatasetID.read()

h5py/_proxy.pyx in h5py._proxy.dset_rw()

h5py/_proxy.pyx in h5py._proxy.H5PY_H5Dread()

OSError: Can't read data (this Blosc library does not have support for the 'snappy' compressor, but only for: blosclz,lz4,lz4hc,zlib,zstd)

[Thomas Kluyver]

Hi Sal,

h5py doesn't include any Blosc library, so I guess it's being picked up from pytables.

Maybe installing this conda package would help? https://anaconda.org/conda-forge/blosc-hdf5-plugin

Thomas

--
You received this message because you are subscribed to the Google Groups "h5py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to h5py+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/h5py/15bc0dab-ee1c-4298-a2de-b5c662d4f916o%40googlegroups.com.

[Jerome Kieffer]

On Thu, 2 Jul 2020 20:22:04 -0700 (PDT)
Sal Abbasi <abbas...@gmail.com> wrote:

> Hi,
>
> I have a 30 GB hdf5 file with datasets compressed using the snappy
> algorithm. I recently upgraded anaconda, h5py and pytables and now I can
> no longer read this file. When I try to read a dataset using the following
> code:

I guess you are missing the blosc compression plugin for hdf5. Maybe you can try an:
`conda install hdf5plugin`
Cheers,

Jerome

[Sal Abbasi]

Thanks Thomas and Jerome. 

Installing hdf5plugin allows me to read the file as long as I don't use "import tables" or use the pandas function pd.read_hdf to read a pandas dataframe from hdf5 anywhere in my code.

I think the issue is that pytables uses a different version of the blosc library that does not have a snappy filter, and pandas uses pytables under the cover.  As soon as I use the pandas read_hdf5 function, h5py seems to start using the blosc library without snappy from that point forward.  It does not matter whether or not I have imported hdf5plugin. 

Does anyone know of a way to rebuild pytables using the same blosc library as hdf5plugin? I tried setting BLOSC_DIR using export BLOSC_DIR=/Users/sal/anaconda3/envs/py37/lib/python3.7/site-packages/hdf5plugin/plugins before running conda install pytables as suggested in the pytables documentation but that does not seem to work.

[Thomas Kluyver]

On Fri, 3 Jul 2020 at 15:37, Sal Abbasi <abbas...@gmail.com> wrote:

Does anyone know of a way to rebuild pytables using the same blosc library as hdf5plugin? I tried setting BLOSC_DIR using export BLOSC_DIR=/Users/sal/anaconda3/envs/py37/lib/python3.7/site-packages/hdf5plugin/plugins before running conda install pytables as suggested in the pytables documentation but that does not seem to work.

conda packages are pre-compiled, so 'conda install' will never build something from source. You'll need to try a different way of building it. If you get stuck with that, pytables has its own mailing list you can ask: https://groups.google.com/forum/#!forum/pytables-users

Best wishes,

Thomas

[Sal Abbasi]

Will do.  Thanks for the help.

On Friday, July 3, 2020 at 9:45:41 AM UTC-5, Thomas Kluyver wrote: