hdf5plugin 2.0 beta version providing blosc, bitshuffle and lz4 compression filters for h5py

[Thomas Vincent]

Dear all,

We've recently been reworking the hdf5plugin project (https://github.com/silx-kit/hdf5plugin) in order to provide a generic way to enable some HDF5 compression filters in h5py.

It now provides blosc, bitshuffle and lz4 compression/decompression filters and supports Linux, Windows and macOS.

On Linux and macOS it uses dynamic loading of the HDF5 library symbols, so there is no need to compile the filters with the same HDF5 library as h5py.

On Windows, as DLL linking is different, the filters are linked directly to the already loaded HDF5 library.

We made a beta release and wheels are available on pypi: https://pypi.org/project/hdf5plugin/2.0.0b0/

To install it, run: pip install --pre hdf5plugin

To register the filters, just do: import hdf5plugin and the filters are loaded.

Feedbacks and issues are welcome on the github project page: https://github.com/silx-kit/hdf5plugin

Best regards,

    Thomas

[Thomas Vincent]

Dear all,

We've just released hdf5plugin v2.0.0 (https://github.com/silx-kit/hdf5plugin), that provides a generic way to enable blosc, bitshuffle and lz4 compression filters for h5py.

It's available on pypi: https://pypi.org/project/hdf5plugin/

Changelog since previous release:

• Added compression support for Linux and macOS
• Added blosc filter
• Added helper class (Blosc, Bitshuffle and LZ4) to ease providing compression arguments to h5py
• Added tests
• Updated documentation
• Building from source through setup.py
• No longer use the plugin mechanism via HDF5_PLUGIN_PATH environment variable

Best,

    Thomas

[Eli Stevens]

Hi,

Since there isn't a top-level `hdf5plugin` directory in the wheel, I'm having problems getting everything to work under bazel.

  inflating: hdf5plugin-2.0.0b0.data/platlib/hdf5plugin/test.py
  inflating: hdf5plugin-2.0.0b0.data/platlib/hdf5plugin/__init__.py
  inflating: hdf5plugin-2.0.0b0.data/platlib/hdf5plugin/_version.py
  inflating: hdf5plugin-2.0.0b0.data/platlib/hdf5plugin/plugins/libh5bshuf.so
  inflating: hdf5plugin-2.0.0b0.data/platlib/hdf5plugin/plugins/libh5lz4.so
  inflating: hdf5plugin-2.0.0b0.data/platlib/hdf5plugin/plugins/libh5blosc.so
  inflating: hdf5plugin-2.0.0b0.dist-info/RECORD
  inflating: hdf5plugin-2.0.0b0.dist-info/METADATA
  inflating: hdf5plugin-2.0.0b0.dist-info/WHEEL
  inflating: hdf5plugin-2.0.0b0.dist-info/top_level.txt

What's the reason for not having a top-level hdf5plugin directory? How does python find the right place to import the package from?

If I manually edit my sys.path entry for hdf5plugin to include `+ '/hdf5plugin-2.0.0b0.data/platlib'` then it seems to work, but that's too hacky to check in to my work repo.

Thanks,

Eli

--
You received this message because you are subscribed to the Google Groups "h5py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to h5py+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/h5py/33fc7022-eaff-432c-8103-30a799cf3a72%40googlegroups.com.

[Thomas Vincent]

Hi,

This is due to the tweaking of the wheel generation that makes a single wheel compatible with all the versions of python.

I've opened an issue for that: https://github.com/silx-kit/hdf5plugin/issues/48

BTW, I'm not familiar with bazel, why do you need to access the raw content of the wheel instead of installing it?

Best regards,

    Thomas

Le mardi 29 octobre 2019 22:04:06 UTC+1, Eli Stevens a écrit :

Hi,

Since there isn't a top-level `hdf5plugin` directory in the wheel, I'm having problems getting everything to work under bazel.

  inflating: hdf5plugin-2.0.0b0.data/platlib/hdf5plugin/test.py
  inflating: hdf5plugin-2.0.0b0.data/platlib/hdf5plugin/__init__.py
  inflating: hdf5plugin-2.0.0b0.data/platlib/hdf5plugin/_version.py
  inflating: hdf5plugin-2.0.0b0.data/platlib/hdf5plugin/plugins/libh5bshuf.so
  inflating: hdf5plugin-2.0.0b0.data/platlib/hdf5plugin/plugins/libh5lz4.so
  inflating: hdf5plugin-2.0.0b0.data/platlib/hdf5plugin/plugins/libh5blosc.so
  inflating: hdf5plugin-2.0.0b0.dist-info/RECORD
  inflating: hdf5plugin-2.0.0b0.dist-info/METADATA
  inflating: hdf5plugin-2.0.0b0.dist-info/WHEEL
  inflating: hdf5plugin-2.0.0b0.dist-info/top_level.txt

What's the reason for not having a top-level hdf5plugin directory? How does python find the right place to import the package from?

If I manually edit my sys.path entry for hdf5plugin to include `+ '/hdf5plugin-2.0.0b0.data/platlib'` then it seems to work, but that's too hacky to check in to my work repo.

Thanks,

Eli

On Mon, Oct 21, 2019 at 4:43 AM Thomas Vincent <thomas...@esrf.fr> wrote:

Dear all,

We've just released hdf5plugin v2.0.0 (https://github.com/silx-kit/hdf5plugin), that provides a generic way to enable blosc, bitshuffle and lz4 compression filters for h5py.

It's available on pypi: https://pypi.org/project/hdf5plugin/

Changelog since previous release:

• Added compression support for Linux and macOS
• Added blosc filter
• Added helper class (Blosc, Bitshuffle and LZ4) to ease providing compression arguments to h5py
• Added tests
• Updated documentation
• Building from source through setup.py
• No longer use the plugin mechanism via HDF5_PLUGIN_PATH environment variable

Best,

    Thomas

--
You received this message because you are subscribed to the Google Groups "h5py" group.

To unsubscribe from this group and stop receiving emails from it, send an email to h5...@googlegroups.com.

[Thomas Kluyver]

Just to highlight: the wheel format, unlike eggs, is not meant to support importing from the package directly without installing it. It happens to work in many cases, but it's not something you should rely on in general:

https://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file

To unsubscribe from this group and stop receiving emails from it, send an email to h5py+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/h5py/08a47328-8079-4e03-9f0b-38a513d6d990%40googlegroups.com.

[Eli Stevens]

Sigh, I'm not surprised that Bazel isn't interacting with wheels properly. It's pretty apparent that the people who wrote the python support modules weren't familiar with the python ecosystem (a lot of NIH syndrome, not supporting virtualenvs, leaking system packages into what's supposed to be a hermetic build system, not knowing that unzipping a wheel isn't the same as installing it, etc.).

To more directly answer the question: Bazel takes over the construction of the environment that it's build/run targets operate in, and that includes unzipping wheels and putting the resulting directories on the PYTHONPATH. There is no virtualenv to install the packages into, since the python that gets invoked is just the underlying system python. I suspect that the moment we need a wheel that actually needs to compile something during installation, we'll be stuck.

In any case, thank you for looking into this. I appreciate your spending time on it, when it's clear that Bazel is at fault. Having looked at and made local tweaks to Bazel for other issues, I don't see a clean way to fix this without a major overhaul to how Bazel builds python environments (but I'm far from a guru, so maybe I'm missing something).

Cheers,

Eli

To view this discussion on the web visit https://groups.google.com/d/msgid/h5py/CAOvn4qh%3DSrvmDP3wP8Q17aogCtExr0ctgnh74yHhheh%2BeMriSg%40mail.gmail.com.

[Thomas Vincent]

I see. Anyway, I improved the wheel generation of hdf5plugin, so this should be solved (https://github.com/silx-kit/hdf5plugin/pull/49).

Best,

  Thomas

Le mardi 5 novembre 2019 19:15:31 UTC+1, Eli Stevens a écrit :

Sigh, I'm not surprised that Bazel isn't interacting with wheels properly. It's pretty apparent that the people who wrote the python support modules weren't familiar with the python ecosystem (a lot of NIH syndrome, not supporting virtualenvs, leaking system packages into what's supposed to be a hermetic build system, not knowing that unzipping a wheel isn't the same as installing it, etc.).

To more directly answer the question: Bazel takes over the construction of the environment that it's build/run targets operate in, and that includes unzipping wheels and putting the resulting directories on the PYTHONPATH. There is no virtualenv to install the packages into, since the python that gets invoked is just the underlying system python. I suspect that the moment we need a wheel that actually needs to compile something during installation, we'll be stuck.

In any case, thank you for looking into this. I appreciate your spending time on it, when it's clear that Bazel is at fault. Having looked at and made local tweaks to Bazel for other issues, I don't see a clean way to fix this without a major overhaul to how Bazel builds python environments (but I'm far from a guru, so maybe I'm missing something).

Cheers,

Eli

On Tue, Nov 5, 2019 at 9:37 AM Thomas Kluyver <tak...@gmail.com> wrote:

Just to highlight: the wheel format, unlike eggs, is not meant to support importing from the package directly without installing it. It happens to work in many cases, but it's not something you should rely on in general:

https://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file

To view this discussion on the web visit https://groups.google.com/d/msgid/h5py/08a47328-8079-4e03-9f0b-38a513d6d990%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "h5py" group.

To unsubscribe from this group and stop receiving emails from it, send an email to h5...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/h5py/CAOvn4qh%3DSrvmDP3wP8Q17aogCtExr0ctgnh74yHhheh%2BeMriSg%40mail.gmail.com.

[Thomas Kluyver]

On Tue, 5 Nov 2019 at 19:15, Eli Stevens <este...@zoox.com> wrote:

I suspect that the moment we need a wheel that actually needs to compile something during installation, we'll be stuck.

If it installs from wheels published on PyPI, it should never need to compile something - any compilation is done before making the wheel.

In most cases, simply unzipping a wheel will get you a usable package - the FAQ I linked to before is mostly about using wheel packages directly with Python's zipimport functionality, which is considerably dicier. But it's still not guaranteed - as far as I know, what hdf5plugin was doing before is valid, if unusual.

Here's some code I wrote in another project which tries to handle this kind of variation when extracting modules from wheels. As far as I know, it works pretty well, but it only has to handle Windows wheels:

https://github.com/takluyver/pynsist/blob/2.4/nsist/wheels.py#L218-L279

Thomas

[Thomas Vincent]

Dear all,

We recently made a new release of hdf5plugin (v2.3.0) to provide additional compression filters for h5py, namely:

- ZFP compression filter enabling lossy compression.

- FCIDECOMP compression filter

It is available on:

- pypi: https://pypi.org/project/hdf5plugin/2.3.0/
- conda: https://anaconda.org/conda-forge/hdf5plugin

Changelog: https://github.com/silx-kit/hdf5plugin/releases
Please, report any issue here: https://github.com/silx-kit/hdf5plugin/issues

Best,
   Thomas