[MOTION] Drop HDF5 1.6 support in PyTables 2.4

[Antonio Valentino]

Hi list,
support for HDF5 1.6 has been deprecated in PyTables 2.3 and we planned
HDF5 1.6 removal for PyTables 3 (see gh-105).

The switch from HDF5 1.6 API to 1.8 API do not introduces any
incompatibility in the Python API so I propose to re-target gh-105 to
PyTables 2.4 and perform the switch now.

The porting has been done in [1] and the code is ready to be merged in
the develop branch. Minimum HDF5 version is now 1.8.4.

[1] https://github.com/avalentino/PyTables/tree/issue105

cheers

--
Antonio Valentino

[Anthony Scopatz]

The last release of HDF5 1.6 was in Nov. 2009 which doesn't seem 

like all that long ago. That said, going to 1.8 only a minor version number 

bump.  Moreover, you have already done the work and the 1.6 compatibility 

I seem to recall has caused some problems.

So I guess I am tentatively +1 for this.  

However, what do we tell people who want 1.6.x?   I see two basic options:

• Use PyTables 2.3 or before
• Upgrade to HDF5 1.8

Is everyone ok with these?

Be Well

Anthony

[Josh Ayers]

Based on this link [1], there doesn't appear to be any compatibility issues accessing a file in 1.8 that was originally created in 1.6.  That seems like one of the biggest issues that would prevent users from upgrading.  It also means that if they use HDF5 outside of PyTables, those applications will still work after the upgrade.

I'm +1 as well (for what that's worth).

[1] http://www.hdfgroup.org/HDF5/doc/ADGuide/WhatsNew180.html

Josh

[Francesc Alted]

Wow, very thoughtful job Antonio. I'm impressed.

Add another +1 for 1.6 removal. Supporting only HDF5 1.8 series is
going to be beneficial for the development in the future (specially now
that 1.10 should be released soon).

Thanks for the nice work!

--
Francesc Alted

[Antonio Valentino]

Hi Francesc,

Thank you.

I have two questions for you :)

a. I found comments like the following "THG team has decided to fix an
API inconsistency in the definition of the H5Z_class_t structure in
version 1.8.3" see [1] and this is the reason why I set 1.8.4 as
minimum supported version.
Unfortunately I was not able to find any reference to such an API
change in the HDF5 cheangelog [2].
Can you please confirm the issue or provide some pointer to related
doc? I'm trying to understand if the minimum supported version can
be safely set to 1.8.0
b. I performed some little changes to the blosc code [3]. Can you
please check them and, in case, include them in main blosc repo?

[1]
https://github.com/avalentino/PyTables/commit/ca8d14aba07dee4717ad55ae531b363406d852dc#L2L25
[2] http://www.hdfgroup.org/HDF5/doc/ADGuide/Changes.html
[3]
https://github.com/avalentino/PyTables/commit/ca8d14aba07dee4717ad55ae531b363406d852dc#diff-0

cheers

--
Antonio Valentino

[Josh Moore]

> Antonio Valentino

Sorry for chiming in late (was enjoying Italy). My only concern would be if there is any major distro where someone is stuck on 1.6, but that seems highly unlikely. It might be worth writing to pytables-users to just see if there's anyone in that boat. If there are no vetoes and no one has done so by this evening, I'll send an email.

Cheers,
the other Josh :)

[Francesc Alted]

Well, the original code (before your modifications) could cope with HDF5
< 1.8.3 and HDF5 >= 1.8.3, so I'd say that the minimum HDF5 supported
version can be safely set to 1.8.0. Having said that, my preference
would be to set the minimum version to something more recent, like 1.8.3
or 1.8.4, but this mostly depends on the version that distros currently
ship. But if we would have to choose, I'd take 1.8.3 as the minimum, as
it simplifies the code a lot.

The issue with H5Z_class_t have briefly been discussed in the HDF5
mailing list:

http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/2009-May/000297.html

The referred ticket is in the old trac:

http://www.pytables.org/trac-bck/ticket/225

> b. I performed some little changes to the blosc code [3]. Can you
> please check them and, in case, include them in main blosc repo?

Good point. This was a bit trickier, but here it is my solution:

https://github.com/FrancescAlted/blosc/commit/a9ad9191dad821af874d49adbf03271331aecc94

Ciao,

--
Francesc Alted

[Anthony Scopatz]

+1 for 1.8.3

[Antonio Valentino]

Hi Josh,

Il 11/04/2012 12:54, Josh Moore ha scritto:
>
> On Apr 9, 2012, at 10:54 AM, Antonio Valentino wrote:
>
>> Hi Francesc,
>>
>> Il 07/04/2012 23:11, Francesc Alted ha scritto:
>>> On 4/6/12 2:15 PM, Antonio Valentino wrote:
>>>> Hi list,
>>>> support for HDF5 1.6 has been deprecated in PyTables 2.3 and we planned
>>>> HDF5 1.6 removal for PyTables 3 (see gh-105).
>>>>
>>>> The switch from HDF5 1.6 API to 1.8 API do not introduces any
>>>> incompatibility in the Python API so I propose to re-target gh-105 to
>>>> PyTables 2.4 and perform the switch now.
>>>>
>>>> The porting has been done in [1] and the code is ready to be merged in
>>>> the develop branch. Minimum HDF5 version is now 1.8.4.
>>>>
>>>>
>>>> [1] https://github.com/avalentino/PyTables/tree/issue105

[CUt]

>
> Sorry for chiming in late (was enjoying Italy). My only concern would be if there is any major distro where someone is stuck on 1.6, but that seems highly unlikely. It might be worth writing to pytables-users to just see if there's anyone in that boat. If there are no vetoes and no one has done so by this evening, I'll send an email.
>
> Cheers,
> the other Josh :)

I know by sure that

* Fedora/RedHat: all currently supported version use HDF5 >= 1.8.3 [1].
The only exception is EPEL5 released in the early 2007 [2]
* Debian: all recent versions use HDF5 >= 1.8.3, only oldstable (lenny),
released in Feb 2009, has HDF5 < 1.8.3
* Ubuntu: all recent versions use HDF5 >= 1.8.3, only Ubuntu 8.04.4 LTS
(Hardy Heron) has HDF5 < 1.8.3

not sure which is the situation for Mandriva, opensuse, CentOS,
ScientificLinux and other major distributions.

Anyway, IMHO it is a good idea to ask for some feedback from the user
mailing list. Please go ahead.

[1]
https://admin.fedoraproject.org/updates/hdf5?_csrf_token=e78cdf54f75038b4c2fe244d288aed8d820b88e2
[2] http://fedoraproject.org/wiki/Red_Hat_Enterprise_Linux#History

cheers

--
Antonio Valentino

[Antonio Valentino]

Hi Francesc,

Well if the H5Z_class_t has been modified in 1.8.3 we should restore the
old code in src/H5Zbzip2.c and src/H5Zlzo.c. Also we should drop the
H5_NO_DEPRECATED_SYMBOLS build flag in order to use H5Z_class1_t (the
old version of the structure).

> would be to set the minimum version to something more recent, like 1.8.3
> or 1.8.4, but this mostly depends on the version that distros currently
> ship. But if we would have to choose, I'd take 1.8.3 as the minimum, as
> it simplifies the code a lot.
>

+1 for 1.8.3

> The issue with H5Z_class_t have briefly been discussed in the HDF5
> mailing list:
>
> http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/2009-May/000297.html
>
>
> The referred ticket is in the old trac:
>
> http://www.pytables.org/trac-bck/ticket/225
>

OK, thank you.

>> b. I performed some little changes to the blosc code [3]. Can you
>> please check them and, in case, include them in main blosc repo?
>
> Good point. This was a bit trickier, but here it is my solution:
>
> https://github.com/FrancescAlted/blosc/commit/a9ad9191dad821af874d49adbf03271331aecc94
>

Umm, in ubuntu 11.10 and debian (up to HDF5 < 1.8.8) the library is
compiled to have 1.6 as default API hence H5_USE_16_API is automatically
defined.
While this is OK for blosc itself it will not work for the new version
of PyTables that uses compatibility macros assuming that the API version
is 1.8.

Probably it is better to check the H5Z and H5E API separately using
H5Z_class_t_vers and H5Epush_vers instead of H5_USE_16_API.

If it makes sense for you I can try to write a temptative solution in
the PyTables/issue105 branch.

cheers

--
Antonio Valentino

[Antonio Valentino]

Hi Francesc,

Il 11/04/2012 20:32, Antonio Valentino ha scritto:
> Hi Francesc,
>
> Il 11/04/2012 17:42, Francesc Alted ha scritto:
>> On 4/9/12 3:54 AM, Antonio Valentino wrote:

[CUT]

>
>>> b. I performed some little changes to the blosc code [3]. Can you
>>> please check them and, in case, include them in main blosc repo?
>>
>> Good point. This was a bit trickier, but here it is my solution:
>>
>> https://github.com/FrancescAlted/blosc/commit/a9ad9191dad821af874d49adbf03271331aecc94
>>
>
> Umm, in ubuntu 11.10 and debian (up to HDF5 < 1.8.8) the library is
> compiled to have 1.6 as default API hence H5_USE_16_API is automatically
> defined.
> While this is OK for blosc itself it will not work for the new version
> of PyTables that uses compatibility macros assuming that the API version
> is 1.8.
>

Oh sorry, that is not correct, the only problem is that I set
H5_NO_DEPRECATED_SYMBOLS

Removing that define all should work fine even if blosc will use the
HDF5 API 1.6.

> Probably it is better to check the H5Z and H5E API separately using
> H5Z_class_t_vers and H5Epush_vers instead of H5_USE_16_API.
>
> If it makes sense for you I can try to write a temptative solution in
> the PyTables/issue105 branch.

ciao

--
Antonio Valentino

[Antonio Valentino]

Hi Francesc,

Il 11/04/2012 21:36, Antonio Valentino ha scritto:
> Hi Francesc,
>
> Il 11/04/2012 20:32, Antonio Valentino ha scritto:
>> Hi Francesc,
>>
>> Il 11/04/2012 17:42, Francesc Alted ha scritto:
>>> On 4/9/12 3:54 AM, Antonio Valentino wrote:
>
> [CUT]
>
>>
>>>> b. I performed some little changes to the blosc code [3]. Can you
>>>> please check them and, in case, include them in main blosc repo?
>>>
>>> Good point. This was a bit trickier, but here it is my solution:
>>>
>>> https://github.com/FrancescAlted/blosc/commit/a9ad9191dad821af874d49adbf03271331aecc94
>>>
>>
>> Umm, in ubuntu 11.10 and debian (up to HDF5 < 1.8.8) the library is
>> compiled to have 1.6 as default API hence H5_USE_16_API is automatically
>> defined.
>> While this is OK for blosc itself it will not work for the new version
>> of PyTables that uses compatibility macros assuming that the API version
>> is 1.8.
>>
>
> Oh sorry, that is not correct, the only problem is that I set
> H5_NO_DEPRECATED_SYMBOLS
>
> Removing that define all should work fine even if blosc will use the
> HDF5 API 1.6.
>

Please ignore the above comment. If I use the current version of blosc
I'm not able to build the PyTables/issue105 branch even if I remove
H5_NO_DEPRECATED_SYMBOLS :(

[Antonio Valentino]

Hi Francesc,

Il 11/04/2012 20:32, Antonio Valentino ha scritto:

>>> >> b. I performed some little changes to the blosc code [3]. Can you
>>> >> please check them and, in case, include them in main blosc repo?
>> >
>> > Good point. This was a bit trickier, but here it is my solution:
>> >
>> > https://github.com/FrancescAlted/blosc/commit/a9ad9191dad821af874d49adbf03271331aecc94
>> >
> Umm, in ubuntu 11.10 and debian (up to HDF5 < 1.8.8) the library is
> compiled to have 1.6 as default API hence H5_USE_16_API is automatically
> defined.
> While this is OK for blosc itself it will not work for the new version
> of PyTables that uses compatibility macros assuming that the API version
> is 1.8.
>
> Probably it is better to check the H5Z and H5E API separately using
> H5Z_class_t_vers and H5Epush_vers instead of H5_USE_16_API.
>
> If it makes sense for you I can try to write a temptative solution in
> the PyTables/issue105 branch.

Please check
https://github.com/avalentino/PyTables/commit/a459f16a1968ef9941f5f35d17048d075741637b

ciao

--
Antonio Valentino

[Francesc Alted]

Works beautifully. Committed in blosc main repository:

https://github.com/FrancescAlted/blosc/commit/8df338a2d7e7b5a7a625db505243d06b75a352df

Grace mille!

--
Francesc Alted

[Antonio Valentino]

Hi,

Il 11/04/2012 12:54, Josh Moore ha scritto:
>

> On Apr 9, 2012, at 10:54 AM, Antonio Valentino wrote:
>
>> Hi Francesc,
>>
>> Il 07/04/2012 23:11, Francesc Alted ha scritto:
>>> On 4/6/12 2:15 PM, Antonio Valentino wrote:
>>>> Hi list,
>>>> support for HDF5 1.6 has been deprecated in PyTables 2.3 and we planned
>>>> HDF5 1.6 removal for PyTables 3 (see gh-105).
>>>>
>>>> The switch from HDF5 1.6 API to 1.8 API do not introduces any
>>>> incompatibility in the Python API so I propose to re-target gh-105 to
>>>> PyTables 2.4 and perform the switch now.
>>>>
>>>> The porting has been done in [1] and the code is ready to be merged in
>>>> the develop branch. Minimum HDF5 version is now 1.8.4.
>>>>
>>>>
>>>> [1] https://github.com/avalentino/PyTables/tree/issue105
>>>

[CUT]

>
> Sorry for chiming in late (was enjoying Italy). My only concern would be if there is any major distro where someone is stuck on 1.6, but that seems highly unlikely. It might be worth writing to pytables-users to just see if there's anyone in that boat. If there are no vetoes and no one has done so by this evening, I'll send an email.
>
> Cheers,
> the other Josh :)

OK, no feedback from users up to now.
IMHO we can declare the motion passed.

I will merge the issue105 ASAP.

best regards

--
Antonio Valentino

[Josh Moore]

+1
~J