HDF5 and IDL

[Matt Newville]

Hi Folks,

I made a little progress on figuring out some of the issues that I
reported last week for reading HDF5 files with IDL.

My basic conclusion is:
The HDFView program supplied by the HDF Group alters
HDF5 Files in such a way that IDL can not open them.

The essential test is to perform a checksum (say, with shasum) before
and after opening an HDF5 file with HDFView. While the file size is
not changed, the checksums *ARE* different -- the file has been
altered in the first few hundred bytes. IDL's H5F_OPEN (which appears
to be a thin wrapper on the routine of the same name for the HDF5 C
library) fails on the files that have been altered by HDFView.

I do not know why HDFView alters the files this way, or why this
change means that IDL can not read them. I suspect that other
third-party applications using HDF5 v1.6 may have a similar problem
reading files altered by HDFView. I will write to the HDF5 mailing
list.

More extensive notes are at
http://cars9.uchicago.edu/pybeamline/DataFormats/H5UsageNotes/HDF5AndIDL

Has anyone else seen similar problems?

--Matt Newville

[Chris Jacobsen]

Sounds pretty bizarre! One quick way to look at changes in the
header:
% h5dump file.h5 >before.txt
% hdfview file.h5
% h5dump file.h5 >after.txt
% diff before.txt after.txt

[Matt Newville]

Actually, h5dump gives the same results (modulo the file name) for the
altered and unaltered file. The data (groups, datasets, attributes)
are not changed, only the header is changed. In addition, neither
h5diff nor h5stat shows any differences between the altered and
unaltered files.

Since these tests were all identical, and the file sizes were the
same, it took me a while to even try a diff or shasum on these files.

--Matt Newville <newville at cars.uchicago.edu>

[Gerd Wellenreuther]

Sounds to me like you should write a bugreport for the HDFview. (I
expect they agree that a programm called xyz-view should change anything
in a file ...)

Cheers, Gerd

Matt Newville schrieb:

> --
>
> You received this message because you are subscribed to the Google Groups "Methods for the analysis of hyperspectral image data" group.
> To post to this group, send email to ma...@googlegroups.com.
> To unsubscribe from this group, send email to mahid+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mahid?hl=en.
>
>
>

[Nicholas H]

Hi Folks,

HDFview does 'change' hdf files. At least version 1.8.3 does. The fix
for us is to use a previous version of the viewer, and wait until the
next version of HDFview is released, which 'should' fix the problem.

We saw this problem about 2 months ago. HDF are aware of the problem.
See the response from he...@hdfgroup.org below.

I have the story straight now. There were actually *two* problems in
HDF5 which caused an HDF5 file to be updated. One, was fixed in 1.8.3,
and the other will be fixed in HDF5 1.8.4. Actually both issues are
listed in the HDF5 1.8.3 Release Notes, which you can see
here:

ftp://ftp.hdfgroup.org/HDF5/current/src/hdf5-1.8.3-RELEASE.txt

Under "Bug Fixes", you will see the Symbol Table Issue:

- Modified library to be able to open files with corrupt root group
symbol table messages, and correct these errors if they are found.
Such files can only be successfully opened with write access.
NAF - 2009/03/23 - 1189

Under "Known Problems", you will see the other issue, which had to do
with the timestamp getting modified:

* There is a known issue in which HDF5 will change the timestamp on a
file
simply by opening it with read/write permissions, even if the file
is
not modified in any way. This is due to the way in which HDF5
manages
the file superblock. A fix is currently underway and should be
included
in the 1.8.4 release of HDF5. MAM - 2009/04/28

I asked the developer to try the development version of HDFView that
is using a pre-release of 1.8.4. It definitely does not update a good
HDF5 file.


Regards,

Nick Hauser.