Read or Write a compound datatype with h5py in python

richie

unread,

Jul 8, 2015, 10:30:43 PM7/8/15

to h5...@googlegroups.com

Hi, I just notice the existence of this mailing list.

This is a question I posted early today in stackoverflow: stackoverflow_post

I want to use hdf5 file among some C++, matlab, and python code.

My h5 file works well in C++ and matlab, but cannot be read with h5py.

Is data types like H5T_STD_B64LE not well supported by h5py?

I attached a small h5 file as the test file.

Thanks!

In [2]: f = h5py.File('art.mips.log.h5', 'r')

In [3]: f.keys()

Out[3]: [u'mem']

In [4]: f['mem']

Out[4]: <repr(<h5py._hl.dataset.Dataset at 0x29f70d0>) failed: TypeError: No NumPy equivalent for TypeBitfieldID exists>

The hdf5 file format is as follows:

$ h5dump -H art.mips.log.h5

HDF5 "art.mips.log.h5" { GROUP "/" { DATASET "mem" {

DATATYPE H5T_COMPOUND {

H5T_STD_B64LE "clock";

H5T_STD_B32LE "addr";

H5T_STD_B32LE "operation";

H5T_ARRAY { [16] H5T_STD_B32LE } "data";

}

DATASPACE SIMPLE { ( 68614468 ) / ( H5S_UNLIMITED ) }

ATTRIBUTE "CLASS" {

DATATYPE H5T_STRING {

STRSIZE 6;

STRPAD H5T_STR_NULLTERM;

CSET H5T_CSET_ASCII;

CTYPE H5T_C_S1;

}

DATASPACE SCALAR

}

ATTRIBUTE "FIELD_0_NAME" {

DATATYPE H5T_STRING {

STRSIZE 6;

STRPAD H5T_STR_NULLTERM;

CSET H5T_CSET_ASCII;

CTYPE H5T_C_S1;

}

DATASPACE SCALAR

}

ATTRIBUTE "FIELD_1_NAME" {

DATATYPE H5T_STRING {

STRSIZE 5;

STRPAD H5T_STR_NULLTERM;

CSET H5T_CSET_ASCII;

CTYPE H5T_C_S1;

}

DATASPACE SCALAR

}

ATTRIBUTE "FIELD_2_NAME" {

DATATYPE H5T_STRING {

STRSIZE 10;

STRPAD H5T_STR_NULLTERM;

CSET H5T_CSET_ASCII;

CTYPE H5T_C_S1;

}

DATASPACE SCALAR

}

ATTRIBUTE "FIELD_3_NAME" {

DATATYPE H5T_STRING {

STRSIZE 5;

STRPAD H5T_STR_NULLTERM;

CSET H5T_CSET_ASCII;

CTYPE H5T_C_S1;

}

DATASPACE SCALAR

}

ATTRIBUTE "TITLE" {

DATATYPE H5T_STRING {

STRSIZE 4;

STRPAD H5T_STR_NULLTERM;

CSET H5T_CSET_ASCII;

CTYPE H5T_C_S1;

}

DATASPACE SCALAR

}

ATTRIBUTE "VERSION" {

DATATYPE H5T_STRING {

STRSIZE 4;

STRPAD H5T_STR_NULLTERM;

CSET H5T_CSET_ASCII;

CTYPE H5T_C_S1;

}

DATASPACE SCALAR

}

sanity.log.h5

Andrew Collette

unread,

Jul 9, 2015, 4:13:28 PM7/9/15

to h5...@googlegroups.com

Hi,

> I want to use hdf5 file among some C++, matlab, and python code.
> My h5 file works well in C++ and matlab, but cannot be read with h5py.
>
> Is data types like H5T_STD_B64LE not well supported by h5py?

The HDF5 bitfield types have no direct equivalent in Python, so
they're not supported in h5py.

You could try using the astype property to convert to e.g. a 64-bit
integer, but it depends on whether or not HDF5 can convert from
bitfield to integer:

http://docs.h5py.org/en/latest/high/dataset.html#Dataset.astype

Andrew

richie

unread,

Jul 9, 2015, 4:22:58 PM7/9/15

to h5...@googlegroups.com

Hi Andrew,

Thanks for your reply. Can you give a little bit more pointers on astype?

I tried this way last night, it did not work though.

I guess the problem is I should not use "H5T_NATIVE_B*" type when the file is generated in C code. Nevertheless matlab is able to read this out.

In [4]: comp_type = np.dtype([('clock', h5py.h5t.STD_U64LE), ('addr', h5py.h5t.STD_U32LE), ('operation', h5py.h5t.STD_U32LE), ('data', h5py.h5t.STD_U32LE, (16,))])

In [5]: with f['mem'].astype(comp_type):

...: out = f['mem']

...:

In [6]: out.dtype

---------------------------------------------------------------------------

TypeError Traceback (most recent call last)

<ipython-input-6-13983d68aade> in <module>()

----> 1 out.dtype

Andrew Collette

unread,

Jul 9, 2015, 5:29:51 PM7/9/15

to h5...@googlegroups.com

Hi,

> Thanks for your reply. Can you give a little bit more pointers on astype?
> I tried this way last night, it did not work though.

You should provide a NumPy dtype, not something in h5py.h5t. For
example, if I had an integer dataset and wanted to read it as float:

with dset.astype('f):
out = dset[:]

Andrew

Reply all

Reply to author

Forward