Extendible H5 Datasets in Matlab using Low-Level Functions

[Heath Barnett]

I am trying to use the low-level matlab HDF5 functions to create a
extendible dataset. I am mostly converting the code from the
"Introduction to HDF5" (in the section on Extendible Datasets) guide
found at the website: <http://hdf.ncsa.uiuc.edu/HDF5/>

I've attached the code that I have written and everything runs fine
and I do get valid h5 files from the script. However, not all of the
data that I am trying to write to these files gets written to the
file. I should see this matrix:
[1 1 1 3 3;
1 1 1 3 3;
1 1 1 0 0;
2 0 0 0 0;
2 0 0 0 0;
2 0 0 0 0;
2 0 0 0 0;
2 0 0 0 0;
2 0 0 0 0;
2 0 0 0 0 ]

but instead I am getting:
[1 1 1 3 3;
1 1 1 3 0;
1 1 1 0 0;
0 0 0 0 0;
0 0 0 0 0;
0 0 0 0 0;
0 0 0 0 0;
0 0 0 0 0;
0 0 0 0 0;
0 0 0 0 0 ]

Am I doing some obviously wrong and just not seeing it?

Thanks in advance,

Heath Barnett

**Begin Code**
%% Constants and variable declaration.
clear; clc;
FILE = 'heath_extend6.h5';
DATASETNAME = 'extendarray';
RANK = 2;

dims0 = [3,3];
dims1 = [3,3];
dims2 = [7,1];
dims3 = [2,2];
maxdims = [];

chunk_dims = [2,5];

data1 = [1,1,1;1,1,1;1,1,1];
data2 = [2;2;2;2;2;2;2];
data3 = [3,3;3,3];

%% For an HDF5 file we need to create a dataspace, file, and dataset.
All
% other variables like cparms, size, offset are used to make the
HDF5 extendable.
% For this segment of code the offset is set to 0 since this is the
first
% time any dataset will be written to the file.
dataspace = H5S.create_simple(RANK,dims0,maxdims);

file = H5F.create(FILE,'H5F_ACC_TRUNC','H5P_DEFAULT', 'H5P_DEFAULT');

cparms = H5P.create('H5P_DATASET_CREATE');

H5P.set_chunk(cparms,chunk_dims)

dataset = H5D.create(file, DATASETNAME, 'H5T_STD_I32BE', dataspace,
'H5P_DEFAULT');

size = [3,3];

H5D.extend(dataset,size)

filespace = H5D.get_space(dataset);

offset = [0,0];

H5S.select_hyperslab(filespace,'H5S_SELECT_SET',offset,[],dims1,[])

H5D.write(dataset,'H5T_STD_I32BE',dataspace,filespace,'H5P_DEFAULT',in
t32(data1));

%% New dimensions are created and the h5 file is extended by the new
% dimensions.

dims0(1) = dims1(1)+dims2(1)
size = [dims0(1),dims0(2)]

H5D.extend(dataset,size)

filespace = H5D.get_space(dataset);

offset = [3, 0];

H5S.select_hyperslab(filespace,'H5S_SELECT_SET',offset,[],dims2,[])

dataspace = H5S.create_simple(RANK,dims2,[]);

H5D.write(dataset,'H5T_STD_I32BE',dataspace,filespace,'H5P_DEFAULT',in
t32(data2));

%% Same as previous session just a second extension.
dims0(2) = dims1(2) + dims3(2)
size = [dims0(1), dims0(2)]

H5D.extend(dataset,size);

filespace = H5D.get_space(dataset);

offset = [0,3];

H5S.select_hyperslab(filespace,'H5S_SELECT_SET',offset,[],dims3,[])

dataspace = H5S.create_simple(RANK,dims3,[]);

H5D.write(dataset,'H5T_STD_I32BE',dataspace,filespace,'H5P_DEFAULT',in
t32(data3));

%% Close all files that have been opened.
H5D.close(dataset);
H5S.close(dataspace);
H5F.close(file);
H5S.close(filespace);
**End Code**

[John Evans]

One thing is that while you are creating the property for an
extendable dataset, you aren't actually using it.

> cparms = H5P.create('H5P_DATASET_CREATE');
> H5P.set_chunk(cparms,chunk_dims)

> dataset = H5D.create(file, DATASETNAME, ...
> 'H5T_STD_I32BE', dataspace, ...
> 'H5P_DEFAULT');

Make that last property for H5D.create out to be cparms instead of
'H5P_DEFAULT'.

[Heath Barnett]

[Heath Barnett]

[Heath Barnett]

My last message got garbled sorry.

The changes you proposed worked well thanks. However, I now have a
question about H5D.extend. When you extend a dataset what gets
written into the new cells? I ask because my out now looks like this:

[1 1 1 3 3
1 1 1 3 3

1 1 1 3 3
2 1 1 3 3
2 1 1 3 3
2 1 1 3 3
2 1 1 3 3
2 1 1 3 3
2 1 1 3 3
2 1 1 3 3]

All of the data that I excplicity told the script to write is there
but there is an odd pattern of repeats in the rest of the cells that
I never bothered to specify a value for. Is this normal for
H5D.extend?

Thanks,

Heath Barnett

[John Evans]

Heath Barnett wrote:

I've noticed that as well. I wouldn't trust the value of any element I
haven't explicitly written to... What I would do instead is to define a
fill value for this dataset. For example, when you are creating the
properties for this dataset, you could do something like the following

cparms = H5P.create('H5P_DATASET_CREATE');
H5P.set_chunk(cparms,chunk_dims)

H5P.set_fill_value(cparms,'H5T_STD_I32BE', int32(-1));
dataset = H5D.create(file, DATASETNAME, 'H5T_STD_I32BE', dataspace, cparms);

Give that a try, and you should see that elements that have not been
explicitly written will be filled in with -1. Choose your own fill value
accordingly.

[Heath Barnett]

Great thanks John. Everything works wonderfully now.

Heath Barnett

[Alexander =]

"John Evans" <John....@mathworks.com> wrote in message
<ef55...@webcrossing.raydaftYaTP>...

> One thing is that while you are creating the property for an
> extendable dataset, you aren't actually using it.
>
> > cparms = H5P.create('H5P_DATASET_CREATE');
> > H5P.set_chunk(cparms,chunk_dims)
> > dataset = H5D.create(file, DATASETNAME, ...
> > 'H5T_STD_I32BE', dataspace, ...
> > 'H5P_DEFAULT');
>
> Make that last property for H5D.create out to be cparms
instead of
> 'H5P_DEFAULT'.
>

I'm running into a related problem when I try running this
code myself. When I try using the cparms ID in the
H5D.create(file_id, dsetname, 'H5T_NATIVE_INT', dataspace,
cparms) line, I get an error message (below) that states
that the identifier cparms is invalad, but it seems to be a
working identifier like any other.

??? Error using ==> H5ML.hdf5>postprocess
An invalid HDF5 identifier was returned from the HDF5 library:
"unable to create dataset"

Error in ==> example_5_tutorial at 38
dataset = H5D.create(file_id, dsetname, 'H5T_NATIVE_INT',
dataspace, cparms);

My code (included at the end) seems to be identical to what
was posted by Heath. Do you have any thoughts?

-Alex

(I only included the code up to where I ran into the error
message)
-------------------------------------------
% example_5_tutorial
clear; clc;

filename = 'SDSextendible.h5';
dsetname = 'ExtendibleAray';

rank = 2;

%setting dimensions
dims = [3 3];
dim1 = [3 3];
dim2 = [7 1];
dim3 = [2 2];

%MAX DIMENSIONS
maxdims = [];
chunk_dims = [2 5];

data1 = ones(3, 3);
data2 = 2*ones(7, 1);
data3 = 3*ones(4, 1);

dataspace = H5S.create_simple(2, dims, []);
%OPENING FILE
file_id = H5F.create(filename, 'H5F_ACC_TRUNC', ...
'H5P_DEFAULT', 'H5P_DEFAULT');

cparms = H5P.create('H5P_DATASET_CREATE');

H5P.set_chunk(cparms, [2 5]);

%% HERE IS WHERE THE ERROR OCCURS
dataset = H5D.create(file_id, dsetname, ...
'H5T_NATIVE_INT', dataspace, cparms);

size_dset = [3 3];

H5D.extend(dataset, size_dset);

filespace = H5D.get_space(dataset)
offset = [0 0];

H5S.select_hyperslab(filespace, 'H5S_SELECT_SET', ...
offset, [], dim1, [])

H5D.write(dataset, 'H5T_NATIVE_INT', dataspace, ...
filespace, 'H5P_DEFAULT', int32(data1))

[Alexander =]

Also, I was reading somewhere that in order to be extendible
the dataspace has to be declared to have unlimited
dimensions in the H5S.create_simple(rank, dims, maxdims)
command, but I don't know how to do this in the matlab
syntax. Any help would be greatly appreciated.

[Alexander =]

"Alexander ="
<alexander.harlie....@gmail.com> wrote in
message <g4iu1t$8n0$1...@fred.mathworks.com>...

New development: the error message I was refering too
earlier regarding the cparms being an invalid identifier
goes away when I give an arbitrarily large maxdims size(I
gave it [100 100]), so I think the error is steming from
that fact. Going back to the last post, I still am uncertain
how to define my dataspace (in Matlab) so that it has
unlimited dimensions. In C, it is :

hsize_t dims[2] = { 3, 3}; /* dataset dimensions
at the creation time */
hsize_t maxdims[2] = {H5S_UNLIMITED, H5S_UNLIMITED};
/*
* Create the data space with unlimited dimensions.
*/
dataspace = H5Screate_simple(RANK, dims, maxdims);

When I try to input the string 'H5S_UNLIMITED' as the
maxdims parameter, I get the error that "The rank parameter
does not correspond to the length of dims in the call to
H5Screate_simple"

[Alexander =]

"Alexander ="
<alexander.harlie....@gmail.com> wrote in

message <g4ivb7$q4h$1...@fred.mathworks.com>...

I figured it out, but anyone who happens to read this later.

You have to pass in the unlimited dimensions as a cell array
with the same rank as specified. For example:

rank =3;
dims = [2 3 4];
maxdims = {'H5S_UNLIMITED', 'H5S_UNLIMITED', ...
'H5S_UNLIMITED'}
dataspace = H5Screate_simple(rank, dims, maxdims)

For when the rank is large, I wrote a loop to automate it

maxdims = cell(1, rank);
for i = 1:rank
maxdims{i} = 'H5S_UNLIMITED';
end

NOTE: Those are curly braces so that an cell array rather
than a char array is created