How to overwrite to an existing file, only range of data? HDF5 can do this ?

[paul analyst]

How to overwrite to an existing file, only range of data?
In HDF5 can do this?
I have an array of zeros 10 x 10
I need an existing file owerwrite range rand (5x5), for example. Existingfile [2: 7.3: 8]
Paul

[Tim Holy]

If I understand correctly, then yes, that's possible. See the HDF5 docs.

--Tim

[paul analyst]

Of course, first I read :)
Is there about reading range array. I need to save a range of In analogy to.

A = reshape (1: 120, 15, 8)
h5write ("/ tmp / test2.h5", "mygroup2 / A", A)
data = h5read ("/ tmp / test2.h5", "mygroup2 / A" (2: 3: 15: 3: 5))

Paul

[Adrian Cuthbertson]

You can also work directly with the HDF5 file as an array object...

using HDF5

hfi=h5open("myfile.h5","w"); # create the file

close(hfi)

 A = reshape(1: 120, 15, 8);

hfi = h5open("myfile.h5","r+") # read/write access

hfi["mygroup/A"] = A

15x8 Array{Int64,2}:

  1  16  31  46  61  76   91  106

...

 14  29  44  59  74  89  104  119

 15  30  45  60  75  90  105  120

data = hfi["mygroup/A"][2:3:15,3:5]

5x3 Array{Int64,2}:

 32  47  62

 35  50  65

 38  53  68

 41  56  71

 44  59  74

 hfi["mygroup/A"][2:3,3:5]

2x3 Array{Int64,2}:

 32  47  62

 33  48  63

hfi["mygroup/A"][2:3,3:5]=[-1 -2 -3; -4 -5 -6]

2x3 Array{Int64,2}:

 -1  -2  -3

 -4  -5  -6

hfi["mygroup/A"][1:5,:]

5x8 Array{Int64,2}:

 1  16  31  46  61  76  91  106

 2  17  -1  -2  -3  77  92  107

 3  18  -4  -5  -6  78  93  108

 4  19  34  49  64  79  94  109

 5  20  35  50  65  80  95  110

foo = hfi["mygroup/A"]

HDF5 dataset: /mygroup/A (file: myfile.h5)

foo[1:5,:]

5x8 Array{Int64,2}:

 1  16  31  46  61  76  91  106

 2  17  -1  -2  -3  77  92  107

 3  18  -4  -5  -6  78  93  108

 4  19  34  49  64  79  94  109

 5  20  35  50  65  80  95  110

close(hfi)

HDF5 is awesome!

[Tim Holy]

See the part about "...incrementally save to very large ..."

--Tim

[Tim Holy]

Do note there are two additional pages of documentation in the doc/ folder.

--Tim

On Sunday, January 04, 2015 06:59:53 AM paul analyst wrote:

[paul analyst]

Thx, form me this moment

hfi["mygroup/A"] = A

was not too clear

Paul

[paul analyst]

Big Thx for big lesson, is no too "dark" now
Paul

[Paul Analyst]

Thx, Tim, 
I have solution but is 2 questions:
1. Whay must be line: fid["mygroup/A"]=rand(2) ?
2.if sum k and l > 9 Julia cant work. Is it to big size for hdf5 or for system (Win7 64 Home Premium) ?


using HDF5
hfi=h5open("bigfile.h5","w")
close(hfi)

k,l=6,3;
fid = h5open("bigfile.h5","r+")
fid["mygroup/A"]=rand(2)
g = fid["mygroup"]
dset = d_create(g, "F", datatype(Float64), dataspace(10^k,10^l))
dset[:,1] = rand(10^k)
h5read("bigfile.h5","mygroup/F",(:,1))
close(fid)
h5read("bigfile.h5","mygroup/F",(:,1:2))

Is OK
but if
k,l=6,4;

julia> close(fid)
HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
  #000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 2070 in H5Fclose(): decrementing file ID failed
    major: Object atom
    minor: Unable to close file
  #001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5I.c line 1464 in H5I_dec_app_ref(): can't decrement ID ref c
ount
    major: Object atom
    minor: Unable to decrement reference count
  #002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1847 in H5F_close(): can't close file
    major: File accessibilty
    minor: Unable to close file
  #003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 2009 in H5F_try_close(): problems closing file
    major: File accessibilty
    minor: Unable to close file
  #004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1161 in H5F_dest(): low level truncate failed
    major: File accessibilty
    minor: Write failed
  #005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5FD.c line 1895 in H5FD_truncate(): driver truncate request f
ailed
    major: Virtual File Layer
    minor: Can't update object
  #006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5FDsec2.c line 900 in H5FD_sec2_truncate(): unable to extend
file properly
    major: Low-level I/O
    minor: Seek failed
ERROR: Error closing file
 in h5f_close at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:1924


julia> h5read("bigfile.h5","mygroup/F",(:,1:2))
HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
  #000: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1594 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5F.c line 1385 in H5F_open(): unable to read superblock
    major: File accessibilty
    minor: Read failed
  #002: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Fsuper.c line 353 in H5F_super_read(): unable to load superb
lock
    major: Object cache
    minor: Unable to protect metadata
  #003: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5AC.c line 1323 in H5AC_protect(): H5C_protect() failed.
    major: Object cache
    minor: Unable to protect metadata
  #004: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5C.c line 3574 in H5C_protect(): can't load entry
    major: Object cache
    minor: Unable to load metadata into cache
  #005: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5C.c line 7954 in H5C_load_entry(): unable to load entry
    major: Object cache
    minor: Unable to load metadata into cache
  #006: /home/abuild/rpmbuild/BUILD/hdf5-1.8.13/src/H5Fsuper_cache.c line 471 in H5F_sblock_load(): truncated file
: eof = 8002864, sblock->base_addr = 0, stored_eoa = 80000002864
    major: File accessibilty
    minor: File has been truncated
ERROR: Error opening file bigfile.h5
 in h5f_open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:2023
 in h5open at C:\Users\SAMSUNG2\.julia\v0.3\HDF5\src\plain.jl:554



Paul


W dniu 2015-01-04 o 18:37, Tim Holy pisze:

[Tim Holy]

On Monday, January 05, 2015 03:17:12 PM Paul Analyst wrote:
> Thx, Tim,
> I have solution but is 2 questions:

> 1. Whay must be line: /fid["mygroup/A"]=rand(2)/ ?

That line just means 'create a variable called "A" inside a group called
"mygroup", and assign it a value of rand(2)'. If you don't need that variable,
you don't need that line. You also don't have to create a group called
"mygroup", if you prefer you can store everything in the top level of the file.

> 2.if sum k and l > 9 Julia cant work. Is it to big size for hdf5 or for
> system (Win7 64 Home Premium) ?

Not sure. It works for me (Kubuntu Linux 14.04).

--Tim

[Paul Analyst]

Tim , thx for hints, but do not work without this line  #fid["mygroup/A"]=rand(2)
becouse I vave not "g" , and nothing to declare in this line dset = d_create(g, "F", datatype(Float64), dataspace(10,10))

dset = d_create( "F", datatype(Float64), dataspace(10,10))
do not work ...

Paul




using HDF5
hfi=h5open("test.h5","w");close(hfi)
fid = h5open("test.h5","r+")
#fid["mygroup/A"]=rand(2)
g = fid["mygroup"]
dset = d_create(g, "F", datatype(Float64), dataspace(10,10))
dset[:,1] = rand(10)
h5read("test.h5","mygroup/F",(:,1))
close(fid)


W dniu 2015-01-05 o 15:39, Tim Holy pisze:

[Tim Holy]

On Monday, January 05, 2015 09:52:28 PM Paul Analyst wrote:
> dset = d_create( "F", datatype(Float64), dataspace(10,10))/

You're missing the `fid`. You have to tell it where (which file, or which group)
you want to create that dataset.

--Tim

[paul analyst]

> 2.if sum k and l > 9 Julia cant work. Is it to big size for hdf5 or for

> system (Win7 64 Home Premium) ?

Not sure. It works for me (Kubuntu Linux 14.04).

I checked on Ubuntu :

If sum of "k" or/and  "l" is more then 7 I have problem with reading vectors (cols)

 

samsung2@samsung2:~$ julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.3 (2014-10-21 20:18 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org release
|__/                   |  i686-linux-gnu

julia> using HDF5

julia> hfi=h5open("F_big.h5","w");close(hfi)

julia> k,l=8,8    ;

julia> fid = h5open("F_big.h5","r+")
HDF5 data file: F_big.h5

julia> fid["mygroup/A"]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
 0.318459
 0.258055

julia> g = fid["mygroup"]
HDF5 group: /mygroup (file: F_big.h5)

julia> dset = d_create(g, "F", datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)

julia> h5read("F_big.h5","mygroup/F",(:,1))
100000000x1 Array{Float64,2}:
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 ⋮ 
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0

julia> close(fid)

julia> hfi=h5open("F_big.h5","w");close(hfi)

julia> k,l=9,8    ;

julia> fid = h5open("F_big.h5","r+")
HDF5 data file: F_big.h5

julia> fid["mygroup/A"]=rand(2)#niepotrzebny
2-element Array{Float64,1}:
 0.129214
 0.4785 

julia> g = fid["mygroup"]
HDF5 group: /mygroup (file: F_big.h5)

julia> dset = d_create(g, "F", datatype(Float64), dataspace(10^k,1*10^l))
HDF5 dataset: /mygroup/F (file: F_big.h5)

julia> h5read("F_big.h5","mygroup/F",(:,1))
ERROR: invalid Array size
 in Array at base.jl:223
 in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
 in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
 in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
 in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602

julia> close(fid)

julia> h5read("F_big.h5","mygroup/F",(1:2,:))
2x100000000 Array{Float64,2}:
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0  0.0

julia> h5read("F_big.h5","mygroup/F",(:,1))
ERROR: invalid Array size
 in Array at base.jl:223
 in _getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1557
 in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1550
 in getindex at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:1620
 in h5read at /home/samsung2/.julia/v0.3/HDF5/src/plain.jl:602

julia>

Paul

 

[paul analyst]

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.3 (2014-10-21 20:18 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org release
|__/                   |  i686-linux-gnu

....

dset[1:10^8,1]=rand(10^8);

julia> size(dset)
(1000000000,10000000)

julia> dset[1:10^9,1]=rand(10^9);
ERROR: invalid Array size
 in rand at random.jl:123

julia>

Julia now is using 32% of ram, 2.5 GB of 7.8 GB aviable..
Ubuntu 14.10
Why ?  free size of HD is now 30GB
Paul

[Tim Holy]

Did you try calculating how much memory the right hand side of that assignment
requires? (Remember that sizeof(Float64) == 8).

--Tim

[Paul Analyst]

one vector 10^9 it is about 16 GB,
Paul

W dniu 2015-01-07 o 13:13, Tim Holy pisze:

[Tim Holy]

On Wednesday, January 07, 2015 01:42:24 PM Paul Analyst wrote:
> one vector 10^9 it is about 16 GB,

...and that explains it. (Compare against your RAM size.)

--Tim