blosc error

[NizamovShawkat]

Hi, all!

I tried to use a new blosc compression method for my task, but found
that it does not fit my task. I have a 2mb blocks of 12bit data packed
as 16bit ints (iotw, at least 25% compressible) but blosc could not
compress it at all, no matter what was the compression level set.
Other compression methods work but are much slower than I need.

During these experiments I found that at some particular case, when I
tried to write blocks of data not as a a string but as a real array,
blosc fails. I have a 64bit ubuntu11.04 system, tables 2.2.1. The
source for the test code is below. If one substitutes blosc with e.g.
lzo/bzip2 it works.

Output in the case of blosc is:

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread
139810921076512:
#000: ../../../src/H5Dio.c line 266 in H5Dwrite(): can't write data
major: Dataset
minor: Write failed
#001: ../../../src/H5Dio.c line 578 in H5D_write(): can't write data
major: Dataset
minor: Write failed
#002: ../../../src/H5Dchunk.c line 1870 in H5D_chunk_write(): unable
to read raw data chunk
major: Low-level I/O
minor: Read failed
#003: ../../../src/H5Dchunk.c line 2742 in H5D_chunk_lock(): data
pipeline read failed
major: Data filters
minor: Filter operation failed
#004: ../../../src/H5Z.c line 1017 in H5Z_pipeline(): filter
returned failure during read
major: Data filters
minor: Read failed
#005: blosc/blosc_filter.c line 232 in blosc_filter(): Blosc
decompression error
major: Data filters
minor: Callback failed
Traceback (most recent call last):
File "/home/shawkat/Workspace/PyTables/src/test33.py", line 12, in
<module>
ea.append(arr2)
File "/usr/local/lib/python2.7/dist-packages/tables/earray.py", line
226, in append
self._append(nparr)
File "hdf5Extension.pyx", line 1027, in
tables.hdf5Extension.Array._append (tables/hdf5Extension.c:8976)
tables.exceptions.HDF5ExtError: Problems appending the elements
Closing remaining open files: /tmp/test_earray.h5... done





Test code:


import tables
import numpy

h5ft = tables.openFile('/tmp/test_earray.h5','w')

filters = tables.Filters(complevel = 1, complib = "blosc", fletcher32
= True)
ea = h5ft.createEArray(h5ft.root, "foo", tables.Int16Col(),
(0,1024,1024), "earray test", filters = filters, expectedrows =
1000000)

for ii in range(100):
arr = numpy.random.randint(0,4096, size=(1024,1024))
arr2 = numpy.asarray(arr, numpy.int16)[numpy.newaxis,:]
ea.append(arr2)
ea.flush()

h5ft.close()

[Francesc Alted]

A Sunday 17 April 2011 12:51:48 NizamovShawkat escrigué:

> Hi, all!
>
> I tried to use a new blosc compression method for my task, but found
> that it does not fit my task. I have a 2mb blocks of 12bit data
> packed as 16bit ints (iotw, at least 25% compressible) but blosc
> could not compress it at all, no matter what was the compression
> level set. Other compression methods work but are much slower than I
> need.
>
> During these experiments I found that at some particular case, when I
> tried to write blocks of data not as a a string but as a real array,
> blosc fails. I have a 64bit ubuntu11.04 system, tables 2.2.1. The
> source for the test code is below. If one substitutes blosc with e.g.
> lzo/bzip2 it works.
>
> Output in the case of blosc is:

[clip]

The problem seems that blosc filter have problems in combination with
the fletcher32 filter. Just disable it and the script will work. I'll
file a ticket on this, but is there any reason on why you are using
fletcher32? Just curious.

Cheers,

--
Francesc Alted

[Francesc Alted]

A Sunday 17 April 2011 13:12:18 Francesc Alted escrigué:

> The problem seems that blosc filter have problems in combination with
> the fletcher32 filter. Just disable it and the script will work.
> I'll file a ticket on this, but is there any reason on why you are
> using fletcher32? Just curious.

FYI:

http://pytables.org/trac/ticket/339

--
Francesc Alted

[NizamovShawkat]

>
> The problem seems that blosc filter have problems in combination with
> the fletcher32 filter.  Just disable it and the script will work.  I'll
> file a ticket on this, but is there any reason on why you are using
> fletcher32?  Just curious.
>

Thank you very much! There is no specific reason to use fletcher32 on
my side. I just took it from pytables example for filters (page 118 in
2.2.1 pdf manual). It uses exactly this line:

filters = Filters(complevel = 1, complib = "blosc", fletcher32= True)

Best regards,
Shavkat Nizamov