[ANN] bloscpack v0.1.0-rc1

[Valentin Haenel]

Hi,

for the last couple of weeks I have been working on a command-line
interface to Blosc. Today, I am pre-releasing v0.1.0 as a release
candidate:

https://github.com/esc/bloscpack/tree/v0.1.0-rc1

Maybe someone out their will find it useful---If you do try it out I
would be very, very happy to hear from you! Feedback is welcome!

Valentin Haenel

[Francesc Alted]

Hey this looks great! I like your implementation. A couple of suggestions:

- As you say in the README, bloscpack has to load in memory all the file
for operation. This can actually defeat the utility of the compressor
in large file scenarios. It would be fantastic if bloscpack could
compress/decompress in small blocks.

- The default benchmark creates a very large file (1.5 GB), and that can
be too much for many machines out there. I'd suggest to reduce it to
something more sensible (128 MB, 256 MB). In the future, when the block
compression would be implement you could raise this value again.

Cheers!

--
Francesc Alted

[Valentin Haenel]

Hi,

* Francesc Alted <fal...@gmail.com> [2012-05-29]:

> On 5/27/12 7:31 PM, Valentin Haenel wrote:
> >Hi,
> >
> >for the last couple of weeks I have been working on a command-line
> >interface to Blosc. Today, I am pre-releasing v0.1.0 as a release
> >candidate:
> >
> >https://github.com/esc/bloscpack/tree/v0.1.0-rc1
> >
> >Maybe someone out their will find it useful---If you do try it out I
> >would be very, very happy to hear from you! Feedback is welcome!
>
> Hey this looks great! I like your implementation. A couple of suggestions:

Thanks! I am glad you like it!

> - As you say in the README, bloscpack has to load in memory all the
> file for operation. This can actually defeat the utility of the
> compressor in large file scenarios. It would be fantastic if
> bloscpack could compress/decompress in small blocks.

You can try setting --nchunks or --chunk-size (see ./blpk c -h).
Although --chunk-size currently only works in bytes and maybe allowing
for a human readable spec (MB, G etc..) would be nice to have. If you
use either chunks will be read, compressed and written chunk-wise.

I just chose the maximum chunk size to be the maximum that Blosc
can handle (2G). I thought this is maybe the best choice for an on-disk
compressed buffer, since the larger the buffer, the better the
compression (or did I miss something?) As for the compression level, it
is 7 by default. I chose this because the synthetic benchmarks seemed to
indicate that there isn't much to be gained by setting it higher and it
might actually be faster. I.e. gives good compression ration speed
trade-off.

> - The default benchmark creates a very large file (1.5 GB), and that
> can be too much for many machines out there. I'd suggest to reduce
> it to something more sensible (128 MB, 256 MB). In the future, when
> the block compression would be implement you could raise this value
> again.

OK. How about this:

https://github.com/esc/bloscpack/commit/dc89b4c
https://github.com/esc/bloscpack/commit/3f8ad42

I changed the way the array is created too, to be a bit nicer on mem.

Also, the memory problem is exaggerated by:

https://github.com/FrancescAlted/python-blosc/blob/master/blosc/blosc_extension.c#L94

So for a 2G file of non-compressible data, for example coming from
random number generator, we get 2G input buffer, 2G buffer or compressed
data and then another 2G for the results string. ;) I was actually able
to bring my work machine with 6G to its knees! ;) I have perhaps a
solution using bytearray, which you can pre-allocate and then shrink
(and maybe also memoryview) but its still in progress and perhaps the
topic of a later discussion.

Thanks for the feedback!

V-

[Francesc Alted]

On 5/29/12 9:07 PM, Valentin Haenel wrote:
> Hi,
>
> * Francesc Alted<fal...@gmail.com> [2012-05-29]:
>> On 5/27/12 7:31 PM, Valentin Haenel wrote:
>>> Hi,
>>>
>>> for the last couple of weeks I have been working on a command-line
>>> interface to Blosc. Today, I am pre-releasing v0.1.0 as a release
>>> candidate:
>>>
>>> https://github.com/esc/bloscpack/tree/v0.1.0-rc1
>>>
>>> Maybe someone out their will find it useful---If you do try it out I
>>> would be very, very happy to hear from you! Feedback is welcome!
>> Hey this looks great! I like your implementation. A couple of suggestions:
> Thanks! I am glad you like it!
>
>> - As you say in the README, bloscpack has to load in memory all the
>> file for operation. This can actually defeat the utility of the
>> compressor in large file scenarios. It would be fantastic if
>> bloscpack could compress/decompress in small blocks.
> You can try setting --nchunks or --chunk-size (see ./blpk c -h).
> Although --chunk-size currently only works in bytes and maybe allowing
> for a human readable spec (MB, G etc..) would be nice to have. If you
> use either chunks will be read, compressed and written chunk-wise.

Yes, specifying the chunksize would help. Also using human readable
specs would be nice.

>
> I just chose the maximum chunk size to be the maximum that Blosc
> can handle (2G). I thought this is maybe the best choice for an on-disk
> compressed buffer, since the larger the buffer, the better the
> compression (or did I miss something?)

Nope. Blosc always tries to compress small blocks, typically in the
order of magnitude of level 1 cache for small compression levels, and in
level 2 for larger ones. So even if you pass very large chunks of
memory, Blosc will split it in small blocks for compression.

I'd say that using a default chunksize of 8 MB would be a far more
sensible figure that 1) allows for maximum compression, 2) consumes far
less memory and 3) allows to compress files up to 2**32 * 2**23 = 2 **
45 bytes = 32 TB which is large enough for a default.

> As for the compression level, it
> is 7 by default. I chose this because the synthetic benchmarks seemed to
> indicate that there isn't much to be gained by setting it higher and it
> might actually be faster. I.e. gives good compression ration speed
> trade-off.
>
>> - The default benchmark creates a very large file (1.5 GB), and that
>> can be too much for many machines out there. I'd suggest to reduce
>> it to something more sensible (128 MB, 256 MB). In the future, when
>> the block compression would be implement you could raise this value
>> again.
> OK. How about this:
>
> https://github.com/esc/bloscpack/commit/dc89b4c
> https://github.com/esc/bloscpack/commit/3f8ad42

Yes, much better. However, your benchmark updates to the README.md say
that compression is actually affected by using a block size as large as
128 MB (contrarily to what I was saying above). Hmm, I need to think
about what is happening here.

BTW, the output for the benchmark on my Mac OSX box is:

create the test data
testfile is:
-n enlarge the testfile
-n .
-n .
-n .
-n .
-n .
-n .
-n .
-n .
-n .
done.
testfile is:
do compression with bloscpack, chunk-size: 128MB
real 18.43
user 9.79
sys 5.32
testfile.blp is:
do compression with gzip
real 185.21
user 176.09
sys 3.06
testfile.gz is:

Apparently the -n is not support by the /bin/sh. From the manpage manual:

"""
Some shells may provide a builtin echo command which is similar or
identical to this utility. Most notably, the builtin echo in sh(1) does
not accept the -n option. Consult the builtin(1) manual page.
"""

Also, the `cut -f5 -d' '` does not seem to work as intended on Mac OSX.

>
> I changed the way the array is created too, to be a bit nicer on mem.
>
> Also, the memory problem is exaggerated by:
>
> https://github.com/FrancescAlted/python-blosc/blob/master/blosc/blosc_extension.c#L94
>
> So for a 2G file of non-compressible data, for example coming from
> random number generator, we get 2G input buffer, 2G buffer or compressed
> data and then another 2G for the results string. ;) I was actually able
> to bring my work machine with 6G to its knees! ;) I have perhaps a
> solution using bytearray, which you can pre-allocate and then shrink
> (and maybe also memoryview) but its still in progress and perhaps the
> topic of a later discussion.

Yes, this is an ugly inefficiency in the Python wrapper that I'd be glad
to discuss on how to solve it.

--
Francesc Alted

[Valentin Haenel]

Hi,

* Francesc Alted <fal...@gmail.com> [2012-05-30]:

> On 5/29/12 9:07 PM, Valentin Haenel wrote:
> >* Francesc Alted<fal...@gmail.com> [2012-05-29]:
> >>On 5/27/12 7:31 PM, Valentin Haenel wrote:
> >>>for the last couple of weeks I have been working on a command-line
> >>>interface to Blosc. Today, I am pre-releasing v0.1.0 as a release
> >>>candidate:
> >>>
> >>>https://github.com/esc/bloscpack/tree/v0.1.0-rc1
> >>>
> >>>Maybe someone out their will find it useful---If you do try it out I
> >>>would be very, very happy to hear from you! Feedback is welcome!
> >>Hey this looks great! I like your implementation. A couple of suggestions:
> >Thanks! I am glad you like it!
> >
> >>- As you say in the README, bloscpack has to load in memory all the
> >>file for operation. This can actually defeat the utility of the
> >>compressor in large file scenarios. It would be fantastic if
> >>bloscpack could compress/decompress in small blocks.
> >You can try setting --nchunks or --chunk-size (see ./blpk c -h).
> >Although --chunk-size currently only works in bytes and maybe allowing
> >for a human readable spec (MB, G etc..) would be nice to have. If you
> >use either chunks will be read, compressed and written chunk-wise.
>
> Yes, specifying the chunksize would help. Also using human readable
> specs would be nice.

I have added this as a TODO.

> >I just chose the maximum chunk size to be the maximum that Blosc
> >can handle (2G). I thought this is maybe the best choice for an on-disk
> >compressed buffer, since the larger the buffer, the better the
> >compression (or did I miss something?)
>
> Nope. Blosc always tries to compress small blocks, typically in the
> order of magnitude of level 1 cache for small compression levels,
> and in level 2 for larger ones. So even if you pass very large
> chunks of memory, Blosc will split it in small blocks for
> compression.

Ah yes, the block-size. But if we have many small chunks, each chunk
does have a 16 byte Blosc header, right. So using less chunks means less
overhead?

> I'd say that using a default chunksize of 8 MB would be a far more
> sensible figure that 1) allows for maximum compression, 2) consumes
> far less memory and 3) allows to compress files up to 2**32 * 2**23
> = 2 ** 45 bytes = 32 TB which is large enough for a default.

Of course the last chunk might then be 15MB in the worst case, but I
think it is OK.

> > As for the compression level, it
> >is 7 by default. I chose this because the synthetic benchmarks seemed to
> >indicate that there isn't much to be gained by setting it higher and it
> >might actually be faster. I.e. gives good compression ration speed
> >trade-off.
> >
> >>- The default benchmark creates a very large file (1.5 GB), and that
> >>can be too much for many machines out there. I'd suggest to reduce
> >>it to something more sensible (128 MB, 256 MB). In the future, when
> >>the block compression would be implement you could raise this value
> >>again.
> >OK. How about this:
> >
> >https://github.com/esc/bloscpack/commit/dc89b4c
> >https://github.com/esc/bloscpack/commit/3f8ad42
>
> Yes, much better. However, your benchmark updates to the README.md
> say that compression is actually affected by using a block size as
> large as 128 MB (contrarily to what I was saying above). Hmm, I
> need to think about what is happening here.

The diff makes it look like, its slower with smaller chunk-size.
However, I think it might be, because I am not using the same array in
the new benchmark. The old one creates a single array, the new one
creates a small array and then uses cat to enlarge it.

When run the with:

a = numpy.linspace(0, 100, 2e8)

I get:

zsh� /usr/bin/time -p ./blpk --force compress --chunk-size 134217728
testfile
real 7.49
user 5.91
sys 1.60
zsh� /usr/bin/time -p ./blpk --force compress testfile
real 9.26
user 7.16
sys 1.75

So actually using a smaller chunk-size makes this particular example go
faster.

I'll perhaps setup a benchmark to clock the effect of the chunk-size on
time? Using log2 spaced chunk-size? What kind of box? NUMA, non-NUMA?
Should I also check how this correlates with compression level? Maybe
this can give a hint for an optimal chunk-size.

Perhaps we could solve this by using a /bin/bash she-bang. But srsly,
I'm not going to code around the deficiencies of some random operating
system ;p

What does 'ls -lah' return? Maybe it is that? I remember the ls on OSX
isn't gnu-ls?

best

V-

[Francesc Alted]

On 5/31/12 3:02 PM, Valentin Haenel wrote:
>
>>> I just chose the maximum chunk size to be the maximum that Blosc
>>> can handle (2G). I thought this is maybe the best choice for an on-disk
>>> compressed buffer, since the larger the buffer, the better the
>>> compression (or did I miss something?)
>> Nope. Blosc always tries to compress small blocks, typically in the
>> order of magnitude of level 1 cache for small compression levels,
>> and in level 2 for larger ones. So even if you pass very large
>> chunks of memory, Blosc will split it in small blocks for
>> compression.
> Ah yes, the block-size. But if we have many small chunks, each chunk
> does have a 16 byte Blosc header, right. So using less chunks means less
> overhead?

Well, suppose that a blocksize is 16 KB (which is rather low), then 16
bytes is a mere .001% overhead. So this overhead should be negligible
in practice.

>> I'd say that using a default chunksize of 8 MB would be a far more
>> sensible figure that 1) allows for maximum compression, 2) consumes
>> far less memory and 3) allows to compress files up to 2**32 * 2**23
>> = 2 ** 45 bytes = 32 TB which is large enough for a default.
> Of course the last chunk might then be 15MB in the worst case, but I
> think it is OK.

Well, what you can do is set the last chunk to less of the default
chunksize (i.e. contain only the remainder, not a whole block plus the
remainder).

>>> As for the compression level, it
>>> is 7 by default. I chose this because the synthetic benchmarks seemed to
>>> indicate that there isn't much to be gained by setting it higher and it
>>> might actually be faster. I.e. gives good compression ration speed
>>> trade-off.
>>>
>>>> - The default benchmark creates a very large file (1.5 GB), and that
>>>> can be too much for many machines out there. I'd suggest to reduce
>>>> it to something more sensible (128 MB, 256 MB). In the future, when
>>>> the block compression would be implement you could raise this value
>>>> again.
>>> OK. How about this:
>>>
>>> https://github.com/esc/bloscpack/commit/dc89b4c
>>> https://github.com/esc/bloscpack/commit/3f8ad42
>> Yes, much better. However, your benchmark updates to the README.md
>> say that compression is actually affected by using a block size as
>> large as 128 MB (contrarily to what I was saying above). Hmm, I
>> need to think about what is happening here.
> The diff makes it look like, its slower with smaller chunk-size.
> However, I think it might be, because I am not using the same array in
> the new benchmark. The old one creates a single array, the new one
> creates a small array and then uses cat to enlarge it.
>

Ah, so this should be the reason. I'd say that, using the same data,
there should be no noticeable difference between using a 8 MB (even
using something as low as 1 MB) chunksize than a 2 GB one.

> When run the with:
>
> a = numpy.linspace(0, 100, 2e8)
>
> I get:
>
> zsh� /usr/bin/time -p ./blpk --force compress --chunk-size 134217728
> testfile
> real 7.49
> user 5.91
> sys 1.60
> zsh� /usr/bin/time -p ./blpk --force compress testfile
> real 9.26
> user 7.16
> sys 1.75
>
> So actually using a smaller chunk-size makes this particular example go
> faster.
>
> I'll perhaps setup a benchmark to clock the effect of the chunk-size on
> time? Using log2 spaced chunk-size? What kind of box? NUMA, non-NUMA?
> Should I also check how this correlates with compression level? Maybe
> this can give a hint for an optimal chunk-size.

You can do that, but I'd bet that going with something similar to 8 MB
would be close to an optimal compression ratio (and also
compression/decompression times). But I might be wrong ;)

:)

>
> What does 'ls -lah' return? Maybe it is that? I remember the ls on OSX
> isn't gnu-ls?

Here it is an example of the output on Mac OSX:

(env1)Francescs-MacBook-Air:bloscpack faltet$ ls -lah testfile*
-rw-r--r-- 2 faltet staff 1.5G May 30 09:48 testfile
-rw-r--r-- 2 faltet staff 638M May 30 09:49 testfile.blp
-rw-r--r-- 2 faltet staff 959M May 30 09:52 testfile.gz

and `cut`here founds the desired info in field 9 instead of 5:

(env1)Francescs-MacBook-Air:bloscpack faltet$ ls -lah testfile* | cut -f
9 -d' '
1.5G
638M
959M

My experience on this says that it would better to rewrite the benchmark
in pure Python. It would be far more portable.

--
Francesc Alted

[Valentin Haenel]

Hi,

* Valentin Haenel <valenti...@gmx.de> [2012-05-31]:

> * Francesc Alted <fal...@gmail.com> [2012-05-30]:
> > On 5/29/12 9:07 PM, Valentin Haenel wrote:
> > >* Francesc Alted<fal...@gmail.com> [2012-05-29]:

> > >>- As you say in the README, bloscpack has to load in memory all the
> > >>file for operation. This can actually defeat the utility of the
> > >>compressor in large file scenarios. It would be fantastic if
> > >>bloscpack could compress/decompress in small blocks.
> > >You can try setting --nchunks or --chunk-size (see ./blpk c -h).
> > >Although --chunk-size currently only works in bytes and maybe allowing
> > >for a human readable spec (MB, G etc..) would be nice to have. If you
> > >use either chunks will be read, compressed and written chunk-wise.
> >
> > Yes, specifying the chunksize would help. Also using human readable
> > specs would be nice.
>
> I have added this as a TODO.

... and implemeneted it:

https://github.com/esc/bloscpack/compare/7623654c...b8ff1e8666

V-

[Valentin Haenel]

Hi,

* Francesc Alted <fal...@gmail.com> [2012-05-31]:

> On 5/31/12 3:02 PM, Valentin Haenel wrote:
> >>>I just chose the maximum chunk size to be the maximum that Blosc
> >>>can handle (2G). I thought this is maybe the best choice for an on-disk
> >>>compressed buffer, since the larger the buffer, the better the
> >>>compression (or did I miss something?)
> >>Nope. Blosc always tries to compress small blocks, typically in the
> >>order of magnitude of level 1 cache for small compression levels,
> >>and in level 2 for larger ones. So even if you pass very large
> >>chunks of memory, Blosc will split it in small blocks for
> >>compression.
> >Ah yes, the block-size. But if we have many small chunks, each chunk
> >does have a 16 byte Blosc header, right. So using less chunks means less
> >overhead?
>
> Well, suppose that a blocksize is 16 KB (which is rather low), then
> 16 bytes is a mere .001% overhead. So this overhead should be
> negligible in practice.

Yeah, I'm sold.

> >>I'd say that using a default chunksize of 8 MB would be a far more
> >>sensible figure that 1) allows for maximum compression, 2) consumes
> >>far less memory and 3) allows to compress files up to 2**32 * 2**23
> >>= 2 ** 45 bytes = 32 TB which is large enough for a default.
> >Of course the last chunk might then be 15MB in the worst case, but I
> >think it is OK.
>
> Well, what you can do is set the last chunk to less of the default
> chunksize (i.e. contain only the remainder, not a whole block plus
> the remainder).

True. I decided to stuff the remainder into the last chunk because I was
anticipating evenly distributed larger chunks and thought it would make
more sense. But the more I think about it, the more likely I am to
change that and make a separate chunk for the leftovers. That way, one
could also guarantee a maximum chunksize.

> >I'll perhaps setup a benchmark to clock the effect of the chunk-size on
> >time? Using log2 spaced chunk-size? What kind of box? NUMA, non-NUMA?
> >Should I also check how this correlates with compression level? Maybe
> >this can give a hint for an optimal chunk-size.
>
> You can do that, but I'd bet that going with something similar to 8
> MB would be close to an optimal compression ratio (and also
> compression/decompression times). But I might be wrong ;)

So, here are the results from a quick benchmark.

chunk_size comp-time
4096 35.779623
5792 25.519103
8192 19.290844
11585 16.565704
16384 12.189406
23170 13.844878
32768 13.903858
46340 13.105773
65536 12.429925
92681 12.222897
131072 11.335518
185363 11.543702
262144 9.532881
370727 8.536612
524288 7.304409
741455 6.884099
1048576 6.542390
1482910 6.385494
2097152 6.573333
2965820 6.328400
4194304 6.245545
5931641 6.129058
8388608 6.103699
11863283 6.072463
16777216 6.168805
23726566 6.083282
33554432 6.303940
47453132 6.184574
67108864 6.360049
94906265 6.391919
134217728 6.370138
189812531 6.474085
268435456 6.365690
379625062 6.634286
536870912 6.540020
759250124 7.082066
1073741824 6.699226
1518500249 7.369446

So, as soon as we get to about half a meg (524288) it doesn't make much
difference anymore how big the chunks are. But we need to bear in mind,
that file caching effects might play a significant role here. However
there is something of a minimum at around 8MB (bit more in fact).
(Surprise surprise) But one would need to do some statistics to check
the significance since these observations are quite noisy. Anyway, its
enough evidence for me to conclude that 8MB is a very good value for
default chunk-size. ;) Thanks!

> My experience on this says that it would better to rewrite the
> benchmark in pure Python. It would be far more portable.

+1

V-

[Francesc Alted]

On 6/5/12 4:56 PM, Valentin Haenel wrote:
> Hi,
>
> * Francesc Alted<fal...@gmail.com> [2012-05-31]:
>> On 5/31/12 3:02 PM, Valentin Haenel wrote:
>>>>> I just chose the maximum chunk size to be the maximum that Blosc
>>>>> can handle (2G). I thought this is maybe the best choice for an on-disk
>>>>> compressed buffer, since the larger the buffer, the better the
>>>>> compression (or did I miss something?)
>>>> Nope. Blosc always tries to compress small blocks, typically in the
>>>> order of magnitude of level 1 cache for small compression levels,
>>>> and in level 2 for larger ones. So even if you pass very large
>>>> chunks of memory, Blosc will split it in small blocks for
>>>> compression.
>>> Ah yes, the block-size. But if we have many small chunks, each chunk
>>> does have a 16 byte Blosc header, right. So using less chunks means less
>>> overhead?
>> Well, suppose that a blocksize is 16 KB (which is rather low), then
>> 16 bytes is a mere .001% overhead. So this overhead should be
>> negligible in practice.
> Yeah, I'm sold.

Actually, I was wrong, it is 0.1%, not a 0.001%, but things do not
change that much either :)

>
>>>> I'd say that using a default chunksize of 8 MB would be a far more
>>>> sensible figure that 1) allows for maximum compression, 2) consumes
>>>> far less memory and 3) allows to compress files up to 2**32 * 2**23
>>>> = 2 ** 45 bytes = 32 TB which is large enough for a default.
>>> Of course the last chunk might then be 15MB in the worst case, but I
>>> think it is OK.
>> Well, what you can do is set the last chunk to less of the default
>> chunksize (i.e. contain only the remainder, not a whole block plus
>> the remainder).
> True. I decided to stuff the remainder into the last chunk because I was
> anticipating evenly distributed larger chunks and thought it would make
> more sense. But the more I think about it, the more likely I am to
> change that and make a separate chunk for the leftovers. That way, one
> could also guarantee a maximum chunksize.

Good. FYI, this is how Blosc works too, so at least your utility would
be consistent with this convention :)

I wouldn't go that fast. It would be nice to add a third column with
the compression ratio too. I don't expect big surprises, but it would
be reassuring that we don't see an unexpected drop in the compression
ratio either.

--
Francesc Alted

[Valentin Haenel]

* Francesc Alted <fal...@gmail.com> [2012-06-05]:

I just clocked it:

chunk_size comp-time decomp-time
524288 7.962348 14.089477
741455 6.580518 13.915469
1048576 6.376429 13.751035
1482910 6.116840 13.428622
2097152 5.983984 15.319350
2965820 5.909490 13.521392
4194304 5.771475 13.386133
5931641 6.017159 13.420380
8388608 5.970614 13.510643
11863283 5.988403 13.267338
16777216 6.218334 13.525590
23726566 6.232693 14.143784
33554432 6.333955 13.990052
47453132 6.637157 14.071174
67108864 6.330848 14.067347
94906265 6.498304 14.056434
134217728 6.562078 13.967526
189812531 6.370958 14.129260
268435456 6.563388 13.831072
379625062 6.539124 14.173098
536870912 6.349423 14.091259
759250124 6.294907 13.712528
1073741824 6.584789 13.856677
1518500249 6.583735 13.769070
2147483647 6.587658 13.695593

Looks good!

V-

[Francesc Alted]

Yeah, interesting. A curious thing is that decompressing takes 2x more
than compressing (probably an effect of writing/reading from OS I/O
cache). But I was referring to compression ratio, that is: (original
filesize) / (compressed filesize).

--
Francesc Alted

[Valentin Haenel]

* Francesc Alted <fal...@gmail.com> [2012-06-06]:

Oh, my bad. I guess writing to disk is more expensive than reading from
it. I'll clock the ratio next.

V-

[Valentin Haenel]

* Valentin Haenel <valenti...@gmx.de> [2012-06-06]:

I guess one could work on making the output nicer, but anyway:

chunk_size comp-time decomp-time ratio
524288 6.500661 15.999133 0.385795
741455 6.806602 13.886007 0.401923
1048576 6.319533 13.620045 0.385780
1482910 6.058260 13.524332 0.387501
2097152 6.100803 13.267586 0.385772
2965820 6.178004 13.089901 0.389236
4194304 6.083805 13.297962 0.385768
5931641 6.079701 13.249913 0.386189
8388608 6.090493 13.235450 0.385767
11863283 6.027715 13.240447 0.385980
16777216 6.176546 14.042878 0.385766
23726566 6.233473 13.994969 0.385842
33554432 6.513346 13.990890 0.385765
47453132 6.556554 13.992711 0.385812
67108864 6.548876 13.925211 0.385765
94906265 6.576076 13.807707 0.385809
134217728 6.602152 14.022800 0.385765
189812531 6.501596 14.073467 0.385788
268435456 6.606041 13.789218 0.385765
379625062 6.555183 14.011529 0.385773
536870912 6.536188 13.910750 0.385765
759250124 6.536242 13.929346 0.385773
1073741824 6.614322 14.153648 0.385765
1518500249 6.595979 13.728878 0.385769
2147483647 6.426127 14.094006 0.385765

:-D

V-

[Francesc Alted]

Ahhh, very, very good! Now I think you can safely choose a chunksize of
something between 2 MB and 8 MB. Provided that you are probably doing
the benchmarks on a machine with a big cache, I'd stay on the low
figures for this range, just to better cover less capable CPUs.

Thanks!

--
Francesc Alted

[Valentin Haenel]

Hi,

* Francesc Alted <fal...@gmail.com> [2012-05-31]:

> On 5/31/12 3:02 PM, Valentin Haenel wrote:
> >>>>- The default benchmark creates a very large file (1.5 GB), and that
> >>>>can be too much for many machines out there. I'd suggest to reduce
> >>>>it to something more sensible (128 MB, 256 MB). In the future, when
> >>>>the block compression would be implement you could raise this value
> >>>>again.
> >>>OK. How about this:
> >>>
> >>>https://github.com/esc/bloscpack/commit/dc89b4c
> >>>https://github.com/esc/bloscpack/commit/3f8ad42
> >>Yes, much better. However, your benchmark updates to the README.md
> >>say that compression is actually affected by using a block size as
> >>large as 128 MB (contrarily to what I was saying above). Hmm, I
> >>need to think about what is happening here.
> >The diff makes it look like, its slower with smaller chunk-size.
> >However, I think it might be, because I am not using the same array in
> >the new benchmark. The old one creates a single array, the new one
> >creates a small array and then uses cat to enlarge it.
> >
>
> Ah, so this should be the reason. I'd say that, using the same
> data, there should be no noticeable difference between using a 8 MB
> (even using something as low as 1 MB) chunksize than a 2 GB one.

Actually there was (also) a bug in the way the number of chunks is
handled when using the chunk-size argument. So this may have been the
reason. Anyway, it is fixed and tested for now.

V-