Fixing valgrid errors by setting INIT_HTAB=1

[Arnaldo Leon]

Hi all,

It turns out I managed to fix a very annoying Valgrind error by setting  INIT_HTAB=1 in lzfP.h:91. It seems that the author of the LZF library wanted to squeeze a little more performance out of the compression speed, but he did so at the cost of reading un-initialized memory. By setting this flag, the htab variable is set to zero before being used. Otherwise you'd get tons of "Conditional jump or move depends on uninitialised value(s)".

Another side benefit is that now the compressed results are identical between runs. Before it was the case that h5diff would report differences, but they were only due to differences in the compressed output, not in the original source data.

This is a one-line change in lzfP.h.. Should I just create a pull request for this?

Kind Regards

-aml

[Thomas Caswell]

I have some concerns, both about diverging from the upstream library and about the exact change here.

If the output data is being changed by zeroing parts of memory is it because there is some entropy being pulled from those bits that is doing useful work or is it noise that is in a region of memory that is there for padding?

Tom 

--
You received this message because you are subscribed to the Google Groups "h5py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to h5py+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/h5py/e3db2972-af0a-439e-837b-c58adb3ce0e8%40googlegroups.com.

--

Thomas Caswell
tcas...@gmail.com

[Arnaldo Leon]

Hi Tom,

I believe the author of LZF was trying to squeeze some performance by not having to initialize memory. Considering that performance was paramount for this library is seem like a sensible trade-off. There is an post about it here:

https://stackoverflow.com/questions/5313150/lzf-may-compress-with-different-algorithms

The main motivation behind cleaning this up would be to reduce the number of valgrind errors and determinism of the output. The reason the latter is important is that tools like h5diff seems to report differences with LZF compressed files even though the data is the same, which is a bit annoying.

-aml