new file integrity checks for LAZ files

[Martin Isenburg]

Hello,

I have added simple (forward- and backward-compatible) integrity checks to LASlib / LASzip that can detect (most) bit corruptions in a LAZ file. As I am refining these checks to also capture the more rare cases (that happen not just in the data section but in the control blocks of the LASzip bitstream) I encourage you to try this out and comment on the behavior (before I finalize and propagate these changes into the DLL). Yu can download the programs used in the examples below here:

http://lastools.org/download/laszip.exe

http://lastools.org/download/bytecopy.exe

I added an option to bytecopy.exe (that can be found in the LAStools\bin directory) that allows you to flip any particular bit in a file.

>> laszip -version

LAStools (by mar...@rapidlasso.com) version 141209

>> bytecopy -h

usage:

bytecopy -i in.abc -o out.abc -subseq 0 10000

bytecopy -i in.abc -flipbit 8000

bytecopy -h

>> copy ..\data\fusa.laz fusa_corrupt.laz

        1 file(s) copied.

>> laszip -i fusa_corrupt.laz -check

SUCCESS for 'fusa_corrupt.laz'

>> bytecopy -i fusa_corrupt.laz -flipbit 8000

>> laszip -i fusa_corrupt.laz -check

ERROR: 'chunk with index 0 of 7 is corrupt' after 369 of 277573 points

FAILED integrity check for 'fusa_corrupt.laz' after 369 of 277573 points

>> bytecopy -i fusa_corrupt.laz -flipbit 8000

>> laszip -i fusa_corrupt.laz -check

SUCCESS for 'fusa_corrupt.laz'

>> bytecopy -i fusa_corrupt.laz -flipbit 8630000

>> laszip -i fusa_corrupt.laz -check

ERROR: 'end-of-file during chunk with index 5' after 277572 of 277573 points

FAILED integrity check for 'fusa_corrupt.laz' after 277572 of 277573 points

>> bytecopy -i fusa_corrupt.laz -flipbit 8630000

>> laszip -i fusa_corrupt.laz -check

SUCCESS for 'fusa_corrupt.laz'

>> bytecopy -i fusa_corrupt.laz -flipbit 8630000

>> bytecopy -i fusa_corrupt.laz -flipbit 8631000

>> laszip -i fusa_corrupt.laz -check

ERROR: 'chunk with index 6 of 7 is corrupt' when reaching end of encoding

FAILED integrity check for 'fusa_corrupt.laz' after 277572 of 277573 points

>> bytecopy -i fusa_corrupt.laz -flipbit 8631000

>> laszip -i fusa_corrupt.laz -check

SUCCESS for 'fusa_corrupt.laz'

Please let me know how this works on your (existing) data. No need to recompress the files as this change only affects the decompressor. Nothing - absolutely nothing - has changed for the compression scheme.

One particular check that is missing (but also for LAS) is an integrity check for the header. A bit-flip changing the scale factor or a wrong offset, for example, would arbitrarily translate or scale all the points. The likely hood this to happen, however, would be rather small given the small size of the LAS header compared to the rest of the file. An obvious solution would be a CRC VLR to be added as an addendum to the LAS spec ... 

Regards,

Martin @rapidlasso

[Martin Isenburg]

Hello,

i've gotten one good feedback on the new '-check' option in laszip.exe and will be moving these checks now also into lasvalidate and lasinfo so that a single tool will do all the checking for you ... unless I hear vehement opposition due to less good experiences with the new "integrity check". 

The Finish Land Survey has apparently run the integrity check across the 69745 LAZ files of their national holdings and successfully located 5 files that had issues. Maybe Risto from nls.fi could share his experience (and the type of issues found in these five files: bit-corruptions or file-truncations ?)

Regard,

Martin @rapidlasso

[Ilves Risto (MML)]

Hi,

 

This kind of error checking is needed that we can be sure that all data is valid.  I checked all our open data LAZ files (http://www.maanmittauslaitos.fi/en/opendata/acquisition) and there was two type of errors.

 

Four of these kind of errors:

 

ERROR: 'end-of-file during chunk with index 88' after 4448648 of 10036765 points

FAILED integrity check for 'L:\laser\automaattinen\2012\20120002_BLOM_Tampere_pohj\1\M4224B1.laz' after 4448648 of 10036765 points

 

and one this kind of error:

 

ERROR: 'chunk with index 0 of 383 is corrupt' after 50000 of 19071631 points

FAILED integrity check for 'L:\laser\automaattinen\2009\20090023_FINNMAP_alue23\1\P3334H1.laz' after 50000 of 19071631 points

ERROR: wrong file signature 'ڍٻa0Ү̨ͭی9ޖ9٢º4ś퓪ߍć�|]v꽂[1]鷖qۑvuՁsҪ~aԳ̏l鎈x͂")ϕ��ƴƘɩur%]ᓐ#;OӺň‑*'

ERROR: cannot open lasreaderlas with file name 'L:\laser\automaattinen\2009\20090023_FINNMAP_alue23\1\P3334H2.laz'

ERROR: could not open lasreader

 

 

 

Risto

 

 

 

***************************************************************

 

       Risto Ilves - risto...@nls.fi

       Kehityspäällikkö, Maastotietotuotanto

       http://www.maanmittauslaitos.fi/

 

  Maanmittauslaitos     National Land Survey of Finland

  Oikokatu 11           Oikokatu 11

  40700 Jyväskylä       FIN-40700 Jyväskylä

 

Puhelin 040 514 6085    Phone   +358 40 514 6085

 

 

 

Lähettäjä: last...@googlegroups.com [mailto:last...@googlegroups.com] Puolesta Martin Isenburg
Lähetetty: 2. tammikuuta 2015 5:34
Vastaanottaja: LAStools - efficient command line tools for LIDAR processing; The LAS room - a friendly place to discuss specifications of the LAS format
Aihe: [LAStools] Re: new file integrity checks for LAZ files

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe