beta-testing LAZip compression with spatial indexing (LAZ with built-in LAX)

[Martin Isenburg]

Hello folks,

I am looking for folks with huge LAZ collections who would like to add spatial indexing to their queries via the tiny LAX files ... but without the tiny LAX files. The (undocumented) new option to '-append' the '-lax' files when running laszip.exe still needs to be tested before unleasing it on all your data.

You may have used the "free" and open source LGPL lasindex before like shown below:

D:\lastools\bin>copy ..\data\fusa.laz fusa.laz

D:\lastools\bin>lasindex -i fusa.laz

D:\lastools\bin>dir fusa.*

04/05/2014  10:38 AM             1,524 fusa.lax

09/20/2011  05:17 PM         1,078,962 fusa.laz

               2 File(s)      1,080,486 bytes

Now you can have lasindex append the LAX file to all of your existing LAZ content *in-place* by adding the '-append' option. However, before running this on your master archive some extensive testing is needed to make sure this new integration of LAZ and LAX is bug-free. Use a copy of your data and try

D:\lastools\bin>del fusa.lax

D:\lastools\bin>lasindex -i fusa.laz -append

D:\lastools\bin>dir fusa.*

04/05/2014  10:43 AM         1,080,546 fusa.laz

The new LAZ file should have exactly the same size as the old LAZ and LAX files combined plus the 60 bytes for the EVLR header. And a lasdiff should (currently) not be able to notice the difference between the LAZ without LAX and the LAZ with LAX.

D:\lastools\bin>lasdiff -i fusa.laz -i ..\data\fusa.laz

checking 'fusa.laz' against '..\data\fusa.laz'

headers are identical.

raw points are identical.

files are identical. both have 277573 points. took 0.592 secs.

Now we need large-scale testing. NOAA. WSL, USGS. OpenTopography, USACE,  ... I am looking at you. (-:

Martin @rapidlasso

--

http://rapidlasso.com - fast tools to be happy with LiDARs

[Jonah Sullivan]

I wrote a couple of python scripts to do the file copying and comparing.

• las2las.py implements multiprocessing to copy from a LAS directory into a LAZ directory (there is the added functionality that it doesn't overwrite previously existing files).
  
• lasdiff.py implements multiprocessing to compare LAZ files in different directories with the same filename and write the output to a log file.
  

I copied 4,071 LAS files to the LAZ format twice, then added an external index to one set and an internal index to the other.

The difference analysis found that sometimes the header has a "different generating_software: 'las2las (version 140403)' 'las2las (ve '", which has nothing to do with the indexing.

It seems that the software version has been truncated, but only sometimes.

Other than that there isn't any problem.

The attached log.txt contains the output of about 1,400 comparisons.

Jonah Sullivan
Geospatial Analyst | National Geographic Information Group
Environmental Geoscience Division | GEOSCIENCE AUSTRALIA
Phone: +61 2 6249 9516 Fax: +61 2 6249 9999 
Email: Jonah.S...@ga.gov.au Web: www.ga.gov.au
Cnr Jerrabomberra Avenue and Hindmarsh Drive Symonston ACT
GPO Box 378 Canberra ACT 2601 Australia
Applying geoscience to Australia’s most important challenges

[Jonah Sullivan]

I tried the difference comparison again, this time using a survey with 217 LAS files (each file representing a 4sqkm tile).

There was no difference in the file contents.

I added a filesize calculation (of the LAZ + LAX or just LAZ when appended).

The filesize difference was 60 bytes, which was expected.

I attached the output log.txt and the updated lasdiff comparison script.

On Saturday, 5 April 2014 19:49:34 UTC+11, Martin Isenburg wrote:

[Martin Isenburg]

I got another off-list test result from the Bavarian Bureau of Landmanagement:

"I tried to add spatial indexing to data from image matching. I copied 900 laz files, then added an external index to one set and an internal index to the other. Adding the internal index was faster. There was exactly the difference of 60 byte in size comparing laz+lax and laz (internal index). This was the same with all files.

 

The message for all files:

 

headers are identical.

raw points are identical.

files are identical. both have …….. points. took 6.008 secs.

So I think works pretty good"

[Kirk Waters - NOAA Federal]

I'm running a test on each of the data sets we've got. Just grabbing the first 5 files per data set, making a copy, appending the index and doing lasdiff with the original. So far so good, but still running. What I haven't started is to do something like las2txt on a small area of each file and see if I get the same results to verify that the indexing is correct. I'd probably be lazy and just count how many records where returned. Is that a test you'd want to see?

Kirk Waters, PhD                     | NOAA Coastal Services Center

Applied Sciences Program       | 2234 South Hobson Ave
843-740-1227                          | Charleston, SC 29405    

www.csc.noaa.gov/digitalcoast

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe