Is LAZ the most compact format to store *raster* DEMs?

[Martin Isenburg]

Hello,

as part of my resolution experiments with the open data from England [1] I had looked at how space efficient various raster format store a typical raster DEM (e.g. a DSM, a DTM, or a CHM). All LAStools can on-the-fly-convert ASC or BIL rasters to points on a grid. Then we can apply LASzip and compress these points without loss. If we use lasgrid as the decompressor we decompress directly to a desired raster format. Here are the results:

98,954,758 --- sj6504_DSM_25cm.asc

26,991,757 --- sj6504_DSM_25cm.asc.zip

64,000,306 --- sj6504_DSM_25cm.bil (incl blw/hdr)

25,895,861 --- sj6504_DSM_25cm.bil.zip (incl blw/hdr)

62,449,640 --- sj6504_DSM_25cm.img

37,540,593 --- sj6504_DSM_25cm.tif (incl tfw)

11,363,324 --- sj6504_DSM_25cm.laz

11,079,214 --- sj6504_DSM_25cm_sort.laz

This raster is 4000 by 4000 = 16,000,000 pixels of which 522,316 are empty. It seems especially for download services of large amounts of raster data it would make a lot of sense to use LAZ over zipped ASC or TIF.

It would be rather easy to write a little open source "demzip.exe" program that can compress and decompress raster grids from and to LAZ (adding a small VLR that stores the information about the original format, resolution, and bounding box). By adding an (optional) point reordering into a space filling curve would also allow I/O  efficient area-of-interest queries.

What is the most compact open (or closed) raster format you are aware of?

Regards,

Martin @rapidlasso

[1] http://rapidlasso.com/2015/09/02/england-releases-national-lidar-dem-with-insane-vertical-resolution/

PS: Below the command lines that were used. The elevation values of the DSM 'sj6504_DSM_25cm.asc' downloaded from England's Environment Agency were

first reduced from femtometer to centimeter resolution as described here [1] and then used as input below.

 

laszip -i sj6504_DSM_25cm.asc ^

          -rescale 0.001 0.001 0.01 ^

          -o sj6504_DSM_25cm.laz

lassort -i sj6504_DSM_25cm.laz ^

           -bucket_size 125 ^

           -odix _sort -olaz

lasgrid -i sj6504_DSM_25cm.laz ^

            -step 0.25 ^

            -o sj6504_DSM_25cm.bil

lasgrid -i sj6504_DSM_25cm.laz ^

            -step 0.25 ^

            -o sj6504_DSM_25cm.tif

lasgrid -i sj6504_DSM_25cm.laz ^

            -step 0.25 ^

            -o sj6504_DSM_25cm.img

[J Luis]

Martin,
 
I've wondered about this too. I made some experiments comparing to 
compressed netCDF and found puzzling results ranging from laz being > 4 
times smaller to only 15% smaller than the nc grid.
 
But I went a bit further. Instead of (laz)compressing a grid of XYZ I 
instead stored only Z by cheating. The trick was to divide Z in 3 chunks 
and put each on the X, Y, Z channels.
 
What puzzled me more was that I have a case where the no-cheating solution 
(compressing all XYZ) was 50% smaller than the one where only Z is stored.
 
I can point you to the code that reproduces this but it would imply that 
you use Julia + Laszip.jl + GMT.jl
 
https://github.com/joa-quim/Laszip.jl
https://github.com/joa-quim/GMT.jl
 
Joaquim

[Jarlath O'Neil-Dunne]

Storage is no overly expensive these days, but people's time is. Providing point clouds in LAZ format and GeoTIFF DEMs makes the data usable to the broadest user base in my experience.

[Joaquim Luis]

The Geophysical community uses mostly netCDF, not  GeoTIFF DEMs.

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe

[Jed Frechette]

So LAZ is hitting a compression ratio of ~9:1. That's more or less equivalent, and maybe a little worse, than you would expect with lossless JPEG2000, e.g.

www.microimages.com/documentation/TechGuides/75jp2AsterDEM.pdf

and there is at least some research that PNG can do even better:

http://www.isprs-ann-photogramm-remote-sens-spatial-inf-sci.net/II-5/313/2014/isprsannals-II-5-313-2014.pdf

Both of those formats are already well supported in the GDAL ecosystem so even if a user was concerned enough about storage space to use something other than GeoTiff (LZW) why would they take the time to implement and use a LAZ raster format?

These comparisons are also pretty meaningless without including a discussion of compression and decompression time, which only the ISPRS paper looks at.

[Martin Isenburg]

Hello,

thank you for those links. I had not heard of anyone actually storing elevation grids as JPEG2000 or in PNG format, but I am happy to hear that LAZ is right up there with those fancy methods that are optimized for 2D rasters. I was more thinking about a usability point of view. For some people workflows I could see value in a demzip tool that is as simple as laszip that goes ASC <-> LAZ or TIF <-> LAZ. And once a raster is stored as LAZ you can apply all kind of point processing operations on it. 

More and more people are using LAStools  and other LAS software packages to operate on dense matching points. Many data samples I have seen - like the output of the SURE algorithm of nframes - are essentially points on a grid. Hence, people already use existing LiDAR software and the LAS or LAZ format to work with photogrammetrically-derived points on a grid that really are ... rasters. Then they mash up these rasters with other LiDAR points and fill gaps with coarser SRTM, Aster DEMs, or TanDEM X rasters. Your average raster format won't allow you to drop a few million LiDAR points into the file and then add a few hundred points from an overlapping grid at lower resolution. 

Oh my god ... suddenly I see all rasters as single-return point clouds with very regular spacing ... help!!! (-;

Martin

--