DEMzip - first independent compression results are in

[Martin Isenburg]

Hello,

here are the first independent compression ratio tests (courtesy of Nathanael Weldon from Mondo). He has run test on some internal DEM. All input files are elevation rasters with 1m resolution generated from aerial LiDAR using las2dem, which are originally stored as uncompressed 32-bit floating point GeoTIFFs. A scatter plot showing the number and the dimensions of the used rasters is also attached.

The closest competitors are those using LERC (0.01m resolution) combined with another compression algorithm, and using a compression algorithm combined with the DISCARD_LSB switch to reduce the floating point precision. He has not compared against half float or integer storage. Here are the results – graphs attached. The various combinations of compressor, predictor and precision were done using GDAL 2.4.1, 

 

Compression Scheme	Median Compression Ratio
DEMzip (no options)	11.0%
DEMzip (-epsg, -nodata_min)	8.4%
GDAL GTiff (COMPRESS=LZW, PREDICTOR=2)	34.5%
GDAL GTiff (COMPRESS=LZW, PREDICTOR=3)	36.0%
GDAL GTiff (COMPRESS=LZW, PREDICTOR=3, DISCARD_LSB=12)	13.0%
GDAL GTiff (COMPRESS=ZSTD, PREDICTOR=3)	27.8%
GDAL GTiff (COMPRESS=LERC_ZSTD, PREDICTOR=3, MAZ_Z_ERROR=0.01)	15.1%
GDAL GTiff (COMPRESS=LERC_ZSTD, PREDICTOR=3, MAZ_Z_ERROR=0.01, DISCARD_LSB=12)	14.5%

 

He writes that "the only compressor that occasionally beats DEMzip is the very dangerous “only use if you know what you’re doing” LZW with predictor 3 and discarding the least significant 12 bits, which also produces somewhat unreliable results (I only guessed at the 12 bits instead of actually thinking about it). So DEMzip with NODATA removal is by far the best compression tested."

Thank you, Nathanael, for these excellent experiments.

Martin

[Brent Edwards]

I just started reading about your DEMZip project. Like many others, I have also faced the computational/storage problems of dealing with FLOAT32 data for elevations with centimeter accuracy.

One of the methods I've been exploring to reduce file size and potentially processing times is to scale the floats to integers. This can be achieved with GDAL, much like the scale factor in LAS files.

Here's a quick example:

1) Convert to INT32 -scale [src_min src_max [dst_min dst_max]]

gdal_translate -of GTiff -ot INT32 -scale 44.99 398.71 4499 39871 input_float32.tif ouput_int32.tif -co COMPRESS=DEFLATE -co PREDICTOR=2 -co NUM_THREADS=ALL_CPUS

2) Keep as is and apply conversion factor for processing

OR

3)  Scale back to FLOAT32 with VRT (3kb file for my test case)

gdal_translate -of VRT -ot Float32 -scale 4499 39871 44.99 398.71 input_int32.tif output_float32.vrt

For my test case, I was able to reduce the file size of my original, uncompressed FLOAT32 from 61mb down to around 10mb per tile, without compromising the centimeter accuracy.

Thoughts?

[Martin Isenburg]

Hello,

you can find the newest demzip prototype and a README in the latest version of LAStools (190728).

http://lascloud.com/download/ 

http://lascloud.com/download/demzip_README.txt 

The typical use would be for DEM, DTM, DSM, CHM, metrics, and GEOID difference rasters.

C:\software\LAStools\bin>demzip -h
usage:
demzip -i dem.tif -o dem.laz
demzip -i dem.laz -o dem.tif
demzip -i dem.bil -o dem.laz
demzip -i dem.laz -o dem.bil
demzip -i dem.asc -o dem.laz
demzip -i dem.laz -o dem.asc
demzip -i dem\*.tif -olaz -cores 3
demzip -i dem\*.bil -olaz -cores 3
demzip -i dem\*.asc -olaz -cores 3
demzip -i dem\*.laz -otif -cores 3
demzip -i dem\*.laz -obil -cores 3
demzip -i dem\*.laz -oasc -cores 3
demzip -i dem\*.asc -odir compressed_dem -olaz -cores 2
demzip -i compressed_dem\*.laz -odir dem -oasc -cores 2
demzip -h

other options:
 -nodata_value -9999      : raster value -9999 considered nodata
 -nodata_min -1000        : raster values -1000 or below considered nodata
 -nodata_min 32768        : raster values 32768 or above considered nodata
 -scale 1.0               : set vertical resolution to meter (or feet)
 -scale 0.1               : set vertical resolution to decimeter (or decifeet)
 -scale 0.01              : set vertical resolution to centimeter (or centifeet)
 -longlat -wgs84          : set horizontal datum to longlat on WGS84
 -longlat -etrs89         : set horizontal datum to longlat on ETRS89
 -longlat -gda94          : set horizontal datum to longlat on GDA94
 -longlat -nad83          : set horizontal datum to longlat on NAD83
 -longlat -nad83_csrs     : set horizontal datum to longlat on NAD83(CSRS)
 -longlat -nad83_2011     : set horizontal datum to longlat on NAD83(2001)
 -longlat -nad83_harn     : set horizontal datum to longlat on NAD83(HARN)
 -utm 32north -wgs84      : set horizontal datum to UTM32 north on WGS84
 -epsg 27700              : set horizontal datum to EPSG code 27700
 -vertical_wgs84          : set vertical datum to WGS84
 -vertical_navd88         : set vertical datum to NAVD88
 -vertical_cgvd2013       : set vertical datum to CGVD2013
 -vertical_nn2000         : set vertical datum to NN2000
 -vertical_dhhn92         : set vertical datum to DHHN92
 -vertical_dhhn2016       : set vertical datum to DHHN2016
 -elevation_survey_feet   : set vertical units from meters to US survey feet
 -sigmaxy 0.5             : horizontal accuracy expected at 0.5 meters (inactive)

Regards,

Martin