Hello,
when Fugro searched for MH370 a user of LAStools at Geoscience Australia was testing which elevation raster format would be most compressed to upload the seafloor rasters produced on the survey vessels over a satellite link. He pointed out to me that in his experiments, LAZ was the most compressed way to store typical elevation rasters in comparison with TIF. ASC. gzipped ASC, BIL, gzipped BIL, IMG, etc.
I am now planning to release DEMzip - a compressor for elevation grids, GEOIDs and similar non-picture image rasters. This will turn TIF. BIL, IMG, and ASC into more compact RasterLAZ files. I'd like to find some people who can test the first demzip.exe and give me useful feedback.
Advantages of storing elevation grids and rasters as RasterLAZ:
(1) highest compression as far as I can tell
(2) directly feed into existing LAZ processing pipelines
(3) raster and points are merging anyways (photogrammetic DSMs)
(4) elevation values with fixed-point resolution, not in floating-point
(5) any (!!!) reasonable raster order is supported
(6) spatial indexing is readily supported
(7) entire range can be used. "nodata" rasters are simply omitted
(8) optional fast coverage decompression (coded separately for LAS 1.4)
(9) storage of additional (LAS-like) attributes supported
(10) display rasters mixed with points via Potree and/or Entwine / Greyhound
Once there is a RasterLAZ driver for GDAL this format could also become interesting for direct exploitation in raster processing packages.
A RasterLAZ file is a LAZ file with an extra VLR that stores the raster relevant extra information such as ncols, nrows, stepx resolution, stepy resolution, ...
Below two examples for DTM rasters in ASC and Geoid grids in TIF
============================
DTM raster: ASC -> RasterLAZ
============================
I've downloaded this ZIP file from the link above. It contains 25 DTMs with 50cm resolution from Englands Open LiDAR portal.
07/07/2019 02:08 PM 162,454,741 LIDAR-DTM-50CM-TQ37ne.zip
I then compressed the 25 ASC files while also adding the missing EPSG code 27700. The individual 25 files go from 631,601,776 bytes uncompressed ASC down to 50,030,353 bytes as RasterLAZ.A decrease of 92% in size. Also compared to the (not directly usable) zipped archive of 162,454,741 bytes the total file size decreases by nearly 70% in size. At a national scale those are *significant* savings in terms of storage, backup, and transmission bandwidth.
>> demzip -i *.asc -olaz -epsg 27700
done with 'tq3575_DTM_50CM.laz'. total time 8.238 sec.
done with 'tq3576_DTM_50CM.laz'. total time 8.274 sec.
done with 'tq3577_DTM_50CM.laz'. total time 7.784 sec.
done with 'tq3578_DTM_50CM.laz'. total time 7.771 sec.
done with 'tq3579_DTM_50CM.laz'. total time 7.761 sec.
done with 'tq3675_DTM_50CM.laz'. total time 8.662 sec.
done with 'tq3676_DTM_50CM.laz'. total time 8.596 sec.
done with 'tq3677_DTM_50CM.laz'. total time 7.731 sec.
done with 'tq3678_DTM_50CM.laz'. total time 7.707 sec.
done with 'tq3679_DTM_50CM.laz'. total time 7.722 sec.
done with 'tq3775_DTM_50CM.laz'. total time 8.54 sec.
done with 'tq3776_DTM_50CM.laz'. total time 8.085 sec.
done with 'tq3777_DTM_50CM.laz'. total time 7.841 sec.
done with 'tq3778_DTM_50CM.laz'. total time 7.799 sec.
done with 'tq3779_DTM_50CM.laz'. total time 7.691 sec.
done with 'tq3875_DTM_50CM.laz'. total time 8.489 sec.
done with 'tq3876_DTM_50CM.laz'. total time 8.637 sec.
done with 'tq3877_DTM_50CM.laz'. total time 8.292 sec.
done with 'tq3878_DTM_50CM.laz'. total time 7.77 sec.
done with 'tq3879_DTM_50CM.laz'. total time 8.312 sec.
done with 'tq3975_DTM_50CM.laz'. total time 9.256 sec.
done with 'tq3976_DTM_50CM.laz'. total time 8.766 sec.
done with 'tq3977_DTM_50CM.laz'. total time 8.689 sec.
done with 'tq3978_DTM_50CM.laz'. total time 7.785 sec.
done with 'tq3979_DTM_50CM.laz'. total time 7.722 sec.
done with all files. total time for 25 files 203.927 sec.
>> lasinfo -i tq3575_DTM_50CM.laz
lasinfo (190706) report for 'tq3575_DTM_50CM.laz'
reporting all LAS header entries:
file signature: 'LASF'
file source ID: 0
global_encoding: 0
project ID GUID data 1-4: 00000000-0000-0000-0000-000000000000
version major.minor: 1.2
system identifier: 'raster compressed as LAZ points'
generating software: 'LAStools (c) by rapidlasso GmbH'
file creation day/year: 0/0
header size: 227
offset to point data: 455
number var. length records: 2
point data format: 0
point data record length: 20
number of point records: 4000000
number of points by return: 4000000 0 0 0 0
scale factor x y z: 0.25 0.25 0.01
offset x y z: 500000 100000 0
min x y z: 535000.00 175000.00 16.39
max x y z: 536000.00 176000.00 64.38
variable length header record 1 of 2:
reserved 0
user ID 'Raster LAZ'
record ID 7113
length after header 80
description 'by LAStools of rapidlasso GmbH'
ncols 2000
nrows 2000
llx 535000
lly 175000
stepx 0.5
stepy 0.5
sigmaxy <not set>
variable length header record 2 of 2:
reserved 0
user ID 'LASF_Projection'
record ID 34735
length after header 40
description 'by LAStools of rapidlasso GmbH'
GeoKeyDirectoryTag version 1.1.0 number of keys 4
key 1024 tiff_tag_location 0 count 1 value_offset 1 - GTModelTypeGeoKey: ModelTypeProjected
key 3072 tiff_tag_location 0 count 1 value_offset 27700 - ProjectedCSTypeGeoKey: OSGB 1936 / British National Grid
key 3076 tiff_tag_location 0 count 1 value_offset 9001 - ProjLinearUnitsGeoKey: Linear_Meter
key 4099 tiff_tag_location 0 count 1 value_offset 9001 - VerticalUnitsGeoKey: Linear_Meter
LASzip compression (version 3.4r1 c2 50000): POINT10 2
reporting minimum and maximum for all LAS point record entries ...
X 140001 143999
Y 300001 303999
Z 1639 6438
intensity 0 0
return_number 1 1
number_of_returns 1 1
edge_of_flight_line 0 0
scan_direction_flag 0 0
classification 0 0
scan_angle_rank 0 0
user_data 0 0
point_source_ID 0 0
number of first returns: 4000000
number of intermediate returns: 0
number of last returns: 4000000
number of single returns: 4000000
overview over number of returns of given pulse: 4000000 0 0 0 0 0 0
histogram of classification of points:
4000000 never classified (0)
ASC
07/07/2019 09:43 PM 24,794,485 tq3575_DTM_50CM.asc
07/07/2019 09:43 PM 25,560,233 tq3576_DTM_50CM.asc
07/07/2019 09:43 PM 23,732,074 tq3577_DTM_50CM.asc
07/07/2019 09:43 PM 23,795,594 tq3578_DTM_50CM.asc
07/08/2019 10:57 AM 23,749,362 tq3579_DTM_50CM.asc
07/07/2019 09:43 PM 27,911,259 tq3675_DTM_50CM.asc
07/07/2019 09:43 PM 27,496,804 tq3676_DTM_50CM.asc
07/07/2019 09:43 PM 23,879,579 tq3677_DTM_50CM.asc
07/07/2019 09:43 PM 23,761,068 tq3678_DTM_50CM.asc
07/07/2019 09:43 PM 23,738,069 tq3679_DTM_50CM.asc
07/07/2019 09:43 PM 27,388,247 tq3775_DTM_50CM.asc
07/07/2019 09:43 PM 25,415,208 tq3776_DTM_50CM.asc
07/07/2019 09:43 PM 24,026,165 tq3777_DTM_50CM.asc
07/07/2019 09:43 PM 23,829,645 tq3778_DTM_50CM.asc
07/07/2019 09:43 PM 23,731,954 tq3779_DTM_50CM.asc
07/07/2019 09:43 PM 26,748,219 tq3875_DTM_50CM.asc
07/07/2019 09:43 PM 27,681,443 tq3876_DTM_50CM.asc
07/07/2019 09:43 PM 25,822,100 tq3877_DTM_50CM.asc
07/07/2019 09:43 PM 23,889,443 tq3878_DTM_50CM.asc
07/07/2019 09:43 PM 23,770,079 tq3879_DTM_50CM.asc
07/07/2019 09:43 PM 27,837,577 tq3975_DTM_50CM.asc
07/07/2019 09:43 PM 27,880,449 tq3976_DTM_50CM.asc
07/07/2019 09:43 PM 27,420,501 tq3977_DTM_50CM.asc
07/07/2019 09:43 PM 23,912,035 tq3978_DTM_50CM.asc
07/07/2019 09:43 PM 23,830,184 tq3979_DTM_50CM.asc
25 Datei(en), 631,601,776 Bytes
RasterLAZ
07/08/2019 01:59 PM 2,111,020 tq3575_DTM_50CM.laz
07/08/2019 01:59 PM 2,021,144 tq3576_DTM_50CM.laz
07/08/2019 01:59 PM 2,024,230 tq3577_DTM_50CM.laz
07/08/2019 01:59 PM 2,099,126 tq3578_DTM_50CM.laz
07/08/2019 01:59 PM 1,942,613 tq3579_DTM_50CM.laz
07/08/2019 01:59 PM 2,051,800 tq3675_DTM_50CM.laz
07/08/2019 01:59 PM 2,302,776 tq3676_DTM_50CM.laz
07/08/2019 02:00 PM 1,965,062 tq3677_DTM_50CM.laz
07/08/2019 02:00 PM 1,919,514 tq3678_DTM_50CM.laz
07/08/2019 02:00 PM 1,906,005 tq3679_DTM_50CM.laz
07/08/2019 02:00 PM 2,142,144 tq3775_DTM_50CM.laz
07/08/2019 02:00 PM 2,147,488 tq3776_DTM_50CM.laz
07/08/2019 02:00 PM 1,917,417 tq3777_DTM_50CM.laz
07/08/2019 02:00 PM 1,839,256 tq3778_DTM_50CM.laz
07/08/2019 02:00 PM 1,873,992 tq3779_DTM_50CM.laz
07/08/2019 02:01 PM 2,119,843 tq3875_DTM_50CM.laz
07/08/2019 02:01 PM 2,150,978 tq3876_DTM_50CM.laz
07/08/2019 02:01 PM 2,059,777 tq3877_DTM_50CM.laz
07/08/2019 02:01 PM 2,036,149 tq3878_DTM_50CM.laz
07/08/2019 02:01 PM 1,829,444 tq3879_DTM_50CM.laz
07/08/2019 02:01 PM 1,980,088 tq3975_DTM_50CM.laz
07/08/2019 02:01 PM 1,813,433 tq3976_DTM_50CM.laz
07/08/2019 02:02 PM 2,081,793 tq3977_DTM_50CM.laz
07/08/2019 02:02 PM 1,885,422 tq3978_DTM_50CM.laz
07/08/2019 02:02 PM 1,809,839 tq3979_DTM_50CM.laz
25 Datei(en), 50,030,353 Bytes
============================
geoid grids: TIF-> RasterLAZ
============================
Compressing 6 NAVD88 geoid files with demzip means going from
202,247,212 bytes in TIF down to 32,202,120 bytes in RasterLAZ, which corresponds to a decrease of 84% in file size.
>> demzip -i navd*.tif -longlat -nad83 -olaz
done with 'navd88-03.laz'. total time 4.061 sec.
done with 'navd88-06.laz'. total time 1.288 sec.
done with 'navd88-09.laz'. total time 4.987 sec.
done with 'navd88-12a.laz'. total time 4.938 sec.
done with 'navd88-12b.laz'. total time 4.906 sec.
done with 'navd88-99.laz'. total time 3.937 sec.
done with all files. total time for 6 files 24.117 sec.
TIF
07/05/2019 10:28 AM 37,304,531 navd88-03.tif
07/05/2019 10:26 AM 15,033,645 navd88-06.tif
07/05/2019 10:28 AM 37,579,719 navd88-09.tif
07/05/2019 10:28 AM 38,237,925 navd88-12a.tif
07/05/2019 10:27 AM 38,238,057 navd88-12b.tif
07/05/2019 10:28 AM 35,853,335 navd88-99.tif
6 Datei(en), 202,247,212 Bytes
RasterLAZ
07/08/2019 08:21 PM 5,962,465 navd88-03.laz
07/08/2019 08:21 PM 1,942,043 navd88-06.laz
07/08/2019 08:21 PM 6,074,111 navd88-09.laz
07/08/2019 08:21 PM 6,132,612 navd88-12a.laz
07/08/2019 08:21 PM 6,132,653 navd88-12b.laz
07/08/2019 08:21 PM 5,958,236 navd88-99.laz
6 Datei(en), 32,202,120 Bytes
>> lasinfo -i navd88-99.laz
lasinfo (190706) report for 'navd88-99.laz'
reporting all LAS header entries:
file signature: 'LASF'
file source ID: 0
global_encoding: 0
project ID GUID data 1-4: 00000000-0000-0000-0000-000000000000
version major.minor: 1.2
system identifier: 'raster compressed as LAZ points'
generating software: 'LAStools (c) by rapidlasso GmbH'
file creation day/year: 0/0
header size: 227
offset to point data: 447
number var. length records: 2
point data format: 0
point data record length: 20
number of point records: 13843203
number of points by return: 13843203 0 0 0 0
scale factor x y z: 0.008333333333333 0.008333333333333 0.01
offset x y z: 0 0 0
min x y z: 171.99166666666716 14.991666666666733 -70.88
max x y z: 300.00833333333418 72.008333333333653 28.10
variable length header record 1 of 2:
reserved 0
user ID 'Raster LAZ'
record ID 7113
length after header 80
description 'by LAStools of rapidlasso GmbH'
ncols 7681
nrows 3421
llx 171.992
lly 14.9917
stepx 0.0166667
stepy 0.0166667
sigmaxy <not set>
variable length header record 2 of 2:
reserved 0
user ID 'LASF_Projection'
record ID 34735
length after header 32
description 'by LAStools of rapidlasso GmbH'
GeoKeyDirectoryTag version 1.1.0 number of keys 3
key 1024 tiff_tag_location 0 count 1 value_offset 2 - GTModelTypeGeoKey: ModelTypeGeographic
key 2048 tiff_tag_location 0 count 1 value_offset 4269 - GeographicTypeGeoKey: GCS_NAD83
key 4099 tiff_tag_location 0 count 1 value_offset 9001 - VerticalUnitsGeoKey: Linear_Meter
LASzip compression (version 3.4r1 c2 50000): POINT10 2
reporting minimum and maximum for all LAS point record entries ...
X 20640 36000
Y 1800 8640
Z -7088 2810
intensity 0 0
return_number 1 1
number_of_returns 1 1
edge_of_flight_line 0 0
scan_direction_flag 0 0
classification 0 0
scan_angle_rank 0 0
user_data 0 0
point_source_ID 0 0
number of first returns: 13843203
number of intermediate returns: 0
number of last returns: 13843203
number of single returns: 13843203
overview over number of returns of given pulse: 13843203 0 0 0 0 0 0
histogram of classification of points:
13843203 never classified (0)