new open LiDAR from Holland (AHN3): check after download

426 views

Skip to first unread message

Martin Isenburg

unread,

Aug 26, 2016, 3:24:11 PM8/26/16

to LAStools - efficient command line tools for LIDAR processing

Hello,

i have learned that parts of the new national height model AHN3 from Holland is already online. The LiDAR data for this updated height model will be collected over the coming years and a part of it is already online. You can access it via a simple download map that uses very large tiles for the data so that the corresponding LAZ files are rather large (also see attached picture).

https://www.pdok.nl/en/node/1026

I have not yet managed to download an entire file completely using a simple Chrome download and after 2.7 GB the download failed (without me noticing). I do not seem to be the only one that this is happening to. Please make sure that you check your downloaded LAZ files for completeness by running laszip with the '-check' option, which will not decompress the file but simply checks for integrity and completeness.

D:\LAStools\bin> laszip -check -i C_39EN1.LAZ

ERROR: 'end-of-file during chunk with index 10041' after 502063770 of 676311665 points

FAILED integrity check for 'C_39EN1.LAZ' after 502063770 of 676311665 points

You will easily find out whether all went okay as you see above. Here how to check many files:

D:\LAStools\bin>laszip -check -i france\*.laz

SUCCESS for 'france\strip1.laz'

SUCCESS for 'france\strip2.laz'

SUCCESS for 'france\strip3.laz'

SUCCESS for 'france\strip4.laz'

SUCCESS for 'france\strip5.laz'

And this is how a successful LiDAR check looks like.

Regards,

Martin @rapidlasso

open_lidar_ahn3_laz_download.jpg

Martin Isenburg

unread,

Sep 4, 2016, 2:45:59 PM9/4/16

to LAStools - efficient command line tools for LIDAR processing

Hello,

I noticed that the AHN3 LiDAR is stored with millimeter resolution (meaning that the scale factors in the LAS header are set to 0.001). You can save *a lot* of storage space and transmission bandwidth by re-compressing the data from millimeter to a - more appropriate - centimeter resolution as shown below:

laszip -i C_04FZ2.LAZ -rescale 0.01 0.01 0.01 -o C_04FZ2_CM.LAZ

dir C_04FZ2*

53,672,001 C_04FZ2.LAZ

38,993,084 C_04FZ2_CM.LAZ

laszip -i C_30EZ1.LAZ -rescale 0.01 0.01 0.01 -o C_30EZ1_CM.LAZ

dir C_30EZ1*

679,888,113 C_30EZ1.LAZ

522,164,184 C_30EZ1_CM.LAZ

This would saves 14,678,917 (157,723,929) bytes in this example tile. This equals 1.2495 (1.19179) bytes or 9.9961 (9.53436) bits per point. This makes sense as each coordinate has one less digit to compress. And we can expect this digit not to contain measured information but random sensor and computation noise. Compressing a random digit between 0 and 9 requires at least log2(10) bits or 3.32192 bits if optimal entropy coding is used. Multiply this by three as it happens for each of the three coordinates x, y, and z of a point and you get 9.9658 bits per point. Wow ... the theoretic estimate matches the observed numbers fairly closely.

Assuming AHN 3 will have around a one trillion points (because AHN had over 600 billion) then using scale factors of 0.01 instead of 0.001 will require 9,965,784,284,662 bits or 1.13 Terabytes (!!!!!!) less data for the entire national height model after compressing it with LASzip. Save the Dutch taxpayers a lot of money that would otherwise be wasted on storing and transmitting the random and meaningless millimeter digit of a National LiDAR point cloud obtained with airborne laser scanning ... (-:

Regards,

Martin @rapidlasso

===============================================

E:\LAStools\bin>lasinfo C_04FZ2.LAZ

lasinfo (160804) report for C_04FZ2.LAZ

reporting all LAS header entries:

file signature: 'LASF'

file source ID: 0

global_encoding: 1

project ID GUID data 1-4: 000007DE-0002-0000-4841-004E00000000

version major.minor: 1.2

system identifier: 'Riegl LMS-Q'

generating software: 'Fugro LAS Processor'

file creation day/year: 258/2014

header size: 227

offset to point data: 227

number var. length records: 0

point data format: 1

point data record length: 28

number of point records: 11747626

number of points by return: 10435974 888167 284040 102789 36656

scale factor x y z: 0.001 0.001 0.001

offset x y z: 135000 589990 -10

min x y z: 135000.000 589818.166 -1.349

max x y z: 135949.398 591322.976 21.233

LASzip compression (version 2.2r0 c2 50000): POINT10 2 GPSTIME11 2

reporting minimum and maximum for all LAS point record entries ...

X 0 949398

Y -171834 1332976

Z 8651 31233

intensity 1 65534

return_number 1 5

number_of_returns 1 5

edge_of_flight_line 0 1

scan_direction_flag 0 0

classification 1 26

scan_angle_rank -32 32

user_data 0 190

point_source_ID 244 252

gps_time 78656384.865628 78660987.282402

number of first returns: 10435974

number of intermediate returns: 432638

number of last returns: 10410596

number of single returns: 9531582

overview over number of returns of given pulse: 9531582 1209656 552507 268503 185378 0 0

histogram of classification of points:

2845572 unclassified (1)

8158453 ground (2)

250095 building (6)

458791 water (9)

34715 Reserved for ASPRS Definition (26)

E:\LAStools\bin>lasinfo C_04FZ2_CM.LAZ

lasinfo (160804) report for C_04FZ2_CM.LAZ