without scanner channel lasoptimize will *unoptimize* multi-beam LAS/LAZ files
26 views
Skip to first unread message
Martin Isenburg
Aug 29, 2017, 12:52:41 PM8/29/17
to LAStools - efficient command line tools for LIDAR processing, The LAS room - a friendly place to discuss specifications of the LAS format
Hello,
the guys from UNAVCO / OpenTopography send me an interesting scenario of where lasoptimize fails to optimize the file.
They ran the lasoptimize tool on a number of LAZ files just to see their file sizes grow substantially:
lasoptimize -i C440000_5003000.laz ^
-odix _opt
lasoptimize -i C441000_5005000.laz ^
-odix _opt
25,853,381 C440000_5003000.laz
35,182,061 C440000_5003000_opt.laz
32,038,249 C441000_5005000.laz
50,864,395 C441000_5005000_opt.laz
That seemed bizarre at first. But my initial suspicion was quickly confirmed. It's probably data from a multi-beam system. So I looked at the files with lasview and ran a few lasinfo reports (see at the end of the email) with histograms on user_data and point_source_ID and it quickly became clear that this must be Optech Titan data. I think that is currently the (only?) integrated system available that produces airborne LiDAR with three different beams.
As the file are LAS 1.2 with point type 1 there is no "scanner channel" field. But the scanner channel is often stored to the user data field or is coded into the point source ID as mentioned in this earlier discussion already:
Clearly in this data set both was done (see lasinfo reports). So we can use either of these two fields to "help" lasoptimize to properly arrange the points by scanner channel as follows:
lasoptimize -i C440000_5003000.laz ^
-scanner_channel_in_user_data ^
-odix _user_data
lasoptimize -i C440000_5003000.laz ^
-scanner_channel_in_point_source_ID ^
-odix _point_source
lasoptimize -i C441000_5005000.laz ^
-scanner_channel_in_user_data ^
-odix _user_data
lasoptimize -i C441000_5005000.laz ^
-scanner_channel_in_point_source_ID ^
-odix _point_source
Because the scanner channel is coded into the point source ID as well as stored in the user data field (see lasinfo reports below) either of these commands above give better (but not as good as before) compression.
25,853,381 C440000_5003000.laz
35,182,061 C440000_5003000_opt.laz
27,061,144 C440000_5003000_point_source.laz
27,098,862 C440000_5003000_user_data.laz
32,038,249 C441000_5005000.laz
50,864,395 C441000_5005000_opt.laz
34,941,362 C441000_5005000_point_source.laz
34,863,047 C441000_5005000_user_data.laz
It's not as good as before because in this case the points already have *optimal* compression order. What lasoptimize adds on top is a better order for indexing via a spatial sort ... but I could maybe fine-tune the granularity to make the compression a little higher and the spatial coherence a little coarser.
Regards,
Martin @rapidasso
---------------------
lasinfo -i C440000_5003000.laz -histo user_data 1 -histo point_source 1
lasinfo (170828) report for C440000_5003000.laz
reporting all LAS header entries:
file signature: 'LASF'
file source ID: 0
global_encoding: 0
project ID GUID data 1-4: 00000000-0000-0000-0000-000000000000
version major.minor: 1.2
system identifier: ''
generating software: 'TerraScan'
file creation day/year: 322/2016
header size: 227
offset to point data: 229
number var. length records: 0
point data format: 1
point data record length: 28
number of point records: 8602840
number of points by return: 6926844 1059960 486580 129456 0
scale factor x y z: 0.01 0.01 0.01
offset x y z: 0 0 0
min x y z: 440000.00 5003000.00 987.01
max x y z: 440999.99 5003999.99 1425.87
the header is followed by 2 user-defined bytes
LASzip compression (version 3.0r4 c2 50000): POINT10 2 GPSTIME11 2
reporting minimum and maximum for all LAS point record entries ...
X 44000000 44099999
Y 500300000 500399999
Z 98701 142587
intensity 1 1135
return_number 1 4
number_of_returns 1 4
edge_of_flight_line 0 1
scan_direction_flag 0 1
classification 1 9
scan_angle_rank -49 42
user_data 1 3
point_source_ID 111 613
gps_time 494243.381029 497635.631495
number of first returns: 6926844
number of intermediate returns: 616956
number of last returns: 6924397
number of single returns: 5865357
overview over number of returns of given pulse: 5865357 1146547 1072249 518687 0 0 0
histogram of classification of points:
2213703 unclassified (1)
6189309 ground (2)
11 noise (7)
199817 water (9)
user data histogram with bin size 1
bin 1 has 2650469
bin 2 has 3014674
bin 3 has 2937697
average user data 2.03339 for 8602840 element(s)
point source id histogram with bin size 1
bin 111 has 39566
bin 112 has 22303
bin 113 has 61277
bin 211 has 636347
bin 212 has 727856
bin 213 has 665358
bin 311 has 1008129
bin 312 has 1092211
bin 313 has 1209907
bin 411 has 731705
bin 412 has 916404
bin 413 has 756100
bin 611 has 234722
bin 612 has 255900
bin 613 has 245055
average point source id 339.18 for 8602840 element(s)
lasoptimize -i C441000_5005000.laz -scanner_channel_in_user_data -odix _user_data
lasinfo -i C441000_5005000.laz -histo user_data 1 -histo point_source 1
lasinfo (170828) report for C441000_5005000.laz
reporting all LAS header entries:
file signature: 'LASF'
file source ID: 0
global_encoding: 0
project ID GUID data 1-4: 00000000-0000-0000-0000-000000000000
version major.minor: 1.2
system identifier: ''
generating software: 'TerraScan'
file creation day/year: 322/2016
header size: 227
offset to point data: 229
number var. length records: 0
point data format: 1
point data record length: 28
number of point records: 14327226
number of points by return: 13907006 284918 107195 28107 0
scale factor x y z: 0.01 0.01 0.01
offset x y z: 0 0 0
min x y z: 441000.00 5005000.00 981.77
max x y z: 441999.99 5005999.99 1433.28
the header is followed by 2 user-defined bytes
LASzip compression (version 3.0r4 c2 50000): POINT10 2 GPSTIME11 2
reporting minimum and maximum for all LAS point record entries ...
X 44100000 44199999
Y 500500000 500599999
Z 98177 143328
intensity 1 1096
return_number 1 4
number_of_returns 1 4
edge_of_flight_line 0 1
scan_direction_flag 0 1
classification 1 9
scan_angle_rank -31 32
user_data 1 3
point_source_ID 311 713
gps_time 495534.566481 498747.354700
number of first returns: 13907006
number of intermediate returns: 135686
number of last returns: 13905935
number of single returns: 13621401
overview over number of returns of given pulse: 13621401 355416 237589 112820 0 0 0
histogram of classification of points:
2043461 unclassified (1)
12228647 ground (2)
54 noise (7)
55064 water (9)
user data histogram with bin size 1
bin 1 has 4712828
bin 2 has 4807827
bin 3 has 4806571
average user data 2.00654 for 14327226 element(s)
point source id histogram with bin size 1
bin 311 has 80976
bin 312 has 83738
bin 313 has 84726
bin 411 has 539771
bin 412 has 552933
bin 413 has 579636
bin 511 has 1204463
bin 512 has 1227738
bin 513 has 1221525
bin 611 has 1662247
bin 612 has 1707777
bin 613 has 1684591
bin 711 has 1225371
bin 712 has 1235641
bin 713 has 1236093
average point source id 583.741 for 14327226 element(s)
Martin Isenburg
Aug 30, 2017, 12:23:46 PM8/30/17
to LAStools - efficient command line tools for LIDAR processing, The LAS room - a friendly place to discuss specifications of the LAS format
Hello,
thanks, Benjamin, for sharing these timing results. Note that indexed LAS will always win over indexed LAZ when running on one core and with a fast and available file system (i.e. when *not* being I/O bound). However, indexed LAZ starts winning when you used more than one core and access the file system with many read and/or write requests simultaneously. As soon as your LAS-file based pipeline becomes I/O bound (either because you use many cores or because the file system is slow or otherwise busy) then processing starts stalling as the CPUs are just waiting for data from the file system.
Regards,
Martin @rapidlasso
On Wed, Aug 30, 2017 at 5:13 PM, Benjamin Gross <mbg...@unavco.org> wrote:
Hi Martin,Thanks again for the explanation and diagnostics. For reference, here are our unscientific test results for our example dataset running las2las with an -inside query:LAS v1.2 (187,303,668 points), indexed
LAS: 5.26 GB; 45.19 s
LAZ: 482 MB; 112.91 s
LAZ (lasoptimize'd, scanner channel field set incorrectly): 722 MB; 126.26 s
LAZ (lasoptimize'd”, scanner channel field set correctly): 516 MB; 111.50 s
So it's definitely important to pay attention to where the channel info is with LAS v1.2 files and set the correct flag on lasoptimize. Doing this incorrectly has consequences for both compression and spatial queries on indexed datasets.
--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe
Reply all
Reply to author
Forward
0 new messages