How does offset and scale factor affect file size and spatial resolution?

[Adrian]

Dear lastools community,

for a project on biodiversity I am currently preprocessing several tls derived point clouds in the laz format using lastools. Generally I am very pleased with the utility of lastools particularly in regard to tls point cloud processing! Thanks for the great tool!

However I just approached some issues regarding file size, scale factor and offset that I don´t quite understand. Firstly though I´d like to intoduce you to my workflow.

My workflow looks generally like this:

1. Georeferencing of tls point clouds using Trimble Realworks (v. 11.0.3.514); export point cloud in laz format

2. Normalizing resulting georeferenced point clouds of step one using lasheight; export point cloud in laz format (note that I  introduced several intermediate steps here where point clouds are thinned clipped and ground points are identified using lasthin, lascut and lasground respectively. In each intermediate step a laz file is exported which is then used for the subsequent step)

3. Spatially coregister normalized point clouds (matching) using Trimble Realworks again. Final results are exported in laz format.

As you see the fundamental data is processessed quite heavily and laz files are saved and read frequently in the presented workflow.

So let´s come to my main issue. After step two normalizing of point clouds using lasheight the resulting laz files have a size of approximately 3.5 - 5.5 Gigabyte. As presented above these laz files are then used as input for step three (matching in trimble realworks). After exporting the results of step three, the matched point clouds are however nearly twice the size of the input data. Hereby the amount of points does not have changed considerably. So I don´t quite understand why the size of the files have increased that much. I guess it might have something to do with the scale factor, the extend and/or the offset.

Do you have an explanation?

I attached info files of laz files resulting of the above described fundamental processing steps.

Regarding the info files I have some more questions:

a.) Why are there warning messages in the info files "1_tls_georeferenced.txt" and "2_tls_normalized.txt" and what do they tell me?

b.) How comes that the scale factor of the matched laz files (see "3_tls_matched") looks so weired.

c.) Further it seems that with the scale factor of the matched laz files (see "3_tls_matched") there is a loss of precision. Could it be a solution to rescale  the too large matched laz files (step 3) using a scale factor of 0.0001 0.0001 0.0001 using -rescale and  -clip_to_bounding_box in las2las. Would I then get the same warning message as found in "2_tls_normalized.txt"?

d.) If so, is there a different way to keep precision but also to decrease file size?

I hope I could explain to you my issue in a condensed but precise manner! If not I will be glad if I can provide you the information you need. Thank you allot in advance for you help!

Stay healthy and best wishes,

Adrian

[Martin Isenburg]

Hallo,

The first and the second file have 386,997,705 points. The third file has only 365,564,667. Hence the second should be smaller. But that's not where the big difference is. The third file has larger scale values of 0.001247 instead of 0.0001. This means fewer bits are needed to store each quantized coordinate and this means compression will be more successful. Evident is this also the range. The first and the second file have an x range from -1301699 to 1697967 and from -991646 to 2000210. The third file has an x range from -26533 to 14450. This means there is much less information to store. Same for y and z. Hence the third file should compress much better and be much smaller.

=========================

The first and the second file have these scale factors and offsets.

  

  scale factor x y z:         0.0001 0.0001 0.0001
  offset x y z:               525980 5358550 790

  scale factor x y z:         0.0001 0.0001 0.0001
  offset x y z:               525950 5358530 0

The third file has these (ugly and unprofessional) scale factors and offsets.

  scale factor x y z:         0.001247645557314 0.001247645557314 0.001247645557314
  offset x y z:               526007.36696365592 5358512.9041086473 0

The first and the second file have these min and max values.

  X            -1301699    1697967
  Y            -1333668    1805692
  Z            -1260850    1524895
  intensity           0      65279
  Color R 0 65279
        G 0 65279
        B 0 65279

  X             -991646    2000210
  Y            -1125396    2012371
  Z            -1284789    1547894
  intensity           0      65022
  Color R 0 65022
        G 0 65022
        B 0 65022

The third file has these min and max values.  

  X              -26533      14450

  Y               26638      66939
  Z               -6499     123956
  intensity           0      65022
  Color R 0 65022
        G 0 65022
        B 0 65022

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe
---
You received this message because you are subscribed to the Google Groups "LAStools - efficient tools for LiDAR processing" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lastools+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lastools/8ad4c380-4310-4abd-9e50-3bbea7a9c185n%40googlegroups.com.

[Adrian]

Dear Martin,

thank you very much for your explanation of how scale factor and range affect file size! It helps allot to understand the structure of laz files and the compression of data in more depth. I greatly appreciate your support and time investment, thus I actually don´t want to bother any longer. However unfortunatelly I still don´t quite understand some features of my data, particularly considering your statements.

As you point out one would assume that the size of the third file (after matching) would be considerably smaller considering the scale factor and the range of x and y. Though the file is in reality nearly twice the size of the first and second files. Do you have by any chance an idea why this might be the case? Further, would you consider the third file to be less accurate than the other two and would rescaling to a factor of 0.0001 0.0001 0.0001 increase accuracy?

best wishes,

Adrian

[Martin Isenburg]

Hello,

As you point out one would assume that the size of the third file (after matching) would be considerably smaller considering the scale factor and the range of x and y. Though the file is in reality nearly twice the size of the first and second files. Do you have by any chance an idea why this might be the case?

Yes, The LASzip compressor also compresses the order of the points. For n points the information entropy that can be stored in the order of the points is n*log(n) bits. However, the order of the points is not evident from the lasinfo report. Here a spatial layout diagram would be needed. The LASzip compressor is tuned for compressong points in acquisition order. By permuting the points randomly you will totally blow up compression rates. That;s why lasoptimize exists, which recreated the acquisition assuming there are time stamps in the data.

I would need the actual files to verify that.

 

Further, would you consider the third file to be less accurate than the other two and would rescaling to a factor of 0.0001 0.0001 0.0001 increase accuracy?

No. Once the accuracy is gone it is gone. You would have to redo the processing and convince the software to use the original scaling factors for output.

Regards,

Martin 

[Adrian]

Hello,

thanks allot for your fast and very informative reply! That´ll help to improve our workflow! Unfortunately Trimble Realworks does not seem very helpful in the regard of convincing it to use the original scale factors for output. So we need to figure out a different solution. But anyways, your information are extremly helpful!

I will present your information to my colleague within the next couple of days.

Would it generally be okay for you if I´d upload two laz files containing samples of the original laz files, so you could take a small look?

best wishes,

Adrian

[Adrian]

Dear Martin,

after some more testing in our working group we identified one more pending question we can´t answer ourselves unfortunately. Again refering to the previously uploaded lasinfo reports we are wondering about the excact meaning of the warning message: " WARNING: full resolution of min_y not compatible with y_offset and y_scale_factor:". May you explain this warning message in more detail? Further would you say that the message can be just overlooked? Or is it recommended to alter scale factor and offset in a way that the issue of incompatibility of resolution and scale factor / offset is solved?

Besides that it was unfortunately not possible to reconstruct the above presented issues with a smaller dataset which we could share with you. However all your previous information have been very helpfull, so we were able to adjust our workflow! Thank you allot! We are looking forward to hearing from you again!

best wishes,

Adrian

[Martin Isenburg]

Hello,

we are wondering about the excact meaning of the warning message: " WARNING: full resolution of min_y not compatible with y_offset and y_scale_factor:". May you explain this warning message in more detail? Further would you say that the message can be just overlooked? Or is it recommended to alter scale factor and offset in a way that the issue of incompatibility of resolution and scale factor / offset is solved?

When software writes out a LAS file it often already knows the bounding box and directly stores that information to the min/max fields in the header. However, when the points get written to the LAS file their floating-point coordinates get converted into fixed-point values due to the scale / offset representation of points in the LAS format. This means their bounding box changes slightly and should be adjusted in the header accordingly. But some software does not do this and you have a slightly inconsistent bounding box that originates from the software's internal full floating-point bounding box coordinates from before the file was written. You can fix that with

lasinfo -i data.laz -repair_bb

Or you just ignore it ...

Regards,

Martin

[Adrian]

Hi Martin,

thanks for your very fast reply!

Unfortunately I don´t get -repair_bb to work. Running your line just provides a lasinfo output in the console, but does not alter the header of the laz file. Is there a way to run -repair_bb directly on the lazfile in order to change the bounding box information in the header?

Furthermore I tested running lasoptimize on our TLS lazfiles with rather unfortunate results. It tells me that lasoptimize is not able to handle large amount of points ( 365564667). Do you have a solution?

Best wishes,

Adrian

[Martin Isenburg]

Hello,

 

Unfortunately I don´t get -repair_bb to work. Running your line just provides a lasinfo output in the console, but does not alter the header of the laz file. 

At this point I would need a copy of one of the smaller troublesome file you can provide. 

Furthermore I tested running lasoptimize on our TLS lazfiles with rather unfortunate results. It tells me that lasoptimize is not able to handle large amount of points ( 365564667). Do you have a solution?

Don't bother with lasoptimize. The core function of lasoptimizing is to bring points back into their acquisition order. For that the points need to have GPS time stamp but your TLS data does not have them. The best you could do is lassort them with a small bucket size into a space filling curve. Unfortunately I only have a 2D sort implemented. Maybe do that by z slice. First turn your input into a set of slices of 20, 50 or 100 cm with lassplit. Then lassort each slice into a space filling curve. And then lasmerge them again.

Would be interesting to see how this improves compression. If you can provide a file I would test this out a bit.

Regards,

Martin