LASZIP on 1.4 files

[Kim Mantey]

I have been investigating the use of LASzip for compressing las 1.4 files (pdrf 6-10) to laz files. I am using the version of las tools that had were most recently updated 26 May 2015.  I have been running I have several questions/concerns that I would like to bring up with you regarding compression of las 1.4 files (pdrf 6-10).  I noticed that currently LASzip compresses these files using what is referred to as "compatibility mode".  When I ran LASzip on my las 1.4 files, the resulting laz files were 1.2 with a pdrf of 1.  These particular las files that I compressed to laz and contained EVLRs that held my spatial referencing information, so I lost this information in the compression.  I noticed on the the forum that a file cannot have EVLRs after I ran my test.

 

A second question I have is regarding the difference between classification codes between the LAS 1.2/1.3 standard and LAS 1.4 standards.  If someone is using a laz file that was compressed from 1.4 that had more than 32 classes (perhaps using the new topo-bathy classification codes?) how would this be treated in the compressed laz file?  Would they be retained?  Or would those classifications disappear and those points be put in class 0?  Or is there another way the compression handles this?   Another example are  points in class 10, 17, and 18 (have different definitions between LAS 1.2/1.3 and LAS 1.4 for these classification codes).  I am guessing that for example number two the classification scheme would be retained and the user would need to understand that this was a compressed las 1.4 file.

My third question is related to return numbers and number of returns.  If I have a sensor that utilized more than 8 returns per pulse and this information was populated in my las 1.4 file, how would this be handled in the compressed las 1.2 file?  Would the return number (greater than 5) be retained in the point data record and just not reported in the header?  Or would I lose some of those returns?

Finally, is there still a caveat on the amount of point records (last I heard it was around 4 billion) that can be in a las 1.4 file for it to successfully be compressed?

Is there going to be any development on LASzip that retains the full 1.4 format when moving from las to laz?  Or is the intention to keep LASzip in the compatibility mode and convert to a previous version of the LAS standard when compressing to laz?

I look forward to your response.  Thank you for your time and consideration regarding this matter.  

[Evon Silvia]

See my observations below after using LASzip and the DLL a LOT recently. You may need to update to the latest version to observe some of these features in the DLL, for which Quantum Spatial sponsored Martin to improve this summer.

Evon

On Tue, Oct 20, 2015 at 7:02 AM, Kim Mantey <mante...@gmail.com> wrote:

I have been investigating the use of LASzip for compressing las 1.4 files (pdrf 6-10) to laz files. I am using the version of las tools that had were most recently updated 26 May 2015.  I have been running I have several questions/concerns that I would like to bring up with you regarding compression of las 1.4 files (pdrf 6-10).  I noticed that currently LASzip compresses these files using what is referred to as "compatibility mode".  When I ran LASzip on my las 1.4 files, the resulting laz files were 1.2 with a pdrf of 1.  These particular las files that I compressed to laz and contained EVLRs that held my spatial referencing information, so I lost this information in the compression.  I noticed on the the forum that a file cannot have EVLRs after I ran my test.

​I know of no way to move spatial referencing information from an EVLR to a VLR in LAStools. The EVLRs of the input data are completely lost. You can, however, redefine the spatial referencing ​information into a new VLR using the widely documented LAStools commands.

 

A second question I have is regarding the difference between classification codes between the LAS 1.2/1.3 standard and LAS 1.4 standards.  If someone is using a laz file that was compressed from 1.4 that had more than 32 classes (perhaps using the new topo-bathy classification codes?) how would this be treated in the compressed laz file?  Would they be retained?  Or would those classifications disappear and those points be put in class 0?  Or is there another way the compression handles this?   Another example are  points in class 10, 17, and 18 (have different definitions between LAS 1.2/1.3 and LAS 1.4 for these classification codes).  I am guessing that for example number two the classification scheme would be retained and the user would need to understand that this was a compressed las 1.4 file.

​The primary differences between the LAS 1.2/3 and LAS 1.4​

 

​point types are as follows:

1. ​Higher resolution scan angle
2. Extended classification scheme (0-255 instead of 0-31*)
3. ​Extended return count (0-15 instead of 0-7)​
   
   
4. ​
   ​
   ​User-specified extra bytes
5. Overlap bit flag

*If you use TerraSolid and similar software, it will appear to the user that classifications 32-255 are available in LAS 1.2 because it ignores/repurposes the withheld, keypoint, and synthetic flags. LAStools uses these flags according to the standard, so the classifications will appear in the 0-31 range with the flags set (e.g., class 139 in TerraScan will appear as class 11+Withheld in LAStools).

If your LAS 1.4 file uses point classifications of 32 and beyond, they will get set to 0 (zero) in the "compatibility" LAZ 1.2 file during compression. However, the original classification was stored in an extrabyte attached to the point in the LAZ file. If you decompress the LAZ file using LASzip or the latest DLL, then the original classification will be restored along with the other LAS 1.4-specific features unless the extrabytes were lost along the way somehow.

My third question is related to return numbers and number of returns.  If I have a sensor that utilized more than 8 returns per pulse and this information was populated in my las 1.4 file, how would this be handled in the compressed las 1.2 file?  Would the return number (greater than 5) be retained in the point data record and just not reported in the header?  Or would I lose some of those returns?

​LASzip will truncate the return count and return number to 7 in the compatibility LAZ, but they will be once again restored as long as the extrabytes are not lost.​

 

Finally, is there still a caveat on the amount of point records (last I heard it was around 4 billion) that can be in a las 1.4 file for it to successfully be compressed?

​LASzip is currently limited to the maximum number of points dictated by an unsigned 32bit integer, or 4,294,967,295.​

 

Is there going to be any development on LASzip that retains the full 1.4 format when moving from las to laz?  Or is the intention to keep LASzip in the compatibility mode and convert to a previous version of the LAS standard when compressing to laz?

​As mentioned above, the LAS 1.4 features are retained in the LAZ file via the extrabytes, while maintaining compatibility with older LAZ readers​. I don't expect Martin to remove this compatibility feature any time soon since he seems to consider it a priority to maintain.

I look forward to your response.  Thank you for your time and consideration regarding this matter.  

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe

[Kim Mantey]

Thanks Evon for your thoughts!

Regarding writing the spatial reference back out to the VLR.  That makes sense!  Thanks.  I think my challenge here is that currently (as far as I am aware) LAS tools do not support OGC WKT for spatial referencing, and that is what is required by 1.4.  If this has changed please let me know, if not, do you have any other suggestions?

Kim

[Andrew Bell]

On Wed, Oct 21, 2015 at 3:22 PM, Evon Silvia <esi...@quantumspatial.com> wrote:

On Tue, Oct 20, 2015 at 7:02 AM, Kim Mantey <mante...@gmail.com> wrote:

I have been investigating the use of LASzip for compressing las 1.4 files (pdrf 6-10) to laz files. I am using the version of las tools that had were most recently updated 26 May 2015.  I have been running I have several questions/concerns that I would like to bring up with you regarding compression of las 1.4 files (pdrf 6-10).  I noticed that currently LASzip compresses these files using what is referred to as "compatibility mode".  When I ran LASzip on my las 1.4 files, the resulting laz files were 1.2 with a pdrf of 1.  These particular las files that I compressed to laz and contained EVLRs that held my spatial referencing information, so I lost this information in the compression.  I noticed on the the forum that a file cannot have EVLRs after I ran my test.

​I know of no way to move spatial referencing information from an EVLR to a VLR in LAStools. The EVLRs of the input data are completely lost. You can, however, redefine the spatial referencing ​information into a new VLR using the widely documented LAStools commands.

If this is a big deal, you can use PDAL (http://pdal.io) to do this, but PDAL doesn't currently decompress LAZ 1.4.  You'd have to decompress to LAS 1.4 with laszip, and then use pdal translate to go to 1.2 LAZ.  PDAL will retain the SRS info and place the data into standard VLR records.

--

Andrew Bell
andrew....@gmail.com

[Martin Isenburg]

Hello Kim,

In the coming days I will make it either an optional mode or the default option in the "LAS 1.4 compatibility mode" of LASzip such that all projection information stored in an ELVR is sutomatically moved to a VLRs so they are not lost.

Would it be a desirable default behaviour for LAStools / LASlib / LASzip to "auto-promote" geo-referenzing information stored in an EVLR in the sense thst it is automatically moved to a VLR whenever a file translation happens?

EVLRs were added for large payloads and easier manipulation of files in-place and what i suggest above does not confluct with either goal ...

Regards,

Martin

[Evon Silvia]

I agree that it is appropriate to "auto-promote" any projection EVLRs to VLRs when a LAS file in ingested, discarding any possible "Superceded" VLRs/EVLRs along the way. Personally, I consider that to be an appropriate and correct implementation of the LAS 1.4 standard.

Does LAStools have a feature to copy VLR georeferencing information from one LAS file (or text file) to the output LAS files? That would enable users to use whatever software package they want to define a WKT projection (PDAL, GeoCue, Geographic Calculator, Global Mapper, etc), then apply the projection info to whatever output files are being produced in that step. Saves the need for Martin to reinvent the wheel.

Evon

[Kirk Waters]

Did anything like this ever happen? I'm asking because I'm looking to remove the projection EVLR when creating one as a VLR, but I don't see the routines to do it in the code. The LASheader class (lasdefinitions.hpp) has a couple choices for remove_vlr and a clean_evlrs, but nothing like a remove_evlr. Perhaps I'm missing the obvious. Looking in the code released 20170625.

Kirk

[Martin Isenburg]

Hello Kirk,

I did add an (undocumented) new switch called '-move_all' to laszip.exe that will move (small) EVLRs to VLRs but only for the case of compressing LAS 1.4 to LAS 1.2 with the old compatibility mode. For the logic in the code see:

https://github.com/LAStools/LAStools/blob/master/src/laszip.cpp#L547

I have not yet added a general choice for doing that always but it's maybe something we should consider for the LASlib library to do by default as the mere existence of any EVLR is incompatible with a *streaming* or *piped* writing of LAZ files. 

Regards,

Martin @rapidlasso

[Kirk Waters - NOAA Federal]

Martin,

Thanks for the information. After checking in the LAS 1.4 format spec, I opted for marking the old georeferencing records as superseded (record id = 7) instead of removing them. Writing new ones as VLRs, not EVLRs.

Regards,

Kirk