LASzip for LAS 1.4 - Call for Large Scale Pre-Release Testing

[Martin Isenburg]

Hello,

The LASzip compression extension - that re-encodes the new LAS 1.4 point types as the legacy LAS 1.2 point types 1 and 3 or the LAS 1.3 point types 4 and 5 before applying standard LAZ - is ready for large scale testing prior to release. Do you have LAS 1.4 with the new point types? Please help us testing!!! Instructions are at the end of the email. You need version 141021 or newer of laszip, lasinfo, and lasdiff that are all part of the open source LAStools.

==============================================

==============================================

As per our press-release for INTERGEO [1] here repeated the advantages of this approach (although ESRI's recent hostile moves on the "LAZ cloning front" seem to be diminishing benefit # 4):

1) existing legacy software can readily read the re-encoded LAS 1.4 files,
2) the popular LASzip compressor can trivially compress and decompress re-encoded LAS 1.4 content,
3) easier adaptation of the new LAS 1.4 point types by mitigating their impact on existing LAS 1.3 workflows, and
4) extra time gives geospatial players the opportunity to jointly develop the final *native* LAS 1.4 compressor and avoid fragmenting compressed LiDAR into incompatible formats.

[1] http://rapidlasso.com/2014/10/06/rapidlasso-announces-laszip-compatibility-mode-for-las-1-4/

==============================================

==============================================

Here how to test on your own files containing the new  LAS 1.4 point types:

==============================================

(1) make sure you have the latest version of LAStools

==============================================

>> laszip -version

LAStools (by mar...@rapidlasso.com) version 141021

==============================================

(2) make sure it is really a new point type

==============================================

>> lasinfo -i las14.las -nc

[...]

 point data format:          7

 point data record length:   40

[...]

==============================================

(3) compress with option '-compatible'

==============================================

>>laszip -i las14.las -compatible -o las14_compatible.laz

WARNING: the new 'compatibility mode' encoding for the new LAS 1.4

         points is not complete. this is just an awesome prototype

         as proof of concept as we gather community input ... (-;

==============================================

(4) de-compress with option '-compatible'

==============================================

>>laszip -i las14_compatible.laz  -compatible -o las14_compatible.las

WARNING: the new 'compatibility mode' encoding for the new LAS 1.4

         points is not complete. this is just an awesome prototype

         as proof of concept as we gather community input ... (-;

==============================================

(5) check identity of contents

==============================================

>>lasdiff -i las14.las -i las14_compatible.las

checking 'las14.las' against 'las14_compatible.las'

headers are identical.

raw points are identical.

files are identical. both have 874463 points. took 0.808 secs.

==============================================

(6) check forward-compatible point type re-coding in LAZ file

==============================================

>> lasinfo -i las14_compatible.laz -nc

[...]

  point data format:          3

  point data record length:   43

[...]

Regards,

Martin @rapidlasso

--

http://rapidlasso.com - open formats to zip your LiDARs

[Alex Yeryomin]

Hi, Martin.

I have processed a couple of real projects by Optech LMS into LAS 1.4. lasinfo reports (just an example for a single line, but all of them look similar):

  version major.minor:        1.4

  point data format:          6

  point data record length:   30

  number of point records:    21247293

  number of points by return: 9015896 8930344 2118928 853049 267062

  extended number of point records: 21247293

  extended number of points by return: 9015896 8930344 2118928 853049 267062 54348 7086 580 0 0 0 0 0 0 0

Please note, LMS generates LAS files using the format 6, not 7 as you asked to test, but it is really a new point type. PDR format 7 is the same as PDR format 6 with the addition of three RGB color channels. This line has 8 discrete, real-time returns. laszip has compressed these files with -compatible option without any issues. Roughly, laz files are 6 times smaller. lasdiff shows no difference between the original LAS 1.4 files and uncompressed LAZ files.

It seems it works! Thank you, Martin.

Regards,

Alex

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe

[Martin Isenburg]

Hello,

who has LAS 1.4 content? Given the unusually low number of replies (compared to similar requests of mine) I am almost about to conclude that there is no significant amount of LiDAR content delivered / stored / requested / produced out there that actually utilizes the new point types 6 to 10 that LAS 1.4 has introduced. Anyone here with lots of LAS 1.4 who can prove me wrong?

Regards,

Martin @rapidlasso

--

http://rapidlasso.com - fast tools for all things LiDAR

[Kirk Waters - NOAA Federal]

Martin,

That's a good question. I do need to do some testing with point types that can hold classes higher than 31, but I haven't come across any tools that will promote an existing data set to the new classes. I'll probably do a modification to your existing las2las so I can do that and test, but do you think that might be something you'd add to the distribution? Something like a -point_type_to X option.

Kirk

--

--
You are subscribed to "The LAS room - a friendly place to discuss the the LAS or LAZ formats" for those who want to see LAS or LAZ succeed as open standards. Go on record with bug reports, suggestions, and concerns about current and proposed specifications.
 
Visit this group at http://groups.google.com/group/lasroom
Post to this group with an email to las...@googlegroups.com
Unsubscribe by email to lasroom+u...@googlegroups.com
---
You received this message because you are subscribed to the Google Groups "The LAS room - a friendly place to discuss the LAS and LAZ formats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lasroom+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kirk Waters - NOAA Federal]

Naturally, now that I look at the code, I see that it is already there. The -set_point_type option should work nicely. Not sure if I missed it in the help or it isn't listed.

Kirk

[Heidemann, Hans]

USGS/3DEP requires, or very soon will, all lidar data in LAS 1.4, PDRF 6-10.

See USGS-NGP Lidar Base Specification, Version 1.2   http://pubs.usgs.gov/tm/11b4/pdf/tm11-B4.pdf

Karl

H. Karl Heidemann, GISP

Physical Scientist, Lidar Science

U.S. Geological Survey

Mundt Federal Building

47914 252nd Street

Sioux Falls, SD  57110

605-594-2861

kheid...@usgs.gov

"Nothing matters very much, and very few things ... matter at all."

- Arthur James Balfour

[Eric Peterson]

I just received some LAS 1.4 files with point type 6.  Your method allowed me to convert them to an older version for display in ESRI Arc programs as a "LAS Dataset".  It worked just fine!  I'm mostly an end-user of the data and often adapt LAS data to fit in ArcMap or ArcScene.  Despite the many complaints about using Arc to handle large point clouds, I've had good results using las2txt to extract the data I need from the LAS, then importing that into a geodatabase.  I could include the classification with older format LAS files.  Your method for LAS v 1.4 puts the higher classifications into "extra bytes".  Is there a way I can export those with las2txt?

Thanks!

[Martin Isenburg]

Hello,

in the meantime I have worked with Eric to solve this. He send me two version of a tiny LAS 1.4 file: this original one using the new point type 6 and one after converting it to a "compatible-mode" LAS 1.4 file that is really a LAS 1.2 files using the old point type 1 (with all the new point attributes being stored as extra bytes). The two lasinfo reports are attached.

The latest version of las2txt can properly report the new classifications of a LAS 1.4 file. This was recently discussed in this thread:

http://groups.google.com/d/topic/lastools/0TFSW3TZMoU/discussion

las2txt -i 268152_las14.las -parse xyzc -stdout | more

6269267.050000 2152541.360000 1439.22000000 71

6269271.070000 2152536.840000 1439.22000000 71

[...]

6269321.000000 2152566.000000 1435.85998535 80

6269323.000000 2152566.000000 1435.85998535 80

[...]

To do the same for the  "compatible-mode" LAS 1.4 file you will need to parse for both the 'c' and the '2' and add (!!!) them. One of them will always be zero. A '0', '1, '2', ... in the parse string will output the first, second, third, ... attribute stored as an "extra byte".

las2txt -i 268152_las14_compatible.las -parse xyzc2 -stdout | more

6269267.050000 2152541.360000 1439.22000000 0 71

6269271.070000 2152536.840000 1439.22000000 0 71

[...]

6269321.000000 2152566.000000 1435.85998535 0 80

6269323.000000 2152566.000000 1435.85998535 0 80

[...]

The large number of decimal digits in the coordinates have to do with "fluff" in the LAS file due to unreasonably small scalar factors for x, y, and z (see lasinfo reports).

Regards,

Martin

--

[Guillaume Tremblay]

Hi all,

What is the status of the experiment?

Here at Trimble, we are considering exporting LAS 1.4 file with point types 6 and 7 for a future release and it seems natural to offer the compressed version as well. Of course, it would make no sense exporting a hacked version of LAZ because it would be useless to other software. Thus we are interested in knowing more about how the experiment is doing. I think having the extra data packed as explained above seems to be a good compromise between compression and compatibility.

Are there any chance to get this in the main LASzip library anytime soon? Is there a lot missing or is it more the need for a common agreement on extra bit packaging?

We are currently using LASzip 2.2.0, to export into LAS and we plan to restrain users from using the LAZ compression or oblige them to downgrade their points (losing data). Of course, we would prefer to offer them LAZ compression of point type 6, 7 and I am sure that other software would use the feature as well if it was available through the official library. That would also contribute to the adoption of both LAZ and LAS 1.4 formats.

I hope we, and other software, will be able to provide this feature to people in the future.
Best regards,

Guillaume Tremblay
Software Engineer | Trimble Geospatial Division
174 Avenue Maréchal de Lattre de Tassigny - Fontenay-sous-Bois - 94120 - France

[Martin Isenburg]

Hello,

the latest versions of laszip.exe (since version 150330) have supported lossless compression and decompression of LAS 1.4 content via the NOAA-sponsored "LAS 1.4 compatibility mode" that was created to buy us time to peacefully resolve the unecessary LiDAR format fragmentation that ESRI has brought upon our community with the "LAZ clone". This laszip.exe compressor can not only handle LAS 1.4 content but it even allows older software (without LAS 1.4 support) to read the compatible content of the new point types from the resulting LAZ files.

http://groups.google.com/d/topic/lasroom/uqHKQH_EaKM/discussion

There are two limitations: (a) the LAS 1.4 files must have fewer that 4 billion points. and (b) the LAS 1.4 file must not use EVLRs. 

This "compatibility mode" is not yet supported by the DLL but there are rumours that a large survey company will become a sponsor of LASzip to expose the new "LAS 1.4 compatibility mode" also via the DLL in the near future. Stay tuned for news on that ...

In the meantime ... I invite Trimble and any other company to consider contributing to the new LASzip for LAS 1.4 that will have exactly all those new functionalities that *you* need as layed out in the "call for input" last year.

http://rapidlasso.com/2014/01/21/call-for-input-on-compression-of-las-1-4/

We will shortly update the community on the suggested feature set and - in particular - ask ESRI which critically important feature they require the compressor of the new LAS 1.4 point types to have so they can directly exploit compressed content in LAZ from the cloud. We would like to work with ESRI to spare their large user community from having to convert Petabytes of LiDAR content from open LAZ into a near-identical but closed variant of LAZ just because we forgot to properly address ESRI's needs. Remember ... the "natural break" in the LAS format due to the new point types allows us to include new features into the compressor, hence now is the time for the ESRIs, Trimbles, Bentleys, Hexagons, Terrasolids, and LP360s to advise us what they need from the LASzip for the new LAS 1.4 point types.

Some of you may be aware that this was discussed at recent technical meeting by both the ASPRS as well as by the OGC as a direct result of the open letter of OSGeo.

http://wiki.osgeo.org/wiki/LIDAR_Format_Letter

Potential outcomes pending agreement between ASPRS, OGC, ESRI, ... are

(1) a proper standardization committee for LAS

(2) a proper standardization committee for LAZ (for the old point types)

(3) a stream-lined but open and joint development process for compressing the new LAS 1.4 point types (this email is already part of this)

(4) a proper standardization committee for LAZ (for the new point types)

Stay tuned for updates on those sudden developments.

Martin @rapidlasso