Trimble Real Works 8.1 reads LAZ. WARNING: it writes corrupt LAS or LAZ

[Martin Isenburg]

Hello,

good news first: Trimble Real Works - starting from release 8.1 - can read LAZ and LAS files. Wonderful! But please *only read* LAZ or LAS files with Trimble Real Works 8.1. Do *not save* LAS or LAZ files with Trimble Real Works 8.1 ... the exported files will be corrupt.

Here the results of my experiment at the Trimble's GeoBusiness 2014 booth yesterday. I asked Ted to load the iconic "fusa.laz" file that is part of the lastools.zip distribution into Trimble Real Works 8.1 and it worked wonderfully. But bad things happen when saving the file back to LAS or LAZ. Here in detail:

* incorrect bounding box

* offset changes

* scale factor changes from cm to 0.1 mm

* return counts and numbers are lost

* intensities change

* GPS times lost

* scan angles lost

* classification lost

* georeferencing lost

* user data + point source ID lost

And when using LAZ a significant compression loss occurs because of the inflation in resolution from cm to 0.1 mm ... and that despite a lot less information being compressed due to the loss of most LAS point attributes. All relevant lasinfo.exe and lasvalidate.exe reports are attached.

In summary: Read but do *not* write LAS or LAZ files with Trimble Real Works 8.1 ... 

Regards,

Martin @rapidlasso

--

http://rapidlasso,com - fast tools to check crappy LiDARs

[Guillaume Tremblay]

Hi there,

I have read the post on the corrupted LAS files produces by TRW and I think there are some unfair informations.

The issue involving the bounding box will be fixed in next TRW release. Until then, the issue holds true and this is absolutely fair to mention it.

On the other hand, the unsupported features is an expected behavior. LAS is used as an import/export format to exchange point clouds with other softwares. Not an ideal format but because it is widespread, TRW provides minimal support for it. TRW is not aware of most LAS point attributes (and vice-versa). Therefore, once the LAS/LAZ has been imported, unsupported point attributes are lost. The same holds true for any other file TRW imports. TRW is not trying to solve aerial scan problems typical LAS file users work on. For TRW users, LAS is mostly a workaround for point-cloud exchange when ASCII would be too big, e57 is not available and other proprietary formats are not an option.

When imported, data is converted to native TRW format and may lose attributes. There is no such things as opening and saving back a LAS file because LAS is not a supported work file format. Thus, stating that TRW corrupts LAS files is unfair. You can import a LAS (importing only supported features). And you can export something else to LAS, exporting only LAS supported features (normals and grid information will be lost, as well as all geometries, measurements, ortho-images, meshes, inspection reports, and everything you may have created in-between).

Exporting a point cloud in LAS/LAZ is a good option for transferring point-clouds between TRW and another software. That’s the use-case covered by the LAZ/LAS import and export, knowing that the only features common to both TRW and LAS are:

• X, Y, Z coordinates
  
• RGB information
  
• Laser intensity information

As for the resolution, TRW typical data is coming from instruments with millimeter accuracy. Therefore, quantification is one order of magnitude below instruments capabilities. Centimeters may make sense for aerial scanning (where LAS comes from in the first place) but is not an option for data coming from terrestrial scanners like TX5 or TX8. But even at that level of accuracy, LAZ still provides a 4 to 5 compression rate compared to e57 and other binary (uncompressed) formats. This is a true added value.

LAS support, like any other import/export is a trade-off between what TRW is meant for and what the partially supported file format was designed to. TRW has been supporting LAS 1.2 for years now with those limitations without special complains. The novelty of TRW 8.1 is not LAS corruption, but both the LAZ and the LAS 1.4 support.

LAZ support is great because it allows to compress the billions of points TRW can export from a typical point-cloud.

Last but not least, the "lost" attributes is a common behavior shared among most point-cloud software. The open source Cloud Compare software does it and TRW competitors from other software vendors behave the same way. This is acceptable for all the same reasons mentioned above.

Having all that in mind, stating that TRW 8.1 will corrupt LAS files and that it should be avoided at all cost is a very strong statement that does not seem accurate.

Best regards,

Guillaume Tremblay
Software Engineer,
Trimble Navigation

[Howard Butler]

On Jun 4, 2014, at 12:11 PM, Guillaume Tremblay <guillaume...@trimble.com> wrote:

> Last but not least, the "lost" attributes is a common behavior shared among most point-cloud software. The open source Cloud Compare software does it and TRW competitors from other software vendors behave the same way. This is acceptable for all the same reasons mentioned above.
>
> Having all that in mind, stating that TRW 8.1 will corrupt LAS files and that it should be avoided at all cost is a very strong statement that does not seem accurate.

I agree that this hyperbole is unnecessary, and it hurts Martin's creditability to blow up relatively minor data translation issues. Paul Ramsey has a great blog post about an ESRI developer disproportionately ringing the bell that's somewhat similar to this scenario [1]. A bad bounding box in a header is definitely annoying, but the LAS ecosystem has many more softwares that do a much poorer job than TRW probably does (I've probably even written some). I know LAStools doesn't believe the bounding box of any files it ingests anyway.

On May 29, 2014, at 4:28 PM, Martin Isenburg <martin....@gmail.com> wrote:

> * incorrect bounding box
> * offset changes
> * scale factor changes from cm to 0.1 mm
> * return counts and numbers are lost
> * intensities change
> * GPS times lost
> * scan angles lost
> * classification lost
> * georeferencing lost
> * user data + point source ID lost

TRW and many other softwares that write LAS data are not data translation software. It is not their job to read an LAS file with perfect fidelity and write it out again -- LAStools exists to do that, and it already does a fantastic job at it. These other softwares writing LAS only need to support writing the data that makes sense to them from their internal working formats to concepts that map in LAS. To compare them otherwise isn't fair.

Howard

[1] http://blog.cleverelephant.ca/2014/02/introspection-double-shot.html

[Heidemann, Hans Karl]

Speaking for myself personally and in no way representing the position of
the USGS or Federal government:

I have to disagree, Howard.

LAS was initially conceived and written to address the Tower Of Babel that
arose from the proliferation of lidar data in wildly varied and largely
undocumented ASCII files. That is was binary and could store more
information in less space than ASCII was a bonus, but was hardly the
impetus. LAS is a file format SPECIFICATION; it is NOT an
eat-what-you-want buffet. It has requirements, which were thoroughly
considered and included in the Specification for good reasons. Regardless
of whether or not an individual user wants or needs the information
supported by those requirements, those requirements must be met for the
file to be a LAS file. Users and software developers have every "right"
to expect that a file with a .las extension meets the requirements of the
LAS specification. To do otherwise defeats the entire purpose of the
format.

Regrettably, we as a community (manufacturers, producers, developers, and
users) have for too long been totally lackadaisical about demanding that
LAS requirements be met. Everybody has been so singularly focused on
"well, it works for ME!" that we have slid right back into the mess of
functionally incompatible lidar (can't honestly call them "LAS") files.
The problem is so pervasive that ASPRS endeavored to have software
developed to validate .las files. Sadly, that effort was terminated for
administrative reasons -- but not for any lack of recognition of the need.


Your assertion that, since we have a piece of software (LAStools) that
does maintain format integrity all is well, is baffling.
Does that mean everybody who wants to use lidar to the extent supported by
the Specification must use LAStools?
Is LAStools the arbiter of LAS compliance? Did this designation come
through the ASPRS Board?
I don't recall such, and I have been deeply involved with the LAS
Committee for many years.

Perhaps software that produces knowingly and deliberately non-compliant
.las files should be required to display a really annoying popup warning
message stating "The files you are about to save are NOT LAS COMPLIANT!" ?

Perhaps the next version of the LAS Specification should include a bit
flag to indicate that although the file has a .las extension and shares
some characteristics of LAS, it is in fact, NOT a LAS file ?
I don't see these as viable, practical, or desirable alternatives. But
devolving into the Format Of Babel is equally unacceptable.

I agree with Martin: If the file does not conform to the file
specification, it is corrupt.

If somebody wants to use LAS, they need to use *LAS*, with all of its
requirements, and be prepared to maintain the integrity of the file and
format.
If they can't manage that, perhaps they should consider working with the
ASPRS to develop a some defined variant that suits their and their
community of use's requirements.
And if even that's too much trouble, they should just use something
different. Write an input library that pulls from a compliant LAS file
what information is needed, and write it out to a nice self-describing
HDF-5 file.
But don't go polluting the "LAS ecosystem" with more "LAS" files that do
not do what they are expected and required to do.


Karl

H. Karl Heidemann, GISP
Physical Scientist, Lidar Science
U.S. Geological Survey
Mundt Federal Building
47914 252nd Street
Sioux Falls, SD  57110
605-594-2861
kheid...@usgs.gov

"Nothing matters very much, and very few things ... matter at all."
-
Arthur James Balfour

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe

[Evon Silvia]

The opinions represented here are my own and do not represent Quantum Spatial's position, and I have no affiliation with Trimble.

I agree with the sentiment that every should endeavor to build correct LAS files, but I don't think that the format has been violated as much as has been represented. Let me consider Martin's concerns:

• incorrect bounding box
  
• Certainly an error that must be corrected, as Trimble has acknowledged.
• offset changes, scale factor changes from cm to 0.1 mm
• Although unfortunate and perhaps misleading, it is not invalid. As long as the values are correct, it is still technically valid.
• return counts and numbers are lost, scan angles lost, user data + point source ID lost
• Required in the LAS specifications and by a strict interpretation they must not be zeroed out, although a zero value for user data, PtSourceID, and ScanAngle are technically valid values.
• intensities change
• Why is this a problem? It seems perfectly reasonable to me that a software package would re-scale intensities so that all output from one software package is internally consistent. Perhaps a warning during import would suffice.
• GPS times lost
• Although unfortunate, this is not corrupt. The LAS v1.0-1.4 specifications all allow PDRF 0, which has no GPS time.
• classification lost
  
• Again, although unfortunate, there is a point class 0 in the ASPRS standard and does not indicate an invalid LAS file. It is not incorrect for software to reset a LAS file's classification.
• georeferencing lost
  
• The LAS specification requires that all LAS files include projection information. Although extremely difficult to accomplish in a standard way, losing this information does make for an invalid LAS file.

In short, the invalid bounding box and loss of georeferencing information do, in fact, indicate an invalid LAS file. Loss of return number/count, scan angle, and PtSourceID also arguably makes the LAS file invalid, but here's something to consider...

The LAS specification supports the concept of "Synthetic" points, explicitly citing the example of photogrammetrically-derived points. In this example, the return count/number, scan angle, and PtSourceID field have no meaning and would sensibly be zeroed, but by the format's strict requirements that would result in an invalid LAS file. For this reason, one could argue that requiring these fields in all cases does not make sense.

On the flip side, LAS processing software (including my own) depends on these fields being correctly populated. One could argue that you can't process ALS, MLS, and TLS data the same way, and therefore the LAS specification must be limited to ALS data. Karl's correct that the LAS specification requires these fields.

Personally, I think this points to a bigger issue that Karl touched on. The ASPRS LAS format is the de-facto standard for binary LiDAR files. Several fields in the LAS specification simply do not translate for MLS, TLS, and photogrammetric data.

Merged data is likely the future. The ASTM e57 committee attempted to create a merged data specification, but it's bogged down in bureaucracy and very few have adopted it. I think ASPRS is in a fantastic position to adapt the LAS format (2.0, perhaps?) to reflect this reality.

Evon

[Jed Frechette]

I was wondering if this thread was going to get any responses. :-)

On Thursday, June 5, 2014 11:24:49 AM UTC-6, Evon Silvia wrote:

The ASTM e57 committee attempted to create a merged data specification, but it's bogged down in bureaucracy and very few have adopted it.

That may be true in the ALS world but I would say adoption has been pretty good in the ground-based world with all of the major hardware and software vendors offering at least some level of support. e57 certainly isn't perfect but thank [diety] they had the sense to write e57validate.cpp at the same time they were writing the spec.

It sounds like RealWorks does have a couple specific issues they should address, and their users should definitely be warned that saving las files may be lossy but the ferver does seem a bit hyperbolic. Should I start complaining that lastools 'corrupts' my ptx files (a well defined text format) when I try to do a txt2las | las2txt round trip? las and e57 are exchange not working formats. Of course programs generating them should generate valid files, as challenging as that may be in some cases, however, expecting every program that touches them to do so losslessly seems like an unrealistically high bar.

[Martin Isenburg]

Hello,

seems like I stirred up emotions more than I meant to. A few weeks ago I received note from Trimble that Real Works was now exporting LAZ. I asked for a sample and - to my chagrin - the bounding box was corrupt so I postponed the news. At the Geo Business Show in London the latest version of Real Works was being demoed at the Trimble booth. I took the opportunity to satisfy my curiosity: load a LAZ file and then save it again as LAS and LAZ. So only after verifying that the bounding box corruption for LAZ exports was in the relased version I decided to make a post. After all ... LAZ files that are corrupt reflect poorly on the http://laszip.org project and - by affiliation - on me. So that could have been the message:

Do not use TRW to export LAS or LAZ because it will corrupt the bounding box.

The bounding box is not just "slightly off" but completely degenerate. Some software use the bounding box to place / scale / translate the contents into the scene. But I did not stop there. You may have noticed that - time permitting - i like to write long detailed reports about my findings. So I added to list all my additional findings to the list to make my post more relevant for those interested in all details.

corruption:
* incorrect bounding box

presision fluff:

* scale factor changes from cm to 0.1 mm

changes in contents:
* intensities change
* offset changes

data loss:

* return counts and numbers are lost

* GPS times lost
* scan angles lost
* classification lost
* georeferencing lost
* user data + point source ID lost

So I learn from this episode to sub-catgorize my bulleted lists in the future instead of just adding to the end. (-:

Other comments:

Guillaume, I don't think the information was un-fair. I admit it was poorly structured and thereby miss-leading but everything I listed was a fact. Only the first bullet item was relevant for the main story: TRW produces corrupt LAS / LAZ files. Everything else were details about additional side-effects that are often quite important to my users.

Howard, I don't think the ESRI story is similar at all.Trimble (accidentaly) miss-used my code and now their users (inadvertingly) produce corrupt files in a format that is tightly connected tomy reputation. Also ... I have seen before that once a software can import and/or export LAZ some people may decide to use it as their decompressor and/or compressor. I have seen this with FME, with Cloud Compare, with Global Mapper, ... they all change the LAS/LAZ contents - some more, some less - during compression and decompression. A summary of those changes seemed like a relevant thing to include in that post to the LAStools list.

Karl, not LAStoole but lasvalidate.exe [1] tries to be the community arbiter of LAS compliance via an open and tranparent concensus process of all those affected and interested. Each time you run the software it states:

D:\lastools\bin>lasvalidate -i chicken_poop.laz
This is version 140513 of the LAS validator. Please contact
me at 'martin....@rapidlasso.com' if you disagree with
validation reports, want additional checks, or find bugs as
the software is still under development. Your feedback will
help to finish it sooner.
needed 1.34 sec for 'chicken_poop.laz'

[1] https://github.com/lasvalidator

Jed. I was hoping that a txt2las / las2txt pipeline (at least when used with the '-iptx' and '-optx' switches) would not modify the PTX contents (beyond the requested quantization). Can you provide an example? Does that also happen for pointzip.exe / pointunzip.exe from http://pointzip.org ... ?

Cheers,

Martin

[Jed Frechette]

On Friday, June 6, 2014 2:10:17 AM UTC-6, Martin Isenburg wrote:

Jed. I was hoping that a txt2las / las2txt pipeline (at least when used with the '-iptx' and '-optx' switches) would not modify the PTX contents (beyond the requested quantization). Can you provide an example?

I haven't actually tested it, but I assume txt2las|las2txt doesn't preserve the transforms in the ptx headers. Or maybe, the -iptx and -optx the switches do apply these and archive/extract them from, a lastools specific field somewhere in the las file? I don't know, they don't seem to be documented. Regardless, I certainly wouldn't consider the failure to do so a bug. It was simply a, perhaps somewhat contrived, example of how we can't necessarily expect data schemas for file formats that store even nominally similar data objects to map to each other exactly.