Improved wording FWF part of LAS 1.3/1.4 specification

[Martin Isenburg]

Hello,

this discussion is in respect to revision 13 of the LAS 1.4
specification but equally applies to the corresponding parts of the
latest LAS 1.3 specification. The relevant pages are page 14/15 and
page 26/27:

http://www.asprs.org/a/society/committees/standards/LAS_1_4_r13.pdf

First RIEGL (late 2011) and then Optech (now) were miss-reading the
FWF part of the LAS specification and have been implementing export
software producing files that were incompatible to those produced by
Leica. In both cases the engineers complained to me - as we were
fixing up the exporters - that the language of the LAS specification
is too vague and subject to interpretation. In fact even what I assume
to be the "ground truth" - namely the Leica exports - are quite
different from what the specification describes.

Why do I consider Leica's output as the "ground-truth"?
(1) I read up on the history of FWF in LAS 1.3. That part of the
specification was proposed (and I assume mostly written) by Paul Galla
of Leica.
(2) It seems that the Leica LAS 1.3 FWF is the only output that is
currently being used in a third vendor's commercial software
(Terrasolid) for actual processing (extracting additional returns).

There are two issues to clarify before we can start improving:

(1) Where is the number of samples per waveform encoded?

(a) in the "Number of Samples" field of the Wave Packet Descriptor
VLRs (of which there are only 255)? This is what Leica stores in this
field, and this was what I read into the wording of the specification.
I quote page 27:

"Number of Samples: The number of samples associated with this
waveform packet type. This value always represents the fully
decompressed waveform packet."

But Leica outputs 256 samples (no matter what) whereas Optech outputs
a great variety of different lengths.

(b) implicitely in the "size" field of the "Wave Packet" in each point
record? This is what Optech does. It is implicit because the stored
number has to be divided by two to get the actual number of (16 bit)
sampled. But this "implicit" computation is a little dangerous because
if the actual samples were stored in some compressed manner then the
size field would no longer be able to fulfill this dual role.

So does Optech now have to (1) store a huge number of Wave Packet
Descriptors each with a different "Number of Samples" field as you can
see in the example histogram of the "size" field below or (2) pad the
waveforms with "zero" values to lower the number of VLRs required?

lasinfo.exe -i optech.laz -histo wavepacket_size 1 -nh -nv -nmm
bin 96 has 58591
bin 104 has 113479
bin 112 has 206085
bin 120 has 607946
bin 128 has 2653437
bin 136 has 2249816
bin 144 has 1193661
bin 152 has 1026636
bin 160 has 899445
bin 168 has 809480
bin 176 has 751317
bin 184 has 711539
bin 192 has 668146
bin 200 has 606943
bin 208 has 540692
bin 216 has 494023
bin 224 has 456205
bin 232 has 409471
bin 240 has 365311
bin 248 has 312900
bin 256 has 258146
bin 264 has 210543
bin 272 has 174621
bin 280 has 144973
bin 288 has 115251
bin 296 has 91945
bin 304 has 73190
bin 312 has 55318
bin 320 has 41798
bin 328 has 31443
bin 336 has 22544
bin 344 has 16012
bin 352 has 11124
bin 360 has 7527
bin 368 has 5220
bin 376 has 3332
bin 384 has 2041
bin 392 has 1342
bin 400 has 973
bin 408 has 562
bin 416 has 410
bin 424 has 184
bin 432 has 191
bin 440 has 119
bin 448 has 39
bin 456 has 35
bin 464 has 25
bin 472 has 15
bin 480 has 15
bin 488 has 10
bin 496 has 5

I seem to remember that also RIEGL was doing it also this way but
their variety was much lower (160, 240, 320, 400, 480). And they are
now exporting PulseWaves which is one way to side-step the issue.
Roland? Guenther? Can you confirm?

(2) What is scaling and direction of the vector stored in the X(t),
Y(t), Z(t) fields of the "Wave Packet" in each point record?

The wording of the specification is really confusing here, but this is
what Leica and RIEGL (and soon also Optech) are storing in the X(t),
Y(t), Z(t) fields.

X(t), Y(t), Z(t) contain a vector that describes the direction of the
laser beam (away from the optical origin of the laser) and the
round-trip (!) distance that the light of the laser travels per
picosecond in meters. Hence, the length of this vector should be
somewhere around 0.299792458 / 1000 / 2 = 0.00014989623. So that the
"location" of each waveform sample for "S" ranging from 0 to
num_samples-1 can be computed as

dist = location - s*temporal;
X_sample = X_return + dist*X(t);
Y_sample = Y_return + dist*Y(t);
Z_sample = Z_return + dist*Z(t);

where "location" is the "Return Point location" field from the Wave
Packet of each point record that stores - i quote the specification
(page 14) - an "offset in picoseconds from the first digitized value
to the location within the waveform packet that the associated return
pulse was detected." and "temporal" is the "Temporal Sample Spacing"
that stores - i quote the specification (page 27) the "temporal sample
spacing in picoseconds. Example values might be 500, 1000, 2000 and so
on, representing digitizer frequencies of 2 GHz, 1 GHz and 500 MHz
respectively."

Hopefully one day get all three big hardware vendors output mutually
compatible LAS 1.3 FWF files ... but issue (1) really needs to be
resolved first. For issue (2) we only need a better wording in the
spec. Seems by nagging has resolved that now. (-: Is anyone aware on
any other software (besides RiPROCESS, LMS, and ALSXX) that generates
LAS 1.3 FWF?

Comments?

Martin @rapidlasso

--
http://rapidlasso.com - fast tools to fix LiDAR specs

[Alex]

As Martin noticed, instead of constructing a lot of Waveform Packet Descriptors for waveforms of different length, Optech LMS writes a single Waveform Packet Descriptor for all waveforms. It sets Bits per sample (16) and Waveform compression type (no compression). LMS uses Waveform packet size  to represent a length of waveform for each return. Because there is no compression, this is the exact length of the waveform (in bytes, not in samples). Thus, the field Number of samples of Waveform Packet Descriptor is unnecessary in this case, and LMS writes a total number of samples to this field. When one run lasinfo, it says:

variable length header record 2 of 2:
  reserved             0
  user ID              'LASF_Spec'
  record ID            100
  length after header  26
  description          ''
  index 1 bits/sample 16 compression 0 samples 2710064 temporal 1000 gain 1, offset 0

[Martin Isenburg]

Hello Alex,

i think there is little choice but to convince Optech to have their
LMS LAS FWF exporter fixed to be specification conform. Writing the
total number of samples into that field broke my software and will -
no doubt - break others too that expect a specification conform LAS
file. I quickly compiled a "special investigation version" of LAStools
called las2txt_WOF and lasview_WOF (WOF = With Optech Fix) to produce
the illustrations I send earlier ... but this software cannot read
"real" LAS FWF files anymore ...

Here a real-world example. A little know fact, but laszip.exe can also
compress the waveform data when run with the option 'waveforms' such
as

laszip.exe -i leica_las13fwf.las -olaz -waveforms

However running this

laszip.exe -i optech_las13fwf.las -olaz -waveforms

currently crashes the compressor because the input does not follow the
specification. Of course it should not crash but exit with an ERROR
message. The next version will exdo so proclaiming "corrupt LAS FWF
file". The LASzip compressor has no chance to compress the waveforms
correctly because suddenly the information about how many samples are
in the waveform is not there anymore. The original reason that there
is both a "number of samples" entry as well as a "size" entry was - in
fact - to allow compression to be added to the official LAS
specification one day (as the specification explicitely states).

By not following the spec Optech LMS LAS 1.3 FWF output will not only
cause unpredictable behaviour in software that expetcs the LAS input
to be specification conform ... it will also prohibit future
developments - such as compression - to be applied to Optech LMS LAS
1.3 FWF data. As the same FWF mechanism will live on in LAS 1.4 FWF
and as full waveform become more and more dominant with REDD+ biomass
campaigns and full waveform algorithms maturing ... I think it is best
to make the LAS FWF output of Optech LMS specification conform as soon
as possible ...

Any one else have an opinion here?

Regards,

Martin @rapidlasso

> --
> --
> You are subscribed to "The LAS room - a friendly place to discuss the the
> LAS or LAZ formats" for those who want to see LAS or LAZ succeed as open
> standards. Go on record with bug reports, suggestions, and concerns about
> current and proposed specifications.
>
> Visit this group at http://groups.google.com/group/lasroom
> Post to this group with an email to las...@googlegroups.com
> Unsubscribe by email to lasroom+u...@googlegroups.com
> ---
> You received this message because you are subscribed to the Google Groups
> "The LAS room - a friendly place to discuss the LAS and LAZ formats" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to lasroom+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Alex Yeryomin]

Martin, I agree. I will inform developers about this issue and possible negative consequences.

Thank you for the hint on laszip and waveforms!

Alex

[Martin Isenburg]

Great to hear that, Alex.

I agree that not being able to specify the number of samples "per
shot" is annoying and one of the major design flaws of LAS 1.3 FWF
(don't get me started (-;). This (and other design issues) are the
reason we started the PulseWaves effort. As far as I can reconstruct
it, this flaw was introduced like this: Leica was the first to want
FWF exports and pushed for this functionality. Their hardware seemed
to be to always produce 256 samples and apparently they did not
anticipate that anyone may want to vary the number of samples some
day. No one seemed to scrutinize Leica's suggested addendums further.
I know Paul Galla tried really hard to get feedback on the draft but
got none. Last year I read up on that.

This was all discussed here:
http://groups.yahoo.com/neo/groups/lasformat/info

And then continued here:
http://lidar.cr.usgs.gov/

But now all of these "official records" of the history of the LAS
format seem gone. It is a pitty. Should such discussions not be
preserved? I would assume that a anyone maintaining an open standard
should. But who am I to speak ... all of the forums that I have
created rely on the goodwill of a large corporation. But they did once
say "do no evil" ... albeit quite some time ago. (-:

> --
> --
> Post to "PulseWaves" by email to pulse...@googlegroups.com
> Unsubscribe by email to pulsewaves+...@googlegroups.com
> Visit this group's message archives at http://pulsewaves.org

>
> ---
> You received this message because you are subscribed to the Google Groups

> "PulseWaves - no pulse left behind" group.

> To unsubscribe from this group and stop receiving emails from it, send an

> email to pulsewaves+...@googlegroups.com.

[Martin Isenburg]

Hello,

i double checked and can confirm that RIEGL's LAS 1.3 FWF exports
already follows what we have have outlined in this thread and is fully
compatible with Leica's LAS 1.3 FWF exports. They simply allocate a
whole bunch of "empty" wavepacket descriptors as VLRs and then
populate them before closing the written file with the waveform
descriptors that were actually used during export. That is one easy
way to deal with the fixed sample block size flaw in LAS 1.3 FWF ...
assuming the sample block sizes do not vary too much.

But it seems that Optech will have to do some padding to avoid using
hundreds of VLRs for the many different waveform sample block sizes
they seem to be exporting with the current incompatible LAS 1.3 FWF
exporter. Attached a sample lasinfo report for a RIEGL LAS 1.3 FWF
file. That's more or less how it should look like ...

Regards,

Martin @rapidlasso