About saving experimental TLS/MLS data into .las format with LAStools

[eetu.p...@fgi.fi]

Hi everyone,

We have been testing LAStools for some time now in our research group and
we are impressed with its performance and properties.

We work a lot with data measured with experimental terrestrial and mobile
scanner systems (see links at the end if interested). How well does the
.las format suit for saving stationary and mobile terrestrial laser
scanning data?

At present, the data coming from our terrestrial and mobile systems is
saved in various ASCII formats that take far too much disk space and are
not particularly efficient to read and write. If the data could be packed
into .laz format, it would improve our data storaging and transfer time
efficiencies significantly from the present situation.

Below are a few different data formats that we use:

* 1st format is used with the ROAMER Mobile Mapping system data (N points
x 8 columns):

Row(Int) Col(Int) X(Double) Y(Double) Z(Double) Intensity(11 bit int)
Time(Double), Range(Double)
...
...

* 2nd data format is ROAMER IMU Data, (N x 19, all double)

Week GPSTime Lat Long H-Ell X-ECEF Y-ECEF Z-ECEF VNorth VEast VUp
Roll Pitch Head Northing Easting Omega Phi Kappa
...
...

* 3rd data format is used with a LEICA TLS (Header + N x 7, .pts):

Header line
X(double) Y(double) Z(double) Intensity(Int) R(Int) B(Int) G(Int)
...
...

* 4th data format is used with the Sensei Mobile mapping system, (N x 9,
all double):
UTM-East UTM-North Ell-Height EchoWidth ProfNumb EchoNumb PointTime
ProfTime PointAngle
...
...

------------------

Is it possible to save this type of data into .las and .laz formats using
LAStools? And what would be the best approach for saving these types of
data with LAStools? Also, does .las format support user-defined data
types?

Regards,
Eetu Puttonen
senior researcher, D.Sc.

E-mail: eetu.puttonen (at) fgi.fi, www.fgi.fi
Tel: +358-9-29555212, Fax: +358-9-29555211

Department of Photogrammetry and Remote Sensing
Finnish Geodetic Institute
P.O. Box 15, Geodeetinrinne 2
02431 Masala, Finland

Mobile Mapping group:
http://www.fgi.fi/fgi/research/researchgroups/mobile-mapping

Active Sensing group:
http://www.fgi.fi/fgi/research/researchgroups/environmental-research-active-sensors-activesensing

[Martin Isenburg]

Hello Eetu,

thank you for this message. This is a wonderful opportunity to shine
some light on the "extra bytes" capabilities of LAS/LAZ that have been
with us since the first 1.0 specification but are only documented
properly with the introduction of the "Extra Bytes VLR" in the 1.4
specification. However, LAStools supports the "Extra Bytes VLR" also
for all previous 1.0 to 1.3 specifications, and since even RIEGL is
doing so we can expect other software vendors to follow suit
eventually. I am CC-ing Mike Childs from Global Mapper which to my
knowledge is another software that already supports the 1.4
specification who could enlighten if Extra Bytes are handled and
whether they are also accessible for earlier specifications.

> We work a lot with data measured with experimental terrestrial and mobile
> scanner systems (see links at the end if interested). How well does the
> .las format suit for saving stationary and mobile terrestrial laser
> scanning data?

Well, not 100% sure since I do not know all your needs. It seems the
E57 format specification is targeting in particular the stationary
laser scanning data. But it may as well be that for your needs LAS and
LAZ will work just fine.

> At present, the data coming from our terrestrial and mobile systems is
> saved in various ASCII formats that take far too much disk space and are
> not particularly efficient to read and write. If the data could be packed
> into .laz format, it would improve our data storaging and transfer time
> efficiencies significantly from the present situation.

I could not agree more. Let's look at some of the ASCII formats you
mention below and see how they can be mapped to LAS.

====================================================
====================================================

The easiest is the data format used with a LEICA TLS (Header + N x 7, .pts):

> Header line
> X(double) Y(double) Z(double) Intensity(Int) R(Int) B(Int) G(Int)
> ...
> ...

It absolutely bedazzles me that Leica and the folks owning their
scanners still predominantly seem to be using this prehistoric format
(-: It did serve us well in the beginning when files were small and
the technology was new, but it seems silly to store massive data sets
that results files of multiple Gigabytes as ASCII. With LAStools you
can turn it into a more efficient LAS/LAZ file with

txt2las -i big.pts -skip 1 -parse xyziRGB -o small.laz

which gives you centimeter accuracy by default. If you have more
precision in your scanner data you can up the stored accuracy to
millimeter or tenth of a millimeter with

txt2las -i big.pts -skip 1 -parse xyziRGB -o small.laz -set_scale
0.001 0.001 0.001
txt2las -i big.pts -skip 1 -parse xyziRGB -o small.laz -set_scale
0.0001 0.0001 0.0001

====================================================
====================================================

For the format is used with the ROAMER Mobile Mapping system data (N
points x 8 columns) it depends whether you need to preserve all the
information. Are the row and col fields needed or are they implied by
the X and Y. Let's assume they are needed we will store them to "extra
bytes" together with the range.

> Row(Int) Col(Int) X(Double) Y(Double) Z(Double) Intensity(11 bit int)
> Time(Double), Range(Double)
> ...
> ...

txt2las -i big.txt -parse 01xyzit2 -o small.laz -add_extra 5 "row"
"row index of point" -add_extra 5 "col" "col index of point"
-add_extra 5 "range" "distance of return [mm]" 0.001

the probably unfamiliar switches '-add_extra' is where the magic happens

-add_extra 5 "row" "row index of point"
-add_extra 5 "col" "col index of point"
-add_extra 5 "range" "distance of return [mm]" 0.001

this creates and documents "extra bytes" in the output LAZ file. run
lasinfo on the resulting 'small.laz' file to see what these switches
did. their order is important as their position in the command line
needs to correspond to the numbers 0, 1, and 2 in the parse string.

====================================================
====================================================

For the data format is used with the Sensei Mobile mapping system, (N
x 9, all double) the situation will be similar:

> UTM-East UTM-North Ell-Height EchoWidth ProfNumb EchoNumb PointTime
> ProfTime PointAngle
> ...
> ...

txt2las -i big.txt -parse xyz01r23 -o small.laz -add_extra 3 "Pulse
Width" "Full width at half maximum [ns]" 0.1 -add_extra 5 "ProfNumb"
"some profile number" -add_extra 10 "ProfTime" "some profile time"
-add_extra 3 "PointAngle" "angle of laser pulse [deg]" 0.01

Now note that I made that up because I do not know about the meaning
full ranges of this data. In particular using a full double (10) to
store the "ProfTime" seems a horrendous waste. A properly scaled
integer would probably be better suited here. The "Extra Bytes" (and
these data type numbers 3, 5, and 10 you seen in the examples above)
is described in the official LAS 1.4 specification:

http://www.asprs.org/LAS_Specification

and RIEGL put out a white paper describing their use of "Extra Bytes"
for various other scanner outputs such as Pulse width, Reflectance,
Deviation, and Amplitude.

http://lastools.org/las14/Whitepaper_LAS_extrabytes_implementation_in_Riegl_software.pdf

You should follow their example to use an as terse as possible data
type (e.g. scaled unsigned short rather than a float) because it will
lead to smaller LAS files and significantly better compressed LAZ
files.

> Is it possible to save this type of data into .las and .laz formats using
> LAStools? And what would be the best approach for saving these types of
> data with LAStools? Also, does .las format support user-defined data
> types?

So, yes, see above, yes.

Regards,

Martin @rapidlasso

[Eetu Puttonen]

Thank you for the quick answer Martin,

There are a few more additional questions whose answers would be helpful.

Do the same parsing options work with lasmerge, lassplit, and the other
commands?

What if there are more than 10 user-defined variables to be saved? The
numbering you gave seems to work from 0 to 9. It is unlikely that more
variable are needed, but still.

And, is it possible to define the range of the user-defined variables in
addition to their precision?

Regards,
Eetu Puttonen

[Martin Isenburg]

Hi,

> Do the same parsing options work with lasmerge, lassplit, and the other
> commands?

no. only txt2las can create LAS/LAZ files with extra bytes from ASCII.

> What if there are more than 10 user-defined variables to be saved? The
> numbering you gave seems to work from 0 to 9. It is unlikely that more
> variable are needed, but still.

currently only 10 are supported. seems a reasonable cut-off for the
time being, adding more would be a trivial exercise in programming.

> And, is it possible to define the range of the user-defined variables in
> addition to their precision?

the "extra bytes" VLRs have a field for the min and max value of the
values. txt2las will currently populate this field with the actual
range of the data. setting it to a particular range is not supported
right now but - again - would be just another trivial exercise in
programming.

(-:

once you have generated some example data sets it would be cool if you
could send me a small snippet of them together with the command line
used to generate them so i can provide folks with a final about the
"extra bytes" experiment.

Martin @rapidlasso

PS: note that currently only the singleton data types 1 to 10 are supported.