Converting ASCII format LiDAR data produced by Optech REALM (.all files) to LAS

[Daniel Clewley]

Dear List,

I was recently asked to convert some ASCII format LiDAR data from the
NERC Airborne Research Facility (NERC-ARF; previously NERC-ARSF) archive
held at NEODC (http://neodc.nerc.ac.uk/) to LAS format. I thought the
LAStools commands I used might be useful for the group - also keen to
hear any feedback on better ways of converting these files to LAS.

NERC-ARF data flown prior to 2009 was processed using the Optech REALM
software into ASCII files with the extension .all containing the
following columns:

1. GPS_time
2. ZZeasting (last return)
3. northing (last return)
4. height(m) (last return)
5. intensity (last return)
6. ZZeasting (first return)
7. northing (first return)
8. height(m) (first return)
9. intensity (first return)

Where ZZ is the UTM zone.

To convert to LAS, first I stripped the UTM zone from the easting using
the following command on Linux:

cat Str_239.all | sed 's/^\( *[.0-9]* *\)..\([.0-9]* *[.0-9]* *[.0-9]*
*[.0-9]* *\)..\([.0-9]* *[.0-9]* *[.0-9]* *[.0-9]*\)/\1\2\3/' >
Str_239_stripped.all

(could just have easily have written some Python to do this which would
have worked on Windows).

Then converted the text file into LAS files with first and last returns:

txt2las -parse txyzissss -set_number_of_returns 2 -set_return_number 2
-i Str_239_stripped.all -o Str_239_last_return.laz
txt2las -parse tssssxyzi -set_number_of_returns 2 -set_return_number 1
-i Str_239_stripped.all -o Str_239_first_return.laz

Finally I merged into a single file:

lasmerge -i Str_239_first_return.laz -i Str_239_last_return.laz -o
Str_239_all.laz

ready for further processing.

As a sidenote, if you are at a UK university or research institute and
have access to JASMIN (http://jasmin.ac.uk/), the open source utilities
from LAStools are installed on their system allowing older ASCII format
files in the NERC-ARF archive to be converted to LAS/LAZ before
downloading, there are some more details available from
https://github.com/arsf/arsf_on_jasmin

Thanks,

Dan


Please visit our new website at www.pml.ac.uk and follow us on Twitter @PlymouthMarine

Winner of the Environment & Conservation category, the Charity Awards 2014.

Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth PL1 3DH, UK.

This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses.

[Terje Mathisen]

Nice!

I like the way you converted each line twice in order to pick up both
the first and the second/last return!

What happened if a pulse had just a single return, i.e. a parking lot?
Would you then get two identical results or would the second return be
filled out with some obviously bogus value?

Personally I would have used perl (which has ports to all interesting
OSs), converting each input line into one or two output lines which I
would pipe through txt2las, but your approach obviously worked just as
well. :-)

Terje

--
- <Terje.M...@tmsw.no>
"almost all programming can be viewed as an exercise in caching"

[Daniel Clewley]

Hi Terje,

I've had a look at a couple of files and for pulses where there should
only be a single return there either a duplicate return or return with
very similar elevation (normally 1 - 2 cm difference). Some files also
have a very small number of pulses where only one return is stored -
these won't parse properly (as las2txt expects a value to skip).

So some tidying up of the point clouds is required for these data,
either by writing a script to work on the ASCII files or using LAStools
to clean up the LAS files.

Thanks,

Dan

[Terje Mathisen]

OK, so something like this:

while (<>) {
chomp;
s/[^\d\s]//g; // Only keep digits and white space
my ($eastlast, $northlast, $heightlast, $intlast, $eastfirst,
$northfirst, $heightfirst, $intfirst) = split;
# Check for one or two valid returns:
if ($eastfirst != $eastlast || $northfirst != northlast ||
$heightlast > $heightfirst + 0.02) { # Skip second duplicate pulse
printf("%s\n", join("\t", $eastfirst, $northfirst,
$heightfirst, $intfirst, 1, 1));
}
else {
printf("%s\n", join("\t", $eastfirst, $northfirst,
$heightfirst, $intfirst, 1, 2));
printf("%s\n", join("\t", $eastlast, $northlast, $heightlast,
$intlast, 2, 2));
}

piped into txt2las -parse xyzirn

Terje