Re: [The LAS room] Where is the WDP Header?

[Martin Isenburg]

Hello Evon,

the easy answer is: don't use LAS 1.3 FWF. Do your clients a favor and discourage them from using LAS 1.3 FWF ... unless their sole objective is to extract additional (weak) LiDAR returns usingTerrasolid from the full waveform output of a Leica scanner. Tell them to consider PulseWaves if their objective is to get full waveform data alongside their discrete returns in anticipation that in a couple of years there will be commercial software that operates directly in the waveform. This is the advise that I give to anyone who is flying LiDAR for forestry applications these days. We are not quite there yet but I can "feel" the full waveform algorithms maturing and PulseWaves will hopefully speed up their commercialization. Disclaimer: I am biased against LAS 1.3 FWF in favor of PulseWaves due to my own agenda ... (-;

It is my opinion (and implementation) that #3 is correct. If ((global_encoding & 2) == 0) and ((global_encoding & 4) == 4) and (start_of_waveform_data_packet_record" == 0) then and only then there is an external WDP file (or a LASzip compressed WDZ file) that has the EVLR header as its 60 byte header. The first waveform data packet starts any time at 60 bytes or later. In my implementation in LASlib I am adding another copy of the descriptors after the 60 byte header and before the first waveform data packet to cross-check them being identical for the two files (which is a good idea in any case and becomes really important when compression is used). Below you can see the relevant code bits from "laswaveform13writer.cpp"

Regards,

Martin

==========

  // create stream

  if (IS_LITTLE_ENDIAN())

  {

    stream = new ByteStreamOutFileLE(file);

  }

  else

  {

    stream = new ByteStreamOutFileBE(file);

  }

  // write extended variable length header variable after variable (to avoid alignment issues)

  U16 reserved = 0xAABB;

  if (!stream->put16bitsLE((U8*)&reserved))

  {

    fprintf(stderr,"ERROR: writing EVLR reserved\n");

    return FALSE;

  }

  I8 user_id[16];

  memset(user_id, 0, 16);

  strcpy(user_id, "LASF_Spec");

  if (!stream->putBytes((U8*)user_id, 16))

  {

    fprintf(stderr,"ERROR: writing EVLR user_id\n");

    return FALSE;

  }

  U16 record_id = 65535;

  if (!stream->put16bitsLE((U8*)&record_id))

  {

    fprintf(stderr,"ERROR: writing EVLR record_id\n");

    return FALSE;

  }

  I64 record_length_after_header = 0;

  if (!stream->put64bitsLE((U8*)&record_length_after_header))

  {

    fprintf(stderr,"ERROR: writing EVLR record_length_after_header\n");

    return FALSE;

  }

  I8 description[32];

  memset(description, 0, 32);

  sprintf(description, "%s by LAStools (%d)", (compressed ? "compressed" : "created"), LAS_TOOLS_VERSION);  

  if (!stream->putBytes((U8*)description, 32))

  {

    fprintf(stderr,"ERROR: writing EVLR description\n");

    return FALSE;

  }

On Sat, Sep 14, 2013 at 12:38 AM, Evon Silvia <esi...@watershedsciences.com> wrote:

Hello LAS experts!

The LAS 1.3 and 1.4 specifications allow/require the storage of the waveform data packets in an external *.wdp file. All of this hinges on the interpretation of the text from page 13 of the 1.3 specification:

Byte offset to Waveform Packet Data: The waveform packet data are stored in the LAS file in an Extended Variable Length Record (or, optionally, in an auxiliary file). The Byte Offset represents the location of the start of this LIDAR points' waveform packet within the waveform data variable length record (or external file) relative to the beginning of the Waveform Packet Data header.

This seems to imply that the external file has a header, but I can't imagine what it might be. The EVLR header, perhaps? I see three possibilities when waveform data is stored externally:

1. There is no WDP header.
2. Include the EVLR header after the point data in the LAS file and the WDP file has no header.
3. Include the EVLR header at the beginning of the WDP file.

How do you interpret the specification? The implications for each possibility is as follows:

1. A 1.3 reader that knows waveform data exists might look for the header after the point data and not find it. The first waveform data packet would have an offset of 0 bytes.
2. The "header" of the waveform data is not with the data itself, and the waveform data packets in the WDP file have no header whatsoever. The first waveform data packet would have an offset of 0 bytes.
3. Any readers looking for the EVLR in a 1.3 LAS file wouldn't find it, and the first waveform data packet would have an offset of 60 bytes.

I'm leaning toward #1 as my interpretation, but if others interpret the specifications another way then my points' waveform data offsets would be completely wrong. Thanks!

Evon

--

Evon Silvia  Geomatics Specialist

P: 541-752-1204 | E: esi...@wsidata.com | W: wsidata.com 

WSI Corvallis, OR WSI Portland, OR WSI Oakland, CA

517 SW 2nd St., Suite 400, Corvallis, OR 97333

--
--
You are subscribed to the LAS room - a friendly place to discuss the specifications of the LAS format. This is a public forum for those who want LAS to succeed as an open standard. Here you can go on record with bug reports, suggestions, and concerns about the current and proposed specifications.
 
Visit this group at http://groups.google.com/group/lasroom
Post to this group with an email to las...@googlegroups.com
Unsubscribe by email to lasroom+u...@googlegroups.com
 
---
You received this message because you are subscribed to the Google Groups "The LAS room - a friendly place to discuss specifications of the LAS format " group.
To unsubscribe from this group and stop receiving emails from it, send an email to lasroom+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Paul....@leicaus.com]

Sent from Lotus Traveler

Martin Isenburg --- [PulseWaves] Re: [The LAS room] Where is the WDP Header? ---

--
--
Post to "PulseWaves" by email to pulse...@googlegroups.com
Unsubscribe by email to pulsewaves+...@googlegroups.com
Visit this group's message archives at http://pulsewaves.org
 
---
You received this message because you are subscribed to the Google Groups "PulseWaves - no pulse left behind" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pulsewaves+...@googlegroups.com.

[Mohsen Azadbakht]

Hi,

I'm working on LAS 1.4 f LiDAR data with an auxiliary WDP file including waveforms.

I guess there is no header written in WDP file & it can be accessed at the end of the LAS header. If you use "LASTOOL.exe" You can read it  (Wave Packets info on page 14 LAS 1.4 specification) by extracting lassinfo from the LASS file. It'll be appeared at the end of the LAS header (you can exclude Lass header & other info):

======================================

variable length header record 1 of 2:

  reserved             0

  user ID              'LASF_Projection'

  record ID            2112

  length after header  0

  description          'WKT description'

variable length header record 2 of 2:

  reserved             0

  user ID              'LASF_Spec'

  record ID            100

  length after header  26

  description          'WPD_1'

  index 1 bits/sample 0 compression 0 samples 0 temporal 0 gain 0, offset 0

WARNING: 2 points outside of header bounding box

number of last returns: 4015852

covered area in square units/kilounits: 2153296/2.15

point density: all returns 2.28 last only 1.86 (per square units)

      spacing: all returns 0.66 last only 0.73 (in units)

real min x smaller than header min x by 0.000310

real max z larger than header max z by 0.000447

========================================

If this is the case, I've got some questions:

When I'm converting lass to text (to see some of the records) by "lastool.exe" of LASTOOLS package, the problem is with some of the fields:

- "Byte offset to Waveform Packet Data": this field is supposed to represent the "location of the beginning of the waveform packet relative to the beginning of the file". But, in my case, all records of this field are ZERO!

- "Waveform packet size in bytes": again all records show the same number (200 bytes), while it's mentioned in LAS spec that: "each wave can be of different size".

It would be possible to read wave packets in WDP file based on these fields. But, I'm suspicious about these numbers really.

What do you suggest in order to make sure about this issue?

Regards,

Mohsen

--

**************************************************************

Mohsen Azadbakht

PhD Student, Dept. of Infrastructure Engineering
The University of Melbourne

Cooperative Research Centre for Spatial Information (CRCSI)

Ph.  +61- (03) 90358559

Mobile: +61- 469955518 

Email: m.aza...@student.unimelb.edu.au

**************************************************************

[Martin Isenburg]

Hello,

either a corrupt LAS 1.4 FWF file or an issue in my software. but this looks corrupt:

variable length header record 2 of 2:

  reserved             0

  user ID              'LASF_Spec'

  record ID            100

  length after header  26

  description          'WPD_1'

  index 1 bits/sample 0 compression 0 samples 0 temporal 0 gain 0, offset 0

one bit per sample and 0 samples per waveform ... this is not a valid wavepacket descriptor. i assume these values should have been set to either 8 or 16 bits per sample and then 200 or 100 samples. but if your offsets are all zero.

- "Byte offset to Waveform Packet Data": this field is supposed to represent the "location of the beginning of the waveform packet relative to the beginning of the file". But, in my case, all records of this field are ZERO!

this is clearly a bug. it should start at 60, 260, 460, 660, 860, ... based on all i have learned about the file so far.

- "Waveform packet size in bytes": again all records show the same number (200 bytes), while it's mentioned in LAS spec that: "each wave can be of different size".

it is often the case that the same size in bytes is repeated (for example for Leica output this is usually always 256). since no compression is used and there is only one descriptor this *must* always be the same size. so this looks okay.

if you do a logical copy of the first 500,000 bytes of each file (*.las and *.wdp) and send those clipped raw bytes to me I can take a quick look. That's how i helped debugging the broken Optech LMS LAS 1.4 exporter over the past few weeks. Seems to be finally fixed but no word yet when Optech will provide their customers with the proper export functionality. Leaves me and my client rather impatient ...

What system produced the LAS 1.4 FWF file that you have? Can you send a full lasinfo.exe and a full lasvalidate.exe report?

Regards,

Martin @LaserPulseWaves

[Alex Yeryomin]

>>  the broken Optech LMS LAS 1.4 exporter

 

In my opinoin, the LAS specification is broken, not the exporter…

 

Martin, thank you for your help with the issues related to waveforms!

 

Alex

 

 

[Mohsen]

Thanks Martin,

I'll send you a copy of the data I have, first thing tomorrow morning.

It's from Riegl for an area in Australia.

Regards,

Mohsen

Sent from my iPad

[Mohsen Azadbakht]

Hi Martin,

The full lasinfo.exe & lasvalidate.exe reports  are attached.

It's not possible to send 500000 bytes from .WDP file since I do not know the exact format of the file (to read it)!

I can share them with you on Dropbox if you want (~ 1.3GB together).

Regards,

Mohsen

[Martin Isenburg]

Hello,

so we have yet another major LiDAR company with a faulty LAS 1.3/1.4 full waveform implementation? Oh boy ... (-: Welcome to the not-so-elusive club, Trimble! With three major players - first RiPROCESS (fixed 1.5 years ago), then Optech LMS (fix still on-going), and now Trimble's LPMaster (bug just discovered) - having miss-implemented the FWF part of the LAS specification I think it is safe to say we really need to fix the description of LAS FWF in the official ASPRS specs ... 

Any trembling Trimblers on this list? Let's fix this! (-:

Martin

PS: for these purposes i wrote a simple copy tool called bytecopy.exe (10 lines of C code) that reads files byte after byte (fgetc) and writes them again (fputc) for "logical" copying of a subset of the file. i will send it to you ...

[Martin Isenburg]

Hello,

in the meantime Mohsen has shared the first 500,000 bytes of the LAS and the WDP file with me (using the bytecopy.exe that is now part of the latest lastools.zip distribution) and I could confirm that the LAS 1.4 FWF component of the gigantic LAS 1.4 file that he received is complete junk ... or some mysteriously novel  interpretation of the LAS FWF specification. Here a look at the xyz coordinate, the return number, the number of returns, the scan angle and the contents of the wavepacket for the first few points in the file:

D:\lastools\bin>las2txt -i test1.las -stdout -parse xyzrnaW

WARNING: no payload for LASF_Projection (not specification-conform).

WARNING: bits per sample for wave packet descr 1 is 0 instead of 8 or 16

WARNING: number of samples for wave packet descr 1 is zero

WARNING: temporal spacing for wave packet descr 1 is zero

245435.895 7416041.183 14.608 1 1 -30 1 0 200 18999.8 2.20397e-014 -1.76163e-014 4.54641e-014

245437.835 7416039.676 16.941 1 2 -30 1 0 200 18000 2.32517e-014 -1.85876e-014 4.80648e-014

245436.750 7416040.543 14.699 2 2 -30 1 0 200 35000 1.15128e-014 -9.20342e-015 2.37987e-014

245437.583 7416039.922 14.715 1 1 -30 1 0 200 18999.9 2.19435e-014 -1.75443e-014 4.54577e-014

245438.376 7416039.330 14.732 1 1 -30 1 0 200 15999.9 2.64043e-014 -2.11137e-014 5.48099e-014

245439.202 7416038.714 14.738 1 1 -30 1 0 200 16000.1 2.63474e-014 -2.10712e-014 5.48088e-014

245439.997 7416038.120 14.769 1 1 -30 1 0 200 17000.1 2.46053e-014 -1.96806e-014 5.12891e-014

245444.827 7416034.300 23.190 1 2 -30 1 0 200 15000 2.78735e-014 -2.22978e-014 5.82227e-014

245440.782 7416037.536 14.743 2 2 -30 1 0 200 81000 4.80736e-015 -3.84572e-015 1.00417e-014

245441.572 7416036.946 14.762 1 1 -29 1 0 200 18000 2.30264e-014 -1.84229e-014 4.81964e-014

245442.397 7416036.331 14.772 1 1 -29 1 0 200 18000.2 2.29764e-014 -1.83855e-014 4.81952e-014

245443.195 7416035.736 14.765 1 1 -29 1 0 200 18999.9 2.16245e-014 -1.73062e-014 4.54549e-014

Maybe this can be fixed. If we repaired the Waveform Packet Descriptor in the VLR with the most likely values (e.g. 8 bits / 200 samples or 16 bits / 100 samples and temporal spacing of 1000 picoseconds), knew how to rescale the strange direction vectors, and rewrite the offsets of the packet descriptor to be the sequence 60, 260, 460, 660, 860, ... or (in case there is no header in the WDP file) to 0, 200, 400, 600, 800 ... this might be salvageable. The points (without waveforms) are okay as the attached image clearly shows. I think the waveforms are fine (as far as I understand there is a RIEGL scanner inside a Trimble system) so it is only the exporter of the processing software that is broken:

  system identifier:          'H56_010-009 EXTRACTION'

  generating software:        'Trimble LPMaster'

  file creation day/year:     46/2013

Regards,

Martin @LaserPulseWaves

[Mohsen Azadbakht]

Hi Martin,

I worked on reading the first couple of bytes of the WDP file that I already sent you, with 4 different scenarios:

1-16 bits per 100 samples, without skipping the first 60 bytes 

2-16 bits per 100 samples, after skipping the first 60 bytes ( as WDP header)

3-16 bits per 200 samples, without skipping the first 60 bytes 

4-16 bits per 200 samples, after skipping the first 60 bytes ( as WDP header)

The best one, according to the *las file records (Number of Returns & Return Point Waveform Location) is the first option(the results are attached).

However, the problem with this one is that, those waveforms with more than one pulse (e.g. the second waveform) are repeated! I can't figure out the reason.

--------------------------------------------------las records for the first 5 waveforms---------------------

245435.895 7416041.183 14.608 1 1 -30 1 0 200 18999.8 2.20397e-014 -1.76163e-014 4.54641e-014

245437.835 7416039.676 16.941 1 2 -30 1 0 200 18000 2.32517e-014 -1.85876e-014 4.80648e-014

245436.750 7416040.543 14.699 2 2 -30 1 0 200 35000 1.15128e-014 -9.20342e-015 2.37987e-014

245437.583 7416039.922 14.715 1 1 -30 1 0 200 18999.9 2.19435e-014 -1.75443e-014 4.54577e-014

245438.376 7416039.330 14.732 1 1 -30 1 0 200 15999.9 2.64043e-014 -2.11137e-014 5.48099e-014

245439.202 7416038.714 14.738 1 1 -30 1 0 200 16000.1 2.63474e-014 -2.10712e-014 5.48088e-014

-----------------------------------------------------------------------------------------------------------------

Regards,

Mohsen