Offset data

[Susana Gonzalez]

Hi there,

I have received some FWF data in WGS84 zone 60S with an offset

 

 

Is this offset “fixable”? Do we think that the offset is created when the data is exported from Trimble LPMaster? Or may be the offset is from the raw files, something to do with the sensor?

 

I have uploaded a sample file here

 

And this is the script that I was using

 

set FWF=R:\Fullwaveform

set OUTPUT=R:\Fullwaveform

 

lasview -i %FWF%\*.las

 

lasinfo.exe -i %FWF%\*.las -odir %OUTPUT% -otxt

 

:: I didn’t repair the boundary box     -repair_bb

 

laszip.exe -i %FWF%\*.las -waveform -o %OUTPUT%\cld_170704_01_223748_001_zipped.laz

 

las2las.exe -i %FWF%\*_zipped.laz -rescale 0.01 0.01 0.01 -remove_vlrs_from_to 4 10 -o %FWF%\cld_170704_01_223748_001.laz

 

lasinfo.exe -i %FWF%\cld_170704_01_223748_001.laz -odix _compress -odir %OUTPUT% -otxt

 

Thanks

Susana

 

Susana Gonzalez - Forest Engineer, LiDAR Science

 

Interpine Group Ltd

Mobile:      021 026 49733   DDI:  +64 7 350 3209 or Australia 0280113645 ext 722
Skype:        sugonar

Address :   99 Sala Street, PO Box 1209, Rotorua 3010, New Zealand
Website :   www.interpine.co.nz

 

Interpine Innovation is Shaping Today’s Forests with the Technology of Tomorrow

He rangahau tenei ra he hangarau apopo

 

[Martin Isenburg]

Hello,

I suspect that the location at which the waveforms are stored in the WDP file by the Trimble exporter as well as the corresponding waveform offsets stored in the LAS file by the Trimble exporter might be in violation with the intended use of waveforms store in an external file. It is my understanding - and I may have miss-understood the intended use of the LAS FWF extension - that the very first digitized waveform data packet can start no earlier than at byte 60 of the external WDP file because the first 60 bytes of the external WDP file should contain the EVLR header but in the Trimble file you provided they start at 0 (see red zero below). It was my understanding that the external storage does not refer to only the data but the entire EVLR containing the waveform block. It seems to me that if you use LAStools to re-write the waveforms that are wrongly (?!?) stored in an external WDP without the EVLR header and starting at byte 0 rather than from byte 60, then LAStools will introduce the 60 byte offset that is supposed to be there without fixing the wrongly stored numbers for waveform offsets that start at 0 in the EVLR.

Using '-waveforms_with_map' instead of '-waveforms' seems to compress it correctly but as LAStools starts the first waveform at 60 instead of at 0 the first waveform will point to the wrong location in the WDP file and be corrupted after compression (see below). I can fix this but we should also figure out whether my assumption that the EVLR header needs to occupy the first 60 bytes of the WDP file is true or not.

Regards,

Martin

------------------------------------

C:\LAStools\bin>las2txt -stdout -parse WV -i cld_170704_01_223748_001.las | more
WARNING: number of samples for wave packet descr 4 is zero
WARNING: number of samples for wave packet descr 5 is zero
WARNING: number of samples for wave packet descr 6 is zero
WARNING: number of samples for wave packet descr 7 is zero
WARNING: number of samples for wave packet descr 8 is zero
WARNING: number of samples for wave packet descr 9 is zero
WARNING: number of samples for wave packet descr 10 is zero
1 0 160 16640.1 -3.61606e-005 -5.96808e-005 0.00013266 16 80 2 2 1 2 1 1 1 2 1 2 2 2 3 5 13 27 45 57 54 39 22 10 6 5 3 3 4 4 5 6 4 4 3 2 3 3 3 3 3 3 2 2 2 2 3 3 2 1 0 1 2 2 2 2 2 2 3 2 2 3 3 2 2 2 2 3 3 2 3 2 1 2 3 2 2 2 2 2 2 1
1 160 160 17640.2 -3.61265e-005 -5.96083e-005 0.000132702 16 80 3 2 3 3 2 2 2 2 2 2 2 2 3 2 6 14 29 48 60 57 41 24 10 6 3 3 3 3 5 6 5 4 3 3 4 4 4 4 3 3 4 3 3 3 3 3 2 2 3 2 3 3 3 4 3 3 2 2 1 2 2 2 2 3 3 2 1 1 1 1 1 2 2 2 2 1 0 2 2 2
1 320 160 16639.7 -3.60943e-005 -5.95395e-005 0.000132744 16 80 0 2 2 2 2 1 1 2 3 3 2 3 3 6 12 26 43 56 55 41 24 11 6 5 4 4 5 5 6 5 4 3 4 4 4 3 2 2 3 3 2 3 3 3 3 2 2 2 3 2 3 2 3 4 4 3 2 2 2 3 3 3 2 3 3 2 1 0 1 2 2 3 3 3 3 3 3 3 2 2
1 480 160 15640.1 -3.60628e-005 -5.94711e-005 0.000132788 16 80 2 2 2 3 2 3 3 2 3 4 5 7 8 10 14 19 23 22 19 14 12 9 6 5 3 2 3 3 3 3 4 3 4 4 4 3 3 3 2 2 3 4 2 3 4 4 4 3 2 0 2 2 3 2 2 3 2 3 2 3 3 3 3 3 2 2 2 2 2 2 3 3 3 3 3 3 2 2 2 1
[...]

C:\LAStools\bin>laszip.exe -i cld_170704_01_223748_001.las -waveforms_with_map -o cld_170704_01_223748_001_zipped.laz

C:\LAStools\bin>las2txt -stdout -parse WV -i  cld_170704_01_223748_001_zipped.laz | more
WARNING: number of samples for wave packet descr 4 is zero
WARNING: number of samples for wave packet descr 5 is zero
WARNING: number of samples for wave packet descr 6 is zero
WARNING: number of samples for wave packet descr 7 is zero
WARNING: number of samples for wave packet descr 8 is zero
WARNING: number of samples for wave packet descr 9 is zero
WARNING: number of samples for wave packet descr 10 is zero
1 0 0 16640.1 -3.61606e-005 -5.96808e-005 0.00013266 16 80 43707 43678 43679 30838 30542 28419 28443 28441 28303 42151 42145 42152 38322 35779 31403 33275 33259 48713 48682 48799 51445 51442 51446 27797 28054 28072 28189 28190 28194 53717 53691 51537 2206 2232 2219 33836 31376 40667 41107 41094 41074 41066 41069 53330 60426 60426 60375 60363 60368 60369 60362 60355 60367 6703 6209 9921 11741 63104 63103 3062 10778 25099 25108 25107 25078 26144 26267 26442 57759 57759 59367 49905 49899 7646 32506 27390 27386 27384 27385 27395
1 146 42 17640.2 -3.61265e-005 -5.96083e-005 0.000132702 16 80 3 2 3 3 2 2 2 2 2 2 2 2 3 2 6 14 29 48 60 57 41 24 10 6 3 3 3 3 5 6 5 4 3 3 4 4 4 4 3 3 4 3 3 3 3 3 2 2 3 2 3 3 3 4 3 3 2 2 1 2 2 2 2 3 3 2 1 1 1 1 1 2 2 2 2 1 0 2 2 2
1 188 41 16639.7 -3.60943e-005 -5.95395e-005 0.000132744 16 80 0 2 2 2 2 1 1 2 3 3 2 3 3 6 12 26 43 56 55 41 24 11 6 5 4 4 5 5 6 5 4 3 4 4 4 3 2 2 3 3 2 3 3 3 3 2 2 2 3 2 3 2 3 4 4 3 2 2 2 3 3 3 2 3 3 2 1 0 1 2 2 3 3 3 3 3 3 3 2 2
1 229 41 15640.1 -3.60628e-005 -5.94711e-005 0.000132788 16 80 2 2 2 3 2 3 3 2 3 4 5 7 8 10 14 19 23 22 19 14 12 9 6 5 3 2 3 3 3 3 4 3 4 4 4 3 3 3 2 2 3 4 2 3 4 4 4 3 2 0 2 2 3 2 2 3 2 3 2 3 3 3 3 3 2 2 2 2 2 2 3 3 3 3 3 3 2 2 2 1
[...]  

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe
---
You received this message because you are subscribed to the Google Groups "LAStools - efficient tools for LiDAR processing" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lastools+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lastools/SG2PR03MB502235B999FC956CC53998688DE70%40SG2PR03MB5022.apcprd03.prod.outlook.com.

[Martin Isenburg]

Hello Susana,

use this new laszip.exe:

https://www.dropbox.com/s/awfcbtdkfy4tc0o/laszip.exe

which can handle the first waveform data packet being stored with an offset of zero in the external WDP file:

C:\LAStools\bin>laszip -version
LAStools (by mar...@rapidlasso.com) version 190621

C:\LAStools\bin>laszip.exe -i cld_170704_01_223748_001.las -waveforms -o cld_170704_01_223748_001_zipped.laz

WARNING: number of samples for wave packet descr 4 is zero
WARNING: number of samples for wave packet descr 5 is zero
WARNING: number of samples for wave packet descr 6 is zero
WARNING: number of samples for wave packet descr 7 is zero
WARNING: number of samples for wave packet descr 8 is zero
WARNING: number of samples for wave packet descr 9 is zero
WARNING: number of samples for wave packet descr 10 is zero

C:\LAStools\bin>las2txt -stdout -parse WV -i  cld_170704_01_223748_001_zipped.laz | more
WARNING: number of samples for wave packet descr 4 is zero
WARNING: number of samples for wave packet descr 5 is zero
WARNING: number of samples for wave packet descr 6 is zero
WARNING: number of samples for wave packet descr 7 is zero
WARNING: number of samples for wave packet descr 8 is zero
WARNING: number of samples for wave packet descr 9 is zero
WARNING: number of samples for wave packet descr 10 is zero

1 146 42 16640.1 -3.61606e-005 -5.96808e-005 0.00013266 16 80 2 2 1 2 1 1 1 2 1 2 2 2 3 5 13 27 45 57 54 39 22 10 6 5 3 3 4 4 5 6 4 4 3 2 3 3 3 3 3 3 2 2 2 2 3 3 2 1 0 1 2 2 2 2 2 2 3 2 2 3 3 2 2 2 2 3 3 2 3 2 1 2 3 2 2 2 2 2 2 1
1 188 42 17640.2 -3.61265e-005 -5.96083e-005 0.000132702 16 80 3 2 3 3 2 2 2 2 2 2 2 2 3 2 6 14 29 48 60 57 41 24 10 6 3 3 3 3 5 6 5 4 3 3 4 4 4 4 3 3 4 3 3 3 3 3 2 2 3 2 3 3 3 4 3 3 2 2 1 2 2 2 2 3 3  2 1 1 1 1 1 2 2 2 2 1 0 2 2 2
1 230 41 16639.7 -3.60943e-005 -5.95395e-005 0.000132744 16 80 0 2 2 2 2 1 1 2 3 3 2 3 3 6 12 26 43 56 55 41 24 11 6 5 4 4 5 5 6 5 4 3 4 4 4 3 2 2 3 3 2 3 3 3 3 2 2 2 3 2 3 2 3 4 4 3 2 2 2 3 3 3 2 3 3  2 1 0 1 2 2 3 3 3 3 3 3 3 2 2
1 271 41 15640.1 -3.60628e-005 -5.94711e-005 0.000132788 16 80 2 2 2 3 2 3 3 2 3 4 5 7 8 10 14 19 23 22 19 14 12 9 6 5 3 2 3 3 3 3 4 3 4 4 4 3 3 3 2 2 3 4 2 3 4 4 4 3 2 0 2 2 3 2 2 3 2 3 2 3 3 3 3 3 2 2 2 2 2 2 3 3 3 3 3 3 2 2 2 1

[...]

[Susana Gonzalez]

Hi Martin,

Thank you for your answer.

 

All WDP file that now are corrupted after the compression, can be revers?

 

Thanks

Susana

To view this discussion on the web visit https://groups.google.com/d/msgid/lastools/CABSWR-G1CriM1VsxZh2QxRKOu8V%2B%2B2hUe3DSdDVN%3DZXacXORyg%40mail.gmail.com.

[Evon Silvia]

Martin --

Your reading of the FWF sections of the LAS 1.4 specification is correct.

Quote from §2.6.5 (emphasis added):

The waveform packet data are stored in the LAS file in an Extended Variable Length Record or in an auxiliary *.wdp file. The Byte Offset represents the location of the start of this Point Record's waveform packet within the waveform data variable length record (or external file) relative to the beginning of the Waveform Data Packets header.

This links to §5.1. Quote here (emphasis added):

The packet of Raw Waveform Amplitude values for all records immediately follow this VLR header. [...]

The implication here is that the external *.wdp file has a header just like a normal EVLR would, which I believe is helpful since it self-describes its own length and the user can therefore detect truncation / data loss.

Therefore, the bare minimum FWF Byte Offset is 60 – the size of a EVLR header. Based on this apparent misunderstanding, we could probably afford to clarify this in the LAS wiki.

Evon

--

Evon Silvia PLS QSI Solutions Architect ASPRS LAS Working Group Chair 1100 NE Circle Blvd, Suite 126, Corvallis, OR 97330 P: (541) 249-5818 E: esi...@quantumspatial.com

To view this discussion on the web visit https://groups.google.com/d/msgid/lastools/SG2PR03MB5022324C51D2507B7A5FB8DA8DE00%40SG2PR03MB5022.apcprd03.prod.outlook.com.

[Martin Isenburg]

Hello Susana,

only the very first return is pointing to a corrupt waveform after compressing the file with the '-waveforms' option of laszip.exe (version 190604 and older). One way of "fixing" the LAZ+ WDZ files would be to simply drop the very first return of every LAZ file with las2las and the '-subseq 1 999999999' option. Ideally you still have the original (yet it seems also corrupt as confirmed by Evon) LAS FWF files from Trimble and recompress them with the latest version of laszip.exe (version 190623 or newer) which should fix the corrupted file and produce specification conform output that starts writing waveforms from byte 60 and not from byte 0. Third option would be me writing a special tool to recover this (in case it is worth a lot to you (-:). 

Regards,

Martin

To view this discussion on the web visit https://groups.google.com/d/msgid/lastools/SG2PR03MB5022324C51D2507B7A5FB8DA8DE00%40SG2PR03MB5022.apcprd03.prod.outlook.com.

[Susana Gonzalez]

Hi Martin (and Evon),

Thanks for all your help.

Yes, I still have the originals (7 disks of 6TB each)

It’s all good, thanks again.

To view this discussion on the web visit https://groups.google.com/d/msgid/lastools/CABSWR-G7UQiBw1uPgkVx1evEsfLdt4nF%2BWcoaUoNfiHLnSFqag%40mail.gmail.com.