Importing ascii with txt2las / Missing values/ Warnings

Paul Magdon

unread,

Mar 23, 2017, 8:24:30 AM3/23/17

to LAStools - efficient tools for LiDAR processing

Hi,

I am trying to convert an ascii file to a las file. The ascii data comes like this (x,y,z,intensity,beam width,???,return,number of returns):

4397946.84 5693254.11 495.07 16 40 3 4 4
4397941.42 5693255.92 529.49 25 49 3 1 3
4397942.22 5693255.66 524.31 25 71 3 2 3
4397946.68 5693254.22 495.05 78 42 3 3 3
4397941.30 5693256.02 529.27 53 43 3 1 2
4397946.51 5693254.34 495.04 116 42 3 2 2
4397940.78 5693256.24 531.72 25 50 3 1 3
4397941.60 5693255.98 526.31 20 *** 3 2 3

txt2las -i ascii.xyz -olas -odir .\tmp -parse xyzi0srn -add_attribute 3 "echo width" "beam with" -set_version 1.2

Which works o.k. except the following warnings and observations:

1.) I get always a warning: WARNING: written 10247677 points but expected 0 points.
What does this say? Why is 0 expected

2.) In some cases I get the warning that the return number of 7 can not be coded with 3 bit.
How can I change the bit number to e.g. 4 for returns and #returns?

3.) For some reason which I don't know, some of the intensity values the text file include '***' as in the last line of the example.
These points are completly skipped by txt2las. Is there an option to set these values to NA?

4.) The original ascii files have the ending *asc. However, if I try to import from these I get the following:
ERROR: was not able to find header
ERROR: cannot open lasreaderasc with file name '4374386_4375598_5683610_5685092_Spur0171.asc'
ERROR: could not open lasreader
My current solution is to rename the files to *xyz. Any better options?

I would appriciate any help.

Cheers Paul

Martin Isenburg

unread,

Mar 23, 2017, 10:17:36 AM3/23/17

to LAStools - efficient command line tools for LIDAR processing

Hello Paul,

I am trying to convert an ascii file to a las file. The ascii data comes like this (x,y,z,intensity,beam width,???,return,number of returns):

4397946.84 5693254.11 495.07 16 40 3 4 4
4397941.42 5693255.92 529.49 25 49 3 1 3
4397942.22 5693255.66 524.31 25 71 3 2 3
4397946.68 5693254.22 495.05 78 42 3 3 3
4397941.30 5693256.02 529.27 53 43 3 1 2
4397946.51 5693254.34 495.04 116 42 3 2 2
4397940.78 5693256.24 531.72 25 50 3 1 3
4397941.60 5693255.98 526.31 20 *** 3 2 3

txt2las -i ascii.xyz -olas -odir .\tmp -parse xyzi0srn -add_attribute 3 "echo width" "beam with" -set_version 1.2

Seems to be true that scientists spend 90% of their time of converting their data from one format to another ... (-:

Note ... the value '3' that you make '???' might be the point source ID. As you have no GPS time stamps your only hope is the point source IDs for being able to do quality checks (flightline alignment and overlap). Unless your input files *are* flight lines.

Which works o.k. except the following warnings and observations:

1.) I get always a warning: WARNING: written 10247677 points but expected 0 points.
What does this say? Why is 0 expected

This is just the way that LASlib is setup right now. A LAS writer needs to write the header first and thus has to populate the field specifying the number of points. For ASCII conversion that knowledge would require an extra pass just to count the points and compute the bounding box (which is also an option and needed when *piping* the output into a LAS file or another processing stage). Hence the number is simply written as zero. When LASlib notices that the number of points written is different from the number initially specified it will fix the header (unless it is *piping* when seeking back is not possible) but report a WARNING. I guess I could omit the WARNING when the "expectation" was 0 as that should always indicate that the value was not known ahead of time.

2.) In some cases I get the warning that the return number of 7 can not be coded with 3 bit.
How can I change the bit number to e.g. 4 for returns and #returns?

But a return number of 7 can be coded with 3 bits. We need an example here.

3.) For some reason which I don't know, some of the intensity values the text file include '***' as in the last line of the example.
These points are completly skipped by txt2las. Is there an option to set these values to NA?

The parser looks for the right number of values. If they are not found the line is skipped. You could search/replace those to (possibly via a pipe) with 0 and then declare 0 to be the nodata value when you specify the "extra bytes".

4.) The original ascii files have the ending *asc. However, if I try to import from these I get the following:
ERROR: was not able to find header
ERROR: cannot open lasreaderasc with file name '4374386_4375598_5683610_5685092_Spur0171.asc'
ERROR: could not open lasreader
My current solution is to rename the files to *xyz. Any better options?

The *.asc ending makes LASlib expect an ASCII *.asc raster file. Currently there is no option to *force* the *.txt reader onto a *.asc file but it may be a good idea to rename those. I seem to remember the same issue when I got the LiDAR from "Nationalpark Bayrischer Wald" ... simply run

rename *.asc *.txt

or

rename *.asc *.xyz

in the Windows command line to give them more appropriate names.

Regards,

Martin @rapidlasso

I would appriciate any help.

Cheers Paul

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe

Paul Magdon

unread,

Mar 24, 2017, 8:18:15 AM3/24/17

to LAStools - efficient tools for LiDAR processing

Hi Martin et al.,

Seems to be true that scientists spend 90% of their time of converting their data from one format to another ... (-:

I fully agree here :).

Note ... the value '3' that you make '???' might be the point source ID. As you have no GPS time stamps your only hope is the point source IDs for being able to do quality checks (flightline alignment and overlap). Unless your input files *are* flight lines.

Thanx for the hint. I changed the command to :

txt2las -i %data%\*.xyz -olas -odir .\tmp -parse xyzipsrn -add_attribute 3 "echo width" "beam with" -set_version 1.2

Which runs fine. So I guess that the old '???' was the point source ID. What would be the best 'lastools' option to check that?

Which works o.k. except the following warnings and observations:

1.) I get always a warning: WARNING: written 10247677 points but expected 0 points.
What does this say? Why is 0 expected

This is just the way that LASlib is setup right now. A LAS writer needs to write the header first and thus has to populate the field specifying the number of points. For ASCII conversion that knowledge would require an extra pass just to count the points and compute the bounding box (which is also an option and needed when *piping* the output into a LAS file or another processing stage). Hence the number is simply written as zero. When LASlib notices that the number of points written is different from the number initially specified it will fix the header (unless it is *piping* when seeking back is not possible) but report a WARNING. I guess I could omit the WARNING when the "expectation" was 0 as that should always indicate that the value was not known ahead of time.

Thanks for the explanation. Makes sense now.

2.) In some cases I get the warning that the return number of 7 can not be coded with 3 bit.
How can I change the bit number to e.g. 4 for returns and #returns?

But a return number of 7 can be coded with 3 bits. We need an example here.

Sorry that was a typo. The warning says : 8 can not be coded with 3 bits

3.) For some reason which I don't know, some of the intensity values the text file include '***' as in the last line of the example.
These points are completly skipped by txt2las. Is there an option to set these values to NA?

The parser looks for the right number of values. If they are not found the line is skipped. You could search/replace those to (possibly via a pipe) with 0 and then declare 0 to be the nodata value when you specify the "extra bytes".

Ok, I replaced the '***' by '0' now using notepad++ (seems to much faster then any regular expression in cmd or powershell). How can I declare 0 as no data for intensity using the 'extra bytes' ?

4.) The original ascii files have the ending *asc. However, if I try to import from these I get the following:
ERROR: was not able to find header
ERROR: cannot open lasreaderasc with file name '4374386_4375598_5683610_5685092_Spur0171.asc'
ERROR: could not open lasreader
My current solution is to rename the files to *xyz. Any better options?

The *.asc ending makes LASlib expect an ASCII *.asc raster file. Currently there is no option to *force* the *.txt reader onto a *.asc file but it may be a good idea to rename those. I seem to remember the same issue when I got the LiDAR from "Nationalpark Bayrischer Wald" ... simply run

rename *.asc *.txt

or

rename *.asc *.xyz

in the Windows command line to give them more appropriate names.

Exactly what I have done :)

Greetings & thank you very much for your support.
Paul

Martin Isenburg

unread,

Mar 24, 2017, 11:29:20 AM3/24/17

to LAStools - efficient command line tools for LIDAR processing

Hello,

So I guess that the old '???' was the point source ID. What would be the best 'lastools' option to check that?

Are your files tiles or flightlines? You can visualize the numbers stored in the point source ID as color-coded flight lines with

lasview -i lidar.laz -color_by_flightline

and print a histogram with

lasinfo -i lidar.laz -nh -nv -nr -nmm -histo point_source 1

2.) In some cases I get the warning that the return number of 7 can not be coded with 3 bit. How can I change the bit number to e.g. 4 for returns and #returns?

But a return number of 7 can be coded with 3 bits. We need an example here.

Sorry that was a typo. The warning says : 8 can not be coded with 3 bits

If you want to properly store 8 or more returns to a LAS / LAZ file you will need to use point type 6 of LAS 1.4 that supports up to 15 returns per pulse. Would that be an option?

3.) For some reason which I don't know, some of the intensity values the text file include '***' as in the last line of the example.
These points are completly skipped by txt2las. Is there an option to set these values to NA?

The parser looks for the right number of values. If they are not found the line is skipped. You could search/replace those to (possibly via a pipe) with 0 and then declare 0 to be the nodata value when you specify the "extra bytes".

Ok, I replaced the '***' by '0' now using notepad++ (seems to much faster then any regular expression in cmd or powershell). How can I declare 0 as no data for intensity using the 'extra bytes' ?

It is not the intensity value column that has the '***' but the "echo width" attribute. There would not have been option an option to set a "nodata" value for the intensity but you can do it for anything stored as an "extra bytes" attribute. This is currently not supported by txt2las but I can easily add this. The question is ... would it help you to set this "no data" value in your processing? I am not aware of any software that already reads and evaluates this value. Having an obviously impossible "echo width" of zero may already be a good enough "no data" value.

Also ... it seems your values for echo width are in tenth of a nanosecond. Here my suggestion for your parse line that first scales these numbers with 0.1 and then stores them with a scale factor of 0.1.

txt2las -i paul.xyz ^

-set_version 1.2 ^

-parse xyzi0prn ^

-add_attribute 3 "echo width" "Full width at half maximum [ns]" 0.1 0.0 0.1 ^

-olaz

las2txt -i paul.laz ^

-parse xyzi0prn ^

-stdout

4397946.84 5693254.11 495.07 16 4.0 3 4 4

4397941.42 5693255.92 529.49 25 4.9 3 1 3

4397942.22 5693255.66 524.31 25 7.1 3 2 3

4397946.68 5693254.22 495.05 78 4.2 3 3 3

4397941.30 5693256.02 529.27 53 4.3 3 1 2

4397946.51 5693254.34 495.04 116 4.2 3 2 2

4397940.78 5693256.24 531.72 25 5.0 3 1 3

4397941.60 5693255.98 526.31 20 0.0 3 2 3

lasinfo -i paul.laz

lasinfo (170322) report for paul.laz

reporting all LAS header entries:

file signature: 'LASF'

file source ID: 0

global_encoding: 0

project ID GUID data 1-4: 00000000-0000-0000-0000-000000000000

version major.minor: 1.2

system identifier: 'LAStools (c) by rapidlasso GmbH'

generating software: 'txt2las (version 170322)'

file creation day/year: 83/2017

header size: 227

offset to point data: 473

number var. length records: 1

point data format: 0

point data record length: 22

number of point records: 8

number of points by return: 3 3 1 1 0

scale factor x y z: 0.01 0.01 0.01

offset x y z: 4300000 5600000 0

min x y z: 4397940.78 5693254.11 495.04

max x y z: 4397946.84 5693256.24 531.72

variable length header record 1 of 1:

reserved 0

user ID 'LASF_Spec'

record ID 4

length after header 192

description 'by LAStools of rapidlasso GmbH'

Extra Byte Descriptions

data type: 3 (unsigned short), name "echo width", description: "Full width at half maximum [ns]", scale: 0.1, offset: 0 (not set)

LASzip compression (version 2.5r2 c2 50000): POINT10 2 BYTE 2

reporting minimum and maximum for all LAS point record entries ...

X 9794078 9794684

Y 9325411 9325624

Z 49504 53172

intensity 16 116

return_number 1 4

number_of_returns 2 4

edge_of_flight_line 0 0

scan_direction_flag 0 0

classification 0 0

scan_angle_rank 0 0

user_data 0 0

point_source_ID 3 3

number of first returns: 3

number of intermediate returns: 2

number of last returns: 3

number of single returns: 0

overview over number of returns of given pulse: 0 2 5 1 0 0 0

histogram of classification of points:

8 never classified (0)

paul.xyz

Paul Magdon

unread,

Apr 4, 2017, 10:36:29 AM4/4/17

to LAStools - efficient tools for LiDAR processing

Hi Martin,

thank you for you support.

1.) I tried your suggestion with the point_source ID. However, it seems to be somethink different as the source point ID. I tried

lasview -i lidar.laz -color_by_flightline

which resulted in:

and the histogram look like this

lasinfo -i lidar.laz -nh -nv -nr -nmm -histo point_source 1

lasinfo (170203) report for mpi\02_las\tmp\4375734_4376672_5683752_5685293_Spur0176.las
point source id histogram with bin size 1
bin 3 has 10477442
bin 4 has 33871
bin 5 has 31259
average point source id 3.00914 for 10542572 element(s)

Thus, the 6. column seem to be somethink different the point source. The files are flight lines which have been cutted. They are name e.g. 4375734_4376672_5683752_5685293_Spur0176.las; 4375419_4376421_5684493_5685769_Spur0177.las. Thus I guess the fliename '_Spur0176' would provide the point source ID. Any option in lastools to extract the point source ID from the file name when running txt2las for a large set of files? The meaning of the 6. column remains unclear any other idea what this could be?

2.)

If you want to properly store 8 or more returns to a LAS / LAZ file you will need to use point type 6 of LAS 1.4 that supports up to 15 returns per pulse. Would that be an option?

Yes 1.4 would also be an option. However, when I run:
txt2las -i paul.xyz -parse xyzi0prn -set_version 1.4 -add_attribute 3 "echo width" "Full width at half maximum [ns]" 0.1 0.0 0.1 -olas -odir E:\LiDAR\google -odix _14
and then
lasinfo -i google\paul_14.las
I get a lot of warnings aggain:
lasinfo (170203) report for google\paul_14.las
WARNING: only 0 bytes until point block after reading 0 of 1 vlrs. skipping remaining vlrs ...

reporting all LAS header entries:
file signature:             'LASF'
file source ID:             0
global_encoding:            0
project ID GUID data 1-4:   00000000-0000-0000-0000-000000000000

version major.minor: 1.4

system identifier: 'LAStools (c) by rapidlasso GmbH'

generating software:        'txt2las (version 170203)'
file creation day/year:     94/2017
header size:                375
offset to point data:       375
number var. length records: 0

point data format:          0
point data record length:   22
number of point records:    8
number of points by return: 3 3 1 1 0
scale factor x y z:         0.01 0.01 0.01
offset x y z:               4300000 5600000 0
min x y z:                  4397940.78 5693254.11 495.04
max x y z:                  4397946.84 5693256.24 531.72

start of waveform data packet record: 0
start of first extended variable length record: 0
number of extended_variable length records: 0
extended number of point records: 8
extended number of points by return: 3 3 1 1 0 0 0 0 0 0 0 0 0 0 0
WARNING: 8 points outside of header bounding box
WARNING: for return 1 real number of points by return (1) is different from header entry (3).
WARNING: for return 2 real number of points by return (0) is different from header entry (3).
WARNING: for return 3 real number of points by return (0) is different from header entry (1).
WARNING: for return 4 real number of points by return (0) is different from header entry (1).
WARNING: for return 5 real number of points by return (1) is different from header entry (0).
WARNING: real extended number of points by return [1] is 1 - different from header entry 3.
WARNING: real extended number of points by return [2] is 0 - different from header entry 3.
WARNING: real extended number of points by return [3] is 0 - different from header entry 1.
WARNING: real extended number of points by return [4] is 0 - different from header entry 1.
WARNING: real extended number of points by return [5] is 1 - different from header entry 0.
real max x larger than header max x by 12674019.580000
real min x smaller than header min x by 97940.780000
real max y larger than header max y by 18605688.410000
real min y smaller than header min y by 93254.110000
real max z larger than header max z by 5443821.390000
real min z smaller than header min z by 495.040000

How to interpret this warnings?

Pura Vida !

Paul

Martin Isenburg

unread,

Apr 5, 2017, 8:52:20 AM4/5/17

to LAStools - efficient command line tools for LIDAR processing

Hello Paul,

true. The conversion to the new LAS 1.4 point types did indeed require some fixing. There is now a new option '-set_point_type 6' that will allow point types 0, 1, 2, 3, 6, 7, and 8 automatically pick either LAS 1.2 or LAS 1.4. In the next release release this will work:

E:\LAStools\bin>more paul_14.xyz

4397946.84 5693254.11 495.07 16 40 3 4 4

4397941.42 5693255.92 529.49 25 49 3 1 9

4397942.22 5693255.66 524.31 25 71 3 8 9

4397946.68 5693254.22 495.05 78 42 3 9 9

4397941.30 5693256.02 529.27 53 43 3 1 2

4397946.51 5693254.34 495.04 116 42 3 2 2

4397940.78 5693256.24 531.72 25 50 3 1 3

4397941.60 5693255.98 526.31 20 0 3 2 3

txt2las -i paul_14.xyz ^

-parse xyzi0prn ^

-set_point_type 6 ^

-add_attribute 3 "echo width" "Full width at half maximum [ns]" 0.1 0.0 0.1 ^

-olas

E:\LAStools\bin>lasinfo paul_14.las

lasinfo (170330) report for paul_14.las

reporting all LAS header entries:

file signature: 'LASF'

file source ID: 0

global_encoding: 0

project ID GUID data 1-4: 00000000-0000-0000-0000-000000000000

version major.minor: 1.4

system identifier: 'LAStools (c) by rapidlasso GmbH'

generating software: 'txt2las (version 170402)'

file creation day/year: 94/2017

header size: 375

offset to point data: 621

number var. length records: 1

point data format: 6

point data record length: 32

number of point records: 0

number of points by return: 0 0 0 0 0

scale factor x y z: 0.01 0.01 0.01

offset x y z: 4300000 5600000 0

min x y z: 4397940.78 5693254.11 495.04

max x y z: 4397946.84 5693256.24 531.72

start of waveform data packet record: 0

start of first extended variable length record: 0

number of extended_variable length records: 0

extended number of point records: 8

extended number of points by return: 3 2 0 1 0 0 0 1 1 0 0 0 0 0 0

variable length header record 1 of 1:

reserved 0

user ID 'LASF_Spec'

record ID 4

length after header 192

description 'by LAStools of rapidlasso GmbH'

Extra Byte Descriptions

data type: 3 (unsigned short), name "echo width", description: "Full width at half maximum [ns]", scale: 0.1, offset: 0 (not set)

reporting minimum and maximum for all LAS point record entries ...

X 9794078 9794684

Y 9325411 9325624

Z 49504 53172

intensity 16 116

return_number 1 7

number_of_returns 2 7

edge_of_flight_line 0 0

scan_direction_flag 0 0

classification 0 0

scan_angle_rank 0 0

user_data 0 0

point_source_ID 3 3

gps_time 0.000000 0.000000

extended_return_number 1 9

extended_number_of_returns 2 9

extended_classification 0 0

extended_scan_angle 0 0

extended_scanner_channel 0 0

number of first returns: 3

number of intermediate returns: 2

number of last returns: 3

number of single returns: 0

overview over extended number of returns of given pulse: 0 2 2 1 0 0 0 0 3 0 0 0 0 0 0

histogram of classification of points:

8 never classified (0)

E:\LAStools\bin>las2txt -i paul_14.las -parse xyzi0prn -stdout

4397946.84 5693254.11 495.07 16 4.0 3 4 4

4397941.42 5693255.92 529.49 25 4.9 3 1 9

4397942.22 5693255.66 524.31 25 7.1 3 8 9

4397946.68 5693254.22 495.05 78 4.2 3 9 9

4397941.30 5693256.02 529.27 53 4.3 3 1 2

4397946.51 5693254.34 495.04 116 4.2 3 2 2

4397940.78 5693256.24 531.72 25 5.0 3 1 3

4397941.60 5693255.98 526.31 20 0.0 3 2 3

--

paul_14.xyz

Paul Magdon

unread,

Apr 5, 2017, 1:07:55 PM4/5/17

to LAStools - efficient tools for LiDAR processing

Hi Martin,

thank you for providing the new las2txt.exe which works just fine! Now I am able to import the data as las v1.4. However, as always if you fix one problem the next shows up :). The txt files are in Gauss-Krüger Coordinates. As I didn't found an option to set the EPSG with lastools I used the older version from liblas to i) set the EPSG and ii) to reporject to UTM 32 N:
C:\OSGeo4W64\bin\las2las -v paul.las --output paul_projected.las --a_srs EPSG:31468 --t_srs EPSG:32632

However, liblas cannot handel las 1.4 files.
Now i tried with lastools the following way:

txt2las_new -i paul.xyz -parse xyzi0prn -set_point_type 6 -add_attribute 3 "echo width" "Full width at half maximum [ns]" 0.1 0.0 0.1 -o paul.las

las2las -i \tmp\paul.las -set_ogc_wkt -epsg 31468 -target_epsg 32632 -o paul_projected.las

When I check with:

lasinfo -i paul_projected.las
I get:

reporting all LAS header entries:
file signature: 'LASF'
file source ID: 0

global_encoding: 16

project ID GUID data 1-4:   00000000-0000-0000-0000-000000000000
version major.minor:        1.4
system identifier:          'LAStools (c) by rapidlasso GmbH'

generating software:        'las2las (version 170203)'
file creation day/year:     83/2017
header size:                375
offset to point data:       1380
number var. length records: 3

point data format:          6
point data record length:   32
number of point records:    0
number of points by return: 0 0 0 0 0
scale factor x y z:         0.01 0.01 0.01

offset x y z:               500000 5600000 0
min x y z:                  585012.76 5680700.27 452.04
max x y z:                  585985.83 5682236.33 595.47

start of waveform data packet record: 0
start of first extended variable length record: 0
number of extended_variable length records: 0

extended number of point records: 10542572
extended number of points by return: 6013833 3340181 1028536 150026 9691 296 9 0 0 0 0 0 0 0 0
variable length header record 1 of 3:

reserved             0
user ID              'LASF_Spec'
record ID            4
length after header 192
description          'by LAStools of rapidlasso GmbH'
    Extra Byte Descriptions
      data type: 3 (unsigned short), name "echo width", description: "Full width at half maximum [ns]", scale: 0.1, offset: 0 (not set)

variable length header record 2 of 3:
reserved             0
user ID              'LASF_Projection'
record ID            34735
length after header 40

description 'by LAStools of rapidlasso GmbH'

    GeoKeyDirectoryTag version 1.1.0 number of keys 4
      key 1024 tiff_tag_location 0 count 1 value_offset 1 - GTModelTypeGeoKey: ModelTypeProjected
      key 3072 tiff_tag_location 0 count 1 value_offset 32632 - ProjectedCSTypeGeoKey: WGS 84 / UTM 32N
      key 3076 tiff_tag_location 0 count 1 value_offset 9001 - ProjLinearUnitsGeoKey: Linear_Meter
      key 4099 tiff_tag_location 0 count 1 value_offset 9001 - VerticalUnitsGeoKey: Linear_Meter
variable length header record 3 of 3:
reserved             0
user ID              'LASF_Projection'
record ID            2112
length after header 611

description 'by LAStools of rapidlasso GmbH'

WKT OGC COORDINATE SYSTEM:
PROJCS["WGS 84 / UTM 32N",GEOGCS["WGS 84",DATUM["World_Geodetic_System_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.01745329251994328,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",9],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32632"]]

reporting minimum and maximum for all LAS point record entries ...

X             8501276    8598583
Y             8070027    8223633
Z               45204      59547
intensity          10        652
return_number       1          7
number_of_returns   1          7

edge_of_flight_line 0          0
scan_direction_flag 0          0
classification      0          0
scan_angle_rank     0          0
user_data           0          0

point_source_ID     3          5
gps_time 0.000000 0.000000
extended_return_number          1      7
extended_number_of_returns      1      7

extended_classification         0      0
extended_scan_angle             0      0
extended_scanner_channel        0      0

number of first returns:        6013833
number of intermediate returns: 1188558
number of last returns:         6013833
number of single returns:       2673652
overview over extended number of returns of given pulse: 2673652 4623290 2635530 561340 46975 1722 63 0 0 0 0 0 0 0 0

histogram of classification of points:

10542572 never classified (0)

Which looks ok and shows that the files are in UTM32 N. However, when I generate a DSM the DSM is shifted roughly 50m to the south. Before, when I used liblas there was no shift. Is lastools and liblast using different EPSG definitions?

Greetings

Paul

Martin Isenburg

unread,

Apr 5, 2017, 1:57:07 PM4/5/17

to LAStools - efficient command line tools for LIDAR processing

Hi Paul,

las2las from LAStools does not change datum. Hence you will be getting the UTM 32 N projection on the datum "Deutsches_Hauptdreiecksnetz" with the "Bessel 1841" spheroid instead of the datum "WGS_1984" with the "WGS84" spheroid. I know I that las2las should emit a WARNING or FAIL instead ... some future version.

To use las2las from libLAS simply encode the LAS 1.4 files with the "LAS 1.4 compatibility mode" of LASzip into a LAZ file:

E:\LAStools\bin>laszip -i paul_14.las

E:\LAStools\bin>lasinfo -i paul_14.laz

lasinfo (170330) report for paul_14.laz

reporting all LAS header entries:

file signature: 'LASF'

file source ID: 0

global_encoding: 0

project ID GUID data 1-4: 00000000-0000-0000-0000-000000000000

version major.minor: 1.2

system identifier: 'LAStools (c) by rapidlasso GmbH'

generating software: 'txt2las (version 170402)'

file creation day/year: 94/2017

header size: 227

offset to point data: 1451

number var. length records: 2

point data format: 1

point data record length: 35

number of point records: 8

number of points by return: 3 2 0 1 0

scale factor x y z: 0.01 0.01 0.01

offset x y z: 4300000 5600000 0

min x y z: 4397940.78 5693254.11 495.04

max x y z: 4397946.84 5693256.24 531.72

variable length header record 1 of 2:

reserved 0

user ID 'LASF_Spec'

record ID 4

length after header 960

description 'by LAStools of rapidlasso GmbH'

Extra Byte Descriptions

data type: 3 (unsigned short), name "echo width", description: "Full width at half maximum [ns]", scale: 0.1, offset: 0 (not set)

data type: 4 (short), name "LAS 1.4 scan angle", description: "additional attributes", scale: 0.006, offset: 0 (not set)

data type: 1 (unsigned char), name "LAS 1.4 extended returns", description: "additional attributes", scale: 1 (not set), offset: 0 (not set)

data type: 1 (unsigned char), name "LAS 1.4 classification", description: "additional attributes", scale: 1 (not set), offset: 0 (not set)

data type: 1 (unsigned char), name "LAS 1.4 flags and channel", description: "additional attributes", scale: 1 (not set), offset: 0 (not set)

variable length header record 2 of 2:

reserved 0

user ID 'lascompatible'

record ID 22204

length after header 156

description 'by LAStools of rapidlasso GmbH'

LASzip compression (version 3.0r0 c2 50000): POINT10 2 GPSTIME11 2 BYTE 2

reporting minimum and maximum for all LAS point record entries ...

X 9794078 9794684

Y 9325411 9325624

Z 49504 53172

intensity 16 116

return_number 1 7

number_of_returns 2 7

edge_of_flight_line 0 0

scan_direction_flag 0 0

classification 0 0

scan_angle_rank 0 0

user_data 0 0

point_source_ID 3 3

gps_time 0.000000 0.000000

number of first returns: 3

number of intermediate returns: 2

number of last returns: 3

number of single returns: 0

WARNING: there is 1 point with return number 6

WARNING: there is 1 point with return number 7

overview over number of returns of given pulse: 0 2 2 1 0 0 3

histogram of classification of points:

8 never classified (0)

Now your file is a standard compressed LAS 1.2 file that any older software can process that has LAZ support such as las2las from libLAS:

C:\OSGeo4W64\bin\las2las -v paul_14.laz --output paul_14_proj.laz --a_srs EPSG:31468 --t_srs EPSG:32632

The resulting file can then be reconverted back to uncompressed LAS 1.4 with laszip:

E:\LAStools\bin>laszip -i paul_14_proj.laz

Regards,

Martin @rapidlasso

=========================

PROJCS["DHDN / Gauss-Kruger zone 4",
    GEOGCS["DHDN",
        DATUM["Deutsches_Hauptdreiecksnetz",
            SPHEROID["Bessel 1841",6377397.155,299.1528128,
                AUTHORITY["EPSG","7004"]],
            AUTHORITY["EPSG","6314"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.01745329251994328,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4314"]],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",12],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",4500000],
    PARAMETER["false_northing",0],
    AUTHORITY["EPSG","31468"],
    AXIS["Y",EAST],
    AXIS["X",NORTH]]

PROJCS["WGS 84 / UTM zone 32N",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.01745329251994328,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",9],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    AUTHORITY["EPSG","32632"],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH]]

Paul Magdon

unread,

Apr 6, 2017, 9:02:10 AM4/6/17

to LAStools - efficient tools for LiDAR processing

Hi Martin,

thanks for the explanantion. Now its clear why the DSM is shifted. I tried your work around which is fine for the small example (paul.xyz) but failed for the entire data set. Here is what I did:

REM convert txt files to las
txt2las_new -i E:\LiDAR\mpi\01_raw\tmp\4375734_4376672_5683752_5685293_Spur0176.xyz -parse xyzi0prn -set_point_type 6 -add_attribute 3 "echo width" "Full width at half maximum [ns]" 0.1 0.0 0.1 -o E:\LiDAR\mpi\02_las\tmp\4375734_4376672_5683752_5685293_Spur0176.las

REM convert to LAZ for compatibility with laslib
laszip -i E:\LiDAR\mpi\02_las\tmp\4375734_4376672_5683752_5685293_Spur0176.las

REM Reproject to UTM 32 datum WGS 1984 with laslib
C:\OSGeo4W64\bin\las2las -v E:\LiDAR\mpi\02_las\tmp\4375734_4376672_5683752_5685293_Spur0176.las --output E:\LiDAR\mpi\03_projected\tmp\4375734_4376672_5683752_5685293_Spur0176.las --a_srs EPSG:32632 --t_srs EPSG:32632

This works. However, there is an error in the new laz file which prevents further processing e.g. lasdenoise. You can see it from
lasinfo -i E:\LiDAR\mpi\03_projected\tmp\4375734_4376672_5683752_5685293_Spur0176.laz

Which shows:
lasinfo (170203) report for E:\LiDAR\mpi\03_projected\tmp\4375734_4376672_5683752_5685293_Spur0176.laz

reporting all LAS header entries:
file signature:             'LASF'
file source ID:             0
global_encoding:            0
project ID GUID data 1-4:   00000000-0000-0000-0000-000000000000
version major.minor:        1.2
system identifier:          'LAStools (c) by rapidlasso GmbH'
generating software:        'txt2las (version 170402)'

file creation day/year:     83/2017
header size:                227
offset to point data:       9627
number var. length records: 6

point data format: 1
point data record length: 35

number of point records: 10542572

number of points by return: 6013833 3340181 1028536 150026 9691

scale factor x y z: 0.01 0.01 0.01
offset x y z: 4300000 5600000 0

min x y z: 4375734.23 5683751.73 452.04
max x y z: 4376671.84 5685292.74 595.47
variable length header record 1 of 6:

reserved             0
user ID              'LASF_Spec'
record ID            4
length after header 960
description          'by LAStools of rapidlasso GmbH'
    Extra Byte Descriptions
      data type: 3 (unsigned short), name "echo width", description: "Full width at half maximum [ns]", scale: 0.1, offset: 0 (not set)
      data type: 4 (short), name "LAS 1.4 scan angle", description: "additional attributes", scale: 0.006, offset: 0 (not set)
      data type: 1 (unsigned char), name "LAS 1.4 extended returns", description: "additional attributes", scale: 1 (not set), offset: 0 (not set)
      data type: 1 (unsigned char), name "LAS 1.4 classification", description: "additional attributes", scale: 1 (not set), offset: 0 (not set)
      data type: 1 (unsigned char), name "LAS 1.4 flags and channel", description: "additional attributes", scale: 1 (not set), offset: 0 (not set)

variable length header record 2 of 6:

reserved             0
user ID              'lascompatible'
record ID            22204
length after header 156
description          'by LAStools of rapidlasso GmbH'

variable length header record 3 of 6:
reserved 43707

user ID 'LASF_Projection'
record ID 34735

length after header 64
description 'GeoTIFF GeoKeyDirectoryTag'
GeoKeyDirectoryTag version 1.1.0 number of keys 7

key 1024 tiff_tag_location 0 count 1 value_offset 1 - GTModelTypeGeoKey: ModelTypeProjected

      key 1025 tiff_tag_location 0 count 1 value_offset 1 - GTRasterTypeGeoKey: RasterPixelIsArea
      key 1026 tiff_tag_location 34737 count 22 value_offset 0 - GTCitationGeoKey: WGS 84 / UTM zone 32N
      key 2049 tiff_tag_location 34737 count 7 value_offset 22 - GeogCitationGeoKey: WGS 84
      key 2054 tiff_tag_location 0 count 1 value_offset 9102 - GeogAngularUnitsGeoKey: Angular_Degree

key 3072 tiff_tag_location 0 count 1 value_offset 32632 - ProjectedCSTypeGeoKey: WGS 84 / UTM 32N
key 3076 tiff_tag_location 0 count 1 value_offset 9001 - ProjLinearUnitsGeoKey: Linear_Meter

variable length header record 4 of 6:
reserved             43707
user ID              'LASF_Projection'
record ID            34737
length after header 30
description          'GeoTIFF GeoAsciiParamsTag'
    GeoAsciiParamsTag (number of characters 30)
      WGS 84 / UTM zone 32N|WGS 84|
variable length header record 5 of 6:
reserved             43707
user ID              'liblas'
record ID            2112
length after header 597
description          'OGR variant of OpenGIS WKT SRS'
variable length header record 6 of 6:
reserved             43707
user ID              'liblas'
record ID            7
length after header 7269
description          'http://liblas.org/schema/'
LASzip compression (version 2.2r0 c2 50000): POINT10 2 GPSTIME11 2

reporting minimum and maximum for all LAS point record entries ...

ERROR: 'chunk with index 0 of 212 is corrupt' after 1953 of 10542572 points
X          -2140398450 2143434216
Y          -2141878466 2146152511
Z          -2145714721 2146156107
intensity           0      65535
return_number       0          7
number_of_returns   0          7
edge_of_flight_line 0          1
scan_direction_flag 0          1
classification      0         31
scan_angle_rank -128        127
user_data           0        254
point_source_ID     3      65498
gps_time -0.000000 0.000000
WARNING: range violates GPS week time specified by global encoding bit 0
WARNING: 1952 points outside of header bounding box
number of first returns:        537
number of intermediate returns: 425
number of last returns:         1105
number of single returns:       474
WARNING: real number of point records (1953) is different from header entry (10542572).
WARNING: for return 1 real number of points by return (263) is different from header entry (6013833).
WARNING: for return 2 real number of points by return (272) is different from header entry (3340181).
WARNING: for return 3 real number of points by return (223) is different from header entry (1028536).
WARNING: for return 4 real number of points by return (233) is different from header entry (150026).
WARNING: for return 5 real number of points by return (243) is different from header entry (9691).
WARNING: there are 274 points with return number 0
WARNING: there are 222 points with return number 6
WARNING: there are 223 points with return number 7
overview over number of returns of given pulse: 207 303 261 260 206 223 226
WARNING: there are 267 points with a number of returns of given pulse of 0

histogram of classification of points:

              74 never classified (0)
              56 unclassified (1)
              61 ground (2)
              62 low vegetation (3)
              70 medium vegetation (4)
              68 high vegetation (5)
              56 building (6)
              74 noise (7)
              41 keypoint (8)
              66 water (9)
              49 rail (10)
              62 road surface (11)
              59 overlap (12)
              67 wire guard (13)
              44 wire conductor (14)
              61 tower (15)
              70 wire connector (16)
              48 bridge deck (17)
              69 Reserved for ASPRS Definition (18)
              53 Reserved for ASPRS Definition (19)
              56 Reserved for ASPRS Definition (20)
              71 Reserved for ASPRS Definition (21)
              74 Reserved for ASPRS Definition (22)
              37 Reserved for ASPRS Definition (23)
              66 Reserved for ASPRS Definition (24)
              86 Reserved for ASPRS Definition (25)
              59 Reserved for ASPRS Definition (26)
              59 Reserved for ASPRS Definition (27)
              78 Reserved for ASPRS Definition (28)
              51 Reserved for ASPRS Definition (29)
              60 Reserved for ASPRS Definition (30)
              46 Reserved for ASPRS Definition (31)
+-> flagged as synthetic: 996
+-> flagged as keypoints: 999
+-> flagged as withheld: 963
real max x larger than header max x by 21357670.320000
real min x smaller than header min x by 21479718.730000
real max y larger than header max y by 21376232.370000
real min y smaller than header min y by 21502536.390000
real max z larger than header max z by 21460965.600000
real min z smaller than header min z by 21457599.250000

When I now run:
REM Remove noise
lasnoise -ignore_class 2 -remove_noise -i E:\LiDAR\mpi\03_projected\tmp\4375734_4376672_5683752_5685293_Spur0176.laz -o E:\LiDAR\mpi\03_projected\tmp\4375734_4376672_5683752_5685293_Spur0176_denoised.laz

I get an error
ERROR: 'chunk with index 0 of 212 is corrupt' after 1953 of 10542572 points

And lastools keeps running forever without progress.

Any ideas what went wrong?

Greetings
Paul

Martin Isenburg

unread,

Apr 7, 2017, 2:28:05 PM4/7/17

to LAStools - efficient command line tools for LIDAR processing

Hello,

turns our that Howard Butler's libLAS has a bug when asked to compress a LAS file with "extra bytes" (meaning a point type that uses more bytes per point than the point type needs that usually contain some user defined attributes). It does not compress the extra bytes (simply omits them) but does not change the LAS header which still indicates that they are there.

I have added a new "check" to LAStools that finds such corrupt LAZ files and exits early. This will be included in the next release of LAStools.

E:\LAStools\bin>lasinfo -i paul_liblas.laz

lasinfo (170408) report for paul_liblas.laz

ERROR: point has size of 35 but items only add up to 28 bytes (LASzip v3.0r0)

please upgrade to the latest release of LAStools (with LASzip)

or contact 'martin....@rapidlasso.com' for assistance.

ERROR: cannot open lasreaderlas with file name 'paul_liblas.laz'

ERROR: cannot open lasreader

This file was generated by simply recompressing the "good" LAZ file 'paul.laz' with las2las from libLAS:

C:\OSGeo4W64\bin\las2las -v paul.laz --output paul_liblas.laz

I do not expect this bug to get fixed any time soon as libLAS development is no longer active and has moved on to PDAL. But there may still be a work-around (see next email).

Below the lasinfo report of the "good" LAZ file 'paul.laz' generated by txt2las / laszip from LAStools:

E:\LAStools\bin>lasinfo -i paul.laz

lasinfo (170408) report for paul.laz

reporting all LAS header entries:

file signature: 'LASF'

file source ID: 0

global_encoding: 0

project ID GUID data 1-4: 00000000-0000-0000-0000-000000000000

version major.minor: 1.2

system identifier: 'LAStools (c) by rapidlasso GmbH'

generating software: 'txt2las (version 170402)'

file creation day/year: 96/2017

header size: 227

offset to point data: 1451

number var. length records: 2

point data format: 1

point data record length: 35

number of point records: 5024

number of points by return: 3051 1503 411 57 2

scale factor x y z: 0.01 0.01 0.01

offset x y z: 4300000 5600000 0

min x y z: 4376272.35 5683751.73 452.29

max x y z: 4376671.84 5683959.33 484.91

variable length header record 1 of 2:

reserved 0

user ID 'LASF_Spec'

record ID 4

length after header 960

description 'by LAStools of rapidlasso GmbH'

Extra Byte Descriptions

data type: 3 (unsigned short), name "echo width", description: "Full width at half maximum [ns]", scale: 0.1, offset: 0 (not set)

data type: 4 (short), name "LAS 1.4 scan angle", description: "additional attributes", scale: 0.006, offset: 0 (not set)

data type: 1 (unsigned char), name "LAS 1.4 extended returns", description: "additional attributes", scale: 1 (not set), offset: 0 (not set)

data type: 1 (unsigned char), name "LAS 1.4 classification", description: "additional attributes", scale: 1 (not set), offset: 0 (not set)

data type: 1 (unsigned char), name "LAS 1.4 flags and channel", description: "additional attributes", scale: 1 (not set), offset: 0 (not set)

variable length header record 2 of 2:

reserved 0

user ID 'lascompatible'

record ID 22204

length after header 156

description 'by LAStools of rapidlasso GmbH'

LASzip compression (version 2.5r2 c2 50000): POINT10 2 GPSTIME11 2 BYTE 2

reporting minimum and maximum for all LAS point record entries ...

X 7627235 7667184

Y 8375173 8395933

Z 45229 48491

intensity 10 346

return_number 1 5

number_of_returns 1 5

edge_of_flight_line 0 0

scan_direction_flag 0 0

classification 0 0

scan_angle_rank 0 0

user_data 0 0

point_source_ID 3 5

gps_time 0.000000 0.000000

number of first returns: 3051

number of intermediate returns: 470

number of last returns: 3051

number of single returns: 1548

overview over number of returns of given pulse: 1548 2184 1062 220 10 0 0

histogram of classification of points:

5024 never classified (0)

Martin Isenburg

unread,

Apr 7, 2017, 10:58:23 PM4/7/17

to LAStools - efficient command line tools for LIDAR processing

Hello Paul,

the next thing to try is to use laszip to decompress back the from the "compatible mode" LAZ file to a LAS file *BUT* while remaining in compatible mode:

E:\LAStools\bin>txt2las -i paul.xyz ......... -o paul14.las

E:\LAStools\bin>laszip -i paul14.las -o paul14comp.laz

E:\LAStools\bin>laszip -i paul14comp.laz -remain_compatible -o paul14comp.las

E:\LAStools\bin>lasinfo -i paul14comp.las

lasinfo (170408) report for paul14comp.las

reporting minimum and maximum for all LAS point record entries ...

X 7627235 7667184

Y 8375173 8395933

Z 45229 48491

intensity 10 346

return_number 1 5

number_of_returns 1 5

edge_of_flight_line 0 0

scan_direction_flag 0 0

classification 0 0

scan_angle_rank 0 0

user_data 0 0

point_source_ID 3 5

gps_time 0.000000 0.000000

number of first returns: 3051

number of intermediate returns: 470

number of last returns: 3051

number of single returns: 1548

overview over number of returns of given pulse: 1548 2184 1062 220 10 0 0

histogram of classification of points:

5024 never classified (0)

now do your conversion with las2las of Howard's libLAS or any other LAS 1.2 only software

C:\OSGeo4W64\bin\las2las -v paul14comp.las --output paul14compproj.las --a_srs EPSG:31468 --t_srs EPSG:32632

and then use LASzip (without any compression or decompression) to go back to LAS 1.4

E:\LAStools\bin>laszip -i paul14compproj.las -o paul14proj.las

I hope that does the trick ... (-:

Regards and Pura Vida!

Martin

On Fri, Apr 7, 2017 at 8:26 PM, Martin Isenburg <martin....@gmail.com> wrote:

Hello,

turns our that Howard Butler's libLAS has a bug when asked to compress a LAS file with "extra bytes" (meaning a point type that uses more bytes per point than the point type needs that usually contain some user defined attributes). It does not compress the extra bytes (simply omits them) but does not change the LAS header which still indicates that they are there.

I have added a new "check" to LAStools that finds such corrupt LAZ files and exits early. This will be included in the next release of LAStools.

E:\LAStools\bin>lasinfo -i paul_liblas.laz
lasinfo (170408) report for paul_liblas.laz
ERROR: point has size of 35 but items only add up to 28 bytes (LASzip v3.0r0)
please upgrade to the latest release of LAStools (with LASzip)

or contact 'martin.isenburg@rapidlasso.com' for assistance.

Reply all

Reply to author

Forward