Error with lastile

[Dave....@mffp.gouv.qc.ca]

Hello !

Why do I get so many error messages during a lastile?

 

Wearing the input data was classified with lasground and cleaned with lasnoise.

 

What should I understand?

 

Thank you !

 

 

WARNING: on-the-fly merged LAS 1.2 files contain too many points (24982103351) f

or single LAS 1.2 file.

WARNING: written 9792461 points but expected 0 points

WARNING: written 10573968 points but expected 0 points

ERROR: reading header.file_signature

ERROR: could not open lasreaderlas for file 'E:\UG23\2016\LAS\Donneesbrutes\filt

re\hauteur\16_2645424f07_02301_db_denoise_class_haut.laz'

WARNING: written 1299431 points but expected 0 points

WARNING: on-the-fly merged LAS 1.2 files contain too many points (24982103351) f

or single LAS 1.2 file.

WARNING: on-the-fly merged LAS 1.2 files contain too many points (24982103351) f

or single LAS 1.2 file.

ERROR: reading header.file_signature

ERROR: could not open lasreaderlas for file 'E:\UG23\2016\LAS\Donneesbrutes\filt

re\hauteur\16_2645424f07_02301_db_denoise_class_haut.laz'

removing E:\UG23\2016\LAS\Donneesbrutes\filtre\hauteur\tuiles\tuile_263200_54250

00.laz

WARNING: written 0 points but expected 0 points

WARNING: on-the-fly merged LAS 1.2 files contain too many points (24982103351) f

or single LAS 1.2 file.

ERROR: reading header.file_signature

ERROR: could not open lasreaderlas for file 'E:\UG23\2016\LAS\Donneesbrutes\filt

re\hauteur\16_2645424f07_02301_db_denoise_class_haut.laz'

WARNING: written 731692 points but expected 0 points

 

Dave Munger

Analyste en géomatique et LiDAR
Unité de gestion Rivière-Péribonka

Direction de la gestion des forêts du Saguenay-Lac-St-Jean
Ministère des Forêts, de la Faune et des Parcs
801, chemin du Pont-Taché Nord, bureau RC-03 

Alma (Québec)  G8B 5B7
Téléphone : 418 668-8319, poste 370
dave....@mffp.gouv.qc.ca
mffp.gouv.qc.ca

 

[Tobias K Kohoutek]

Dave, 

could you paste your command line of lastile and a lasinfo report of the file(s) you want to tile? Your warning quotes on the fly merged point cloud of aprox. 25 Billion points, this is huge.

Cheers,

Tobias

[Dave Munge]

Here is my command line :

 

Lastile –i *.laz –o tuile –tile _size 1400 –buffer 10 –cores 4 –olaz

 

And a lasinfo report :

 

reporting all LAS header entries:

  file signature:             'LASF'

  file source ID:             0

  global_encoding:            1

  project ID GUID data 1-4:   00000000-0000-0000-0000-000000000000

  version major.minor:        1.2

  system identifier:          'LAStools (c) by rapidlasso GmbH'

  generating software:        'lasheight (160329) commercial'

  file creation day/year:     306/2016

  header size:                227

  offset to point data:       569

  number var. length records: 2

  point data format:          1

  point data record length:   30

  number of point records:    4359879

  number of points by return: 2720274 1498864 138691 2046 4

  scale factor x y z:         0.01 0.01 0.01

  offset x y z:               0 0 0

  min x y z:                  250000.00 5393000.00 181.58

  max x y z:                  250999.99 5393999.99 244.71

variable length header record 1 of 2:

  reserved             0

  user ID              'LASF_Projection'

  record ID            34735

  length after header  40

  description          'by LAStools of rapidlasso GmbH'

    GeoKeyDirectoryTag version 1.1.0 number of keys 4

      key 1024 tiff_tag_location 0 count 1 value_offset 1 - GTModelTypeGeoKey: ModelTypeProjected

      key 3072 tiff_tag_location 0 count 1 value_offset 2949 - ProjectedCSTypeGeoKey: NAD83(CSRS) / MTM zone 7

      key 3076 tiff_tag_location 0 count 1 value_offset 9001 - ProjLinearUnitsGeoKey: Linear_Meter

      key 4099 tiff_tag_location 0 count 1 value_offset 9001 - VerticalUnitsGeoKey: Linear_Meter

variable length header record 2 of 2:

  reserved             43707

  user ID              'LASF_Spec'

  record ID            4

  length after header  192

  description          'by LAStools of rapidlasso GmbH'

    Extra Byte Descriptions

      data type: 4 (short), name "height above ground", description: "vertical point to TIN distance", scale: 0.01, offset: 250

the header is followed by 2 user-defined bytes

LASzip compression (version 2.4r1 c2 50000): POINT10 2 GPSTIME11 2 BYTE 2

reporting minimum and maximum for all LAS point record entries ...

  X            25000000   25099999

  Y           539300000  539399999

  Z               18158      24471

  intensity           0       1202

  return_number       1          5

  number_of_returns   1          5

  edge_of_flight_line 0          1

  scan_direction_flag 0          1

  classification      1          2

  scan_angle_rank   -25         27

  user_data           7          7

  point_source_ID 21609      21611

  gps_time 154266614.388402 154268351.559040

number of first returns:        2720274

number of intermediate returns: 140695

number of last returns:         2720971

number of single returns:       1222061

overview over number of returns of given pulse: 1222061 2719846 409791 8161 20 0 0

histogram of classification of points:

         3423266  unclassified (1)

          936613  ground (2)

 

The project includes 6400 tiles.

 

Regards

[Dave Munge]

My biggest question:

What does this type of message mean:

ERROR: 'end-of-file during chunk with index 13' after 684923 of 3688701 points

Thank you !

Le mercredi 11 janvier 2017 10:55:34 UTC-5, Tobias K Kohoutek a écrit :

[Martin Isenburg]

Hello Dave,

when you get the error

ERROR: 'end-of-file during chunk with index 13' after 684923 of 3688701 points

then you have a LASzip compressed file that was truncated *after* compression, for example due to a failed / interrupted ftp transfer or download. Here it happens in the 14th chunk of 50000 compressed points (that has index 13). So only the first 13 * 50000 = 650000 points are correct for sure. 

This really is a critical ERROR because not only are you missing a lot of point but the last two to thirty points before the truncation is detected (and this ERROR message appears) could potentially have corrupted attributes. It's a good thing to check LAZ files after download for truncation and other ftp failures.

laszip -i *.laz -check

lasvalidate -i *.laz -o report.xml

Regards,

Martin

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe

[Martin Isenburg]

Hello Dave,

when you have huge amounts of LiDAR to process then you may stretch the memory footprint of lastile in default mode beyond its 2GB limit. The issue is that too many LAZ tiles will have to be opened at the same time creating this memory footprint that is too large. 

http://rapidlasso.com/lastile

http://lastools.org/download/lastile_README.txt

There are two work arounds:

(a) assure your input LiDAR points are provided in a spatially coherent in the order of files that you give to lastile so that tiles can be finalized (meaning filled completely) long before all points are processed. Then you can add the option '-extra_pass' but you will need to operate on one core only.

lastile -i *.laz ^

          -tile_size 1400 ^

          -buffer 10 ^

          -extra_pass ^

          -o tiles/tile.laz

(b) first create a coarser tiling with tiles that are double or quadruple as and then refine all tiles (or at least all tiles with a minimum of 1 million points) in subsequent steps

lastile -i *.laz ^

          -tile_size 5600 ^

          -buffer 10 ^

          -refine_tiling 1000000 ^

          -o tiles/tile.laz

lastile -i tiles/tile*_5600.laz ^

          -refine_tiles 1000000 ^

          -olaz ^

          -cores 4

lastile -i tiles/tile*_2800.laz ^

          -refine_tiles 1000000 ^

          -olaz ^

          -cores 4

Regards,

Martin @rapidlasso

--

[Dave Munge]

Thank you Martin,

My memory problem seems to apply here too, because when I do on the problematic PC:

Laszip -i * .laz -check

Lastools freezes after displaying an error of type:

ERROR: 'end-of-file during chunk with index 13' after 684923 of 3688701 points

When I pass Lasvalidate everything is fine and all the files are good.

Moreover, when I execute "laszip -i * .laz -check" on another PC, it runs perfectly and all the files are good.

I will try the "-refine_tiling 1000000" because when I ran:

Lastil-i * .laz ^

-tile_size 1400 ^
-buffer 10 ^
-extra_pass ^
-o tiles / tile.laz

Guess what, as I do not use several cores, the same problem of freezing appears. We do not get away with it.

If I understand the operation of lastile, when I make a "-refine_tiling 1000000" the existing tile is divided into 4. All the tiles remain in the same directory and can be used directly by a * .laz after the work?

A very big thank you for your answers always just !!

[Martin Isenburg]

Hi,

My memory problem seems to apply here too, because when I do on the problematic PC:

Laszip -i * .laz -check

Lastools freezes after displaying an error of type:

ERROR: 'end-of-file during chunk with index 13' after 684923 of 3688701 points

This ERROR is not caused by lack of memory but by bit flips in or truncations of the LAZ files. If it only happens on a particular PC then this piece of hardware may have the occasional bus error.

Regards,

Martin

[Ruben Wauters]

Hi, 

I get similar errors as above. Until now I ignored them, but how should I fix this?

My command: lastile -i "E:\....\raw\*.las" -buffer 20 -odir "E:\....\lastile" -full_bb -tile_size 1000 -epsg 31370 -odix "_tile"

My raw data: too much (85 GB), so can't share this

My errors: see picture in attach

--> it is also not clear which tiles are affected because it doesn't display any tile name so don't know which one to trust and so on...

Thanks a lot!

Ruben

Op donderdag 19 januari 2017 om 05:30:54 UTC+1 schreef Martin Isenburg:

[Jochen Rapidlasso]

Hi Ruben,
without testdata it is hard to analyze.
I think you have to try to get closer to the problem with changing parameters to see,
where the problem occurs.
Problems with large files may are also operating system, network or filesystem problems...

May see this post:
https://groups.google.com/g/lastools/c/zBxXBixGB_I/m/rb383WY8AwAJ

Did you check the files using lasvalidate and lasinfo?

- are they full readable?
- can you e.g. just copy them 1:1 using a command like
las2las -i *.laz -odir c:\temp\test

How big are the files?
Can you reproduce the problem on
- other/all files?
- other storage?
- other hardware with (more/less) memory?
...

So this are my first ideas.
After you have checked this I may have some more :)

Best,

Jochen @rapidlasso