lasindex stopped working when having too many las/laz files

[Zeshi Zheng]

Hi all,

I am using LAStools to process 1300 km^2 area of Lidar dataset. In order to make the point number of each file below 1.5 millions, I tiled all .las files into ~5000 500m x 500m .laz tiles. To enable parallel processing of all the tiles, I used lasindex to create .lax file for each individual laz tile. However, lasindex will only create lax files for the first ~500 laz files and then stops. Did anyone ever had similar problems before and happen to know how to solve this? Thanks!

Best,

Zeshi

[Martin Isenburg]

Hello,

It seems that the combination of Wine and lasindex on Ubuntu is the issue. We had some offline discussion on this. The program did not "crash". It ended as usual. And Zeshi was able to index the files, for example, in chunks of 200 files. Zeshi told me in an off-line conversation that he had tried a few more times and I think there is a max number of files that wine+lasindex could handle, which is about 630 files. And above that the program will only process the first 630 files. 

Does Ubuntu expand a *.laz wildcards into one very long command line with all files listed separately when Wine is used? Then it could be that only that many characters for 600 files fit into the command line that Windows is parsing ... 

Regards,

Martin @rapidlasso

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe

[Zeshi Zheng]

Thanks Martin!

I think separating files into folders should be easier for me as I need to use lasground and las2dem later on. If these files are in the same folder Wine will still have the same problem.

Regards,

-Zeshi

[Perdue, Michael]

Hello,

 

Yes, in Unix like environments, wildcards are intercepted by the shell and expansion is performed at that point. *.laz will be expanded into the full list of files for the command. However, in my experience, when the command line exceeds its maximum length it generally fails immediately with a message to the user:

lasindex.exe -i *.laz

-bash: /cygdrive/c/Programs/lastools/bin/lasindex.exe: Argument list too long

So, I suspect that wine is adding its own quirks.

 

I’m not sure what the hard limits on the CLI is for Ubuntu, but there are 3 methods I use that will avoid this pitfall:

Use a list of files;

ls *.laz > listoffiles.txt

lasindex.exe –lof listoffiles.txt

 

Use a loop;

for j in *.laz; do

lasindex.exe –i “$j”

done

 

use GNU parallel;

parallel lasindex.exe –i {} ::: *.laz

or

parallel lasindex.exe –i {} :::: listoffiles.txt

 

GNU parallel is by far the most powerful option. It will automatically detect how many cores are present and parallize the operation, allows the user to build complex workflows into functions and then execute the function in parallel, and will even allow you to distribute the workload across multiple machines if your environment is set up properly (I’ve had success with it on a 48 core beowulf cluster). I use it regularly within Cygwin and have not run into any issues; unfortunately I can’t say whether it will or won’t play well with wine.

 

Cheers,

 

Mike

[Kirk Waters - NOAA Federal]

One of the *nix ways around command line length is xargs. It can prepare can run commands such that they don't exceed the command line length. So, you could do something like:

find . -name '*.laz' | xargs wine lasindex -i

xargs will tack on as many of the laz files as fit on the command line and run subsequent ones until all are taken care of. There are also ways to have it use multiple cores. And yes, in *nix land, file names are expanded on the command line by the shell.

Kirk Waters, PhD                     | NOAA Office for Coastal Management

Applied Sciences Program      | 2234 South Hobson Ave
843-740-1227                          | Charleston, SC 29405    

coast.noaa.gov/digitalcoast

[Martin Isenburg]

Hi,

What you can do is create a text file such as file_list.txt that lists all the names of all the files with one name per line.

Then you give this list of files via the command line switch '-lof' to any of the LAStools. That won't expand the command line under linux.

lasindex -lof file_list.txt

or

wine lasindex -lof file_list.txt

Same with lasground or las2dem or any other tool.

Martin