lasclip - Clip large ALS dataset with many polygons

[Matteo Mura]

Dear lidar-lastools specialists,

I have a question for you.

I am clipping a ALS dataset (about 400 ha) with 197 forest plots polygons of 13m-radius, in order to run subsequently cluodmetrics in FUSION (which requires the clipped point cloud for each plot) and get the ALS metrics for the plots.

The clipping can be carried out in two ways: (i) clip the plots on the whole merged area; (ii) clip the plots on the tiles. The fact is that in both ways lasclip require a lot f time because for each plot-polygon it passes the or (i) whole area or (ii) all the listed tiles, like in the pipelines below:

(i)

lasclip -cores 2 -i %INPUT_LAS%\Merge.laz -poly %INPUT_POLYGONS%\Plot_1.shp -odir %OUTPUT_DIR%  -o Plot_1.laz

lasclip -cores 2 -i %INPUT_LAS%\Merge.laz -poly %INPUT_POLYGONS%\Plot_2.shp -odir %OUTPUT_DIR%  -o Plot_2.laz

lasclip -cores 2 -i %INPUT_LAS%\Merge.laz -poly %INPUT_POLYGONS%\Plot_3.shp -odir %OUTPUT_DIR%  -o Plot_3.laz

…

(ii)

lasclip -cores 2 -i %INPUT_LAS%\*.laz -poly %INPUT_POLYGONS%\Plot_1.shp -odir %OUTPUT_DIR%  -o Plot_1.laz

lasclip -cores 2 -i %INPUT_LAS%\*.laz -poly %INPUT_POLYGONS%\Plot_2.shp -odir %OUTPUT_DIR%  -o Plot_2.laz

lasclip -cores 2 -i %INPUT_LAS%\*.laz -poly %INPUT_POLYGONS%\Plot_3.shp -odir %OUTPUT_DIR%  -o Plot_3.laz

…

[*.laz => all the tiles inside the %INPUT_LAS% directory]

But, if it clips the plot-polygon on the tile where it is located the processes is very fast:

lasclip -cores 2 -i %INPUT_LAS%\tile_n-th.laz -poly %INPUT_POLYGONS%\Plot_1.shp -odir %OUTPUT_DIR%  -o Plot_1.laz

The problem would not exist if lasclip could automatically locate the plot in the listed tile instead of pass all of them.

Is there a way to do that? Or any other way to speed up the process?

Thanks in advance,

Matteo

[Rombouts, Jan (FSA)]

Matteo,

 

You also have to consider that a plot could straddle up to four tiles.

 

Have you considered using lascanopy? The advantage of this tool is that it works through a list of polygons provided in a shape file, one by one, and calculates the metrics you specify for all the points enclosed in each polygon.  It then outputs the metrics in a csv file, one record per polygon. Since you are working with circular plots you may also provide the plots as a list of plot centre coordinates and radii in a simple text file (-loc argument). You use the merged laz file of merge a bunch of laz files on the fly using the –merged argument.

 

lascanopy will churn quite rapidly through 400 ha/197 plots.  Note that lascanopy does not output the lidar points enclosed in the plots (which would be nice). The list of available metrics is extensive but I don’t know how it compares to those offered by FUSION.

 

 

 

Jan

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe

[Susana Gonzalez]

Hi Matteo,

 

Usually I clip the plot-polygon on the tile because as you noted it is very fast, but later I have the inconvenience that Jan noted where there are plots in more than one tile.

 

My suggestion is, for large areas write a list with your plots and the intersected tiles, in the case that you have the plot 1 in two tiles, clip this plot in both tiles and later merge them in one plot.

For example:

lasclip -cores 2 -i %INPUT_LAS%\tile_n-th.laz -poly %INPUT_POLYGONS%\Plot_1.shp -odir %OUTPUT_DIR%  -o Plot_1a.laz

lasclip -cores 2 -i %INPUT_LAS%\tile_n-th.laz -poly %INPUT_POLYGONS%\Plot_1.shp -odir %OUTPUT_DIR%  -o Plot_1b.laz

 

After merging these two subplots to obtain plot 1 you can run cloudmetrics in FUSION.

 

Remember that if you use FUSION to clip the data you must use ClipData because PolyClipData doesn’t normalized the data.

Hopefully you have now a better idea of the general problem with the clipped plots and tile boundaries.

 

Cheers

 

Susana Gonzalez

Forest Engineer, LiDAR Science

 

Interpine Group Ltd
NZ Office :  07 350 3209 extn 722

Australia:    02 8011 3645

Fax :             07 345 7571
Address :     99 Sala Street, PO Box 1209, Rotorua 3010, New Zealand
Website :     www.interpine.co.nz

[Matteo Mura]

Really thank Jan and Susana!

Jan, honestly I did not know about the clipping capability within lascanopy.

Susana, thanks for the clarifications and insights on the issue.

I will test both your suggestions soon!

Thanks again,

Matteo

[Martin Isenburg]

Hello Matteo,

the lack of proper and efficient multi polygon support in lasclip.exe
has been on my TODO list for quite some time. Why are you clipping
(large amounts of) individual plots into LAS/LAZ files for later use
in FUSION? Is there a particular cloud metric that FUSION has, but
that is still missing in lascanopy?

In the newest version of LAStools (140825) released just now

D:\lastools\bin>lasclip -version
LAStools (by mar...@rapidlasso.com) version 140825 (licensed)

you will find a brand-new version of lasclip that has some
improvements that should help you. Namely the options to '-split'. In
either case - be it one large input or many smaller tiles - it is a
VERY GOOD IDEA to create spatial indexing information before doing any
area-of-interest query such as clipping a small area of points from a
large file (or a folder of files).

Here are your options assuming you have a shapefile 'plots.shp' with
500 polygons or so:

:: one file with all plots
lasclip -v -i merged.laz -poly plots.shp -o all_plots.laz
lasclip -v -i tiles\tile*.laz -merged -poly plots.shp -o all_plots.laz

:: one file per plot
lasclip -v -i merged.laz -poly plots.shp -o plot.laz -split
lasclip -v -i tiles\tile*.laz -merged -poly plots.shp -o plot.laz -split

:: one file per tile with all plots of this tile
lasclip -v -i tiles\tile*.laz -poly plots.shp -olaz
lasclip -v -i tiles\tile*.laz -poly plots.shp -olaz -cores 4

:: one file per plot *and* per tile
lasclip -v -i tiles\tile*.laz -poly plots.shp -olaz -split
lasclip -v -i tiles\tile*.laz -poly plots.shp -olaz -split -cores 4

In each case the merged total of all points cut out - no matter if it
is one file, one file per plot, or one file per plot per tile - should
be identical but all tile based workflows using are a lot faster
(especially with multiple cores).

Parallelizing the execution by plot polygon is not (yet) implemented.
For this you currently need to explode the shape file into many
polygons and then do parallel calls to lasclip yourself.

Can someone test and report back ?

Cheers,

Martin @rapidlasso

PS: For other new features you can always check the 'CHANGES.txt' file.

On Sat, Aug 16, 2014 at 3:52 PM, Matteo Mura <mur...@gmail.com> wrote:
> Dear lidar-lastools specialists,
>
> I have a question for you.
>
> I am clipping a ALS dataset (about 400 ha) with 197 forest plots polygons of
> 13m-radius, in order to run subsequently cluodmetrics in FUSION (which
> requires the clipped point cloud for each plot) and get the ALS metrics for
> the plots.
>
> The clipping can be carried out in two ways: (i) clip the plots on the whole
> merged area; (ii) clip the plots on the tiles. The fact is that in both ways
> lasclip require a lot f time because for each plot-polygon it passes the or
> (i) whole area or (ii) all the listed tiles, like in the pipelines below:
>
> (i)
> lasclip -cores 2 -i %INPUT_LAS%\Merge.laz -poly %INPUT_POLYGONS%\Plot_1.shp
> -odir %OUTPUT_DIR% -o Plot_1.laz
> lasclip -cores 2 -i %INPUT_LAS%\Merge.laz -poly %INPUT_POLYGONS%\Plot_2.shp
> -odir %OUTPUT_DIR% -o Plot_2.laz
> lasclip -cores 2 -i %INPUT_LAS%\Merge.laz -poly %INPUT_POLYGONS%\Plot_3.shp
> -odir %OUTPUT_DIR% -o Plot_3.laz

> ...

>
> (ii)
> lasclip -cores 2 -i %INPUT_LAS%\*.laz -poly %INPUT_POLYGONS%\Plot_1.shp
> -odir %OUTPUT_DIR% -o Plot_1.laz
> lasclip -cores 2 -i %INPUT_LAS%\*.laz -poly %INPUT_POLYGONS%\Plot_2.shp
> -odir %OUTPUT_DIR% -o Plot_2.laz
> lasclip -cores 2 -i %INPUT_LAS%\*.laz -poly %INPUT_POLYGONS%\Plot_3.shp
> -odir %OUTPUT_DIR% -o Plot_3.laz

> ...

> [*.laz => all the tiles inside the %INPUT_LAS% directory]
>
> But, if it clips the plot-polygon on the tile where it is located the
> processes is very fast:
>
> lasclip -cores 2 -i %INPUT_LAS%\tile_n-th.laz -poly
> %INPUT_POLYGONS%\Plot_1.shp -odir %OUTPUT_DIR% -o Plot_1.laz
>
> The problem would not exist if lasclip could automatically locate the plot
> in the listed tile instead of pass all of them.
>
> Is there a way to do that? Or any other way to speed up the process?
>
> Thanks in advance,
>
> Matteo
>

[Viv]

Hi Martin,

Our Lastools version is from August 12, 2014, and I've been trying to run lasclip on a large data set - 991 tiles sized at 5-km each. It appears we are having a similar issue as Matteo, but not exactly the same. 

Our shape file has polygons that cover about a third of the 991 tiles. We used lastile to create the tiles, but have no way to generate a shapefile with each tile's name, hence I can't say for certain which of the 1/3 of the 991 tiles intersect with my polygon. I would like to use your example of -merged before performing the clip, but we can't realistically (nor will Lastools accept) a file with that many points. 

Is there a possibility to use -merge, but keep the output in our original 991 tiles? If there was a way to LasSplit on the merged tiles without creating one huge .laz file, that could be an option. Alternately, it would be just as useful to have the points classified within the 991 tiles without having to create a separate output of the same tile. 

We did have some tiles come out empty without any data, just the file name. I've killed the process but may try it again overnight and see what happens. 

Also, another issue we had was the need to call out the directory for the polygon shapefile. Even though I had placed the shapefile within the same folder/directory as the .laz tiles to be processed, lasclip sent back an error that it couldn't find the polygon. When we specifically called out the directory again in the command line it worked.

Example:

Lasclip -i T:hag\working\Area_1\.laz -poly T:hag\working\Area_1\clip_poly.shp  -classify 8 -interior - verbose -odir "T:hag\working\Area_1\PL_Clip\ -odix "-clp8" -olaz

Thanks,

Vivian

[Martin Isenburg]

Hello Vivian,

> Our Lastools version is from August 12, 2014

You - in particular - should get the latest version (see below).

> We
> used lastile to create the tiles, but have no way to generate a shapefile
> with each tile's name, hence I can't say for certain which of the 1/3 of the
> 991 tiles intersect with my polygon.

The latest version of lasboundary can do this "hyper-efficiently". Simply run

lasboundary -i *.laz -use_bb -oshp

with the new option '-use_bb' instructing lasboundary not to read the
points but form a shapefile from the 4 corner points of the bounding
box. While we are at it, if you have LAX files then there is another
very efficient option to get a tighter shape than a bounding box
without parsing the points. The '-use_lax' option will give you an
outline of the points based on their occupancy of the spatial quadtree
that LAX uses for accelerated area-of-index queries.

> I would like to use your example of
> -merged before performing the clip, but we can't realistically (nor will
> Lastools accept) a file with that many points.

Using '-merged' to merge input tiles "on-the-fly" (and not creating a
merged file) has no known limits.

> Is there a possibility to use -merged, but keep the output in our original
> 991 tiles? If there was a way to lassplit on the merged tiles without

> creating one huge .laz file, that could be an option. Alternately, it would
> be just as useful to have the points classified within the 991 tiles without
> having to create a separate output of the same tile.

(a) Do not use '-merged' and (optionally) run on multiple cores. This
will use your 991 tiles as input and produce (up to) 991 tiles as
output (minus the empty ones) each clipped against the polygon:

lassplit -i *.laz -poly clip.shp -odix _clipped -olaz -cores 8

(b) You cannot reclassify in-place. But can you use LASlayers concept
to create the new LAY files (still experimental) in case you lasclip
to classify points but do not want to rewrite each LAZ file:

lassplit -i *.laz -poly lakes.shp -classify_as 9 -olay -cores 8

(c) You can clip the points from all the input tiles into separate
output files - one file per polygon with the new option '-split':

lassplit -i *.laz -merged -poly lakes.shp -o lakes.laz

> Also, another issue we had was the need to call out the directory for the
> polygon shapefile. Even though I had placed the shapefile within the same
> folder/directory as the .laz tiles to be processed, lasclip sent back an
> error that it couldn't find the polygon. When we specifically called out the
> directory again in the command line it worked.
>
> Example:
> Lasclip -i T:hag\working\Area_1\.laz -poly
> T:hag\working\Area_1\clip_poly.shp -classify 8 -interior - verbose -odir
> "T:hag\working\Area_1\PL_Clip\ -odix "-clp8" -olaz

This is the expected behaviour.

Regards,

Martin @rapidlasso

--
http://rapidlasso.com - fast tools to clip your LiDARs

[Matteo Mura]

Martin,

I used cloud metrics of FUSION to get a .csv file with all metrics per plot. Such a file can be easily read in spreadsheet or statistical software for carry out the analysis. Cloud metrics requires as input the las/laz file clips for each plot. But as far as I understood, now also lascanopy is able to output such .csv file with all metrics per plot, am I right?

Speaking about metrics, FUSION covers a lot of more metrics than lascanopy, but most of them has been so far useless from my modest literature knowledge (e.g. L-moments), but two relevant metrics are not yet directly implemented in lascanopy although they can be indirectly extracted by the others: coefficient of variability of height (also at different height strata) and canopy relief ratio. If we could have also this metrics directly estimated by lascanopy should be better.

I hope my modest comments could be somehow useful to you.

I will be back soon with some tests on your new version of lasclip.

All the best,

Matteo

[Queija, Vivian]

Hi Martin,

I'm trying to run the routines you suggested for clipping out lidar points from my shapefile with multiple plots, however, I'm getting an error when I try to use the -split command in LasClip and also an error when I sue the -poly command in LasSplit. 

Also, I must not be using the correct commands to make a output file with all the interior clipped points of my polygons from all my tiles.

Here are the commands I'm using:

lasclip -v -i %projpath%\rcHAG5km\*.laz -odir %projpath%\WHAG\ -odix "_nwHAG" -poly PL_20mBuff.shp -o wire.laz -interior -verbose -cores 4 

I am trying to clip all the points within my polygons to a separate file - all the clipped polygon points to one .laz file.

Plus I would like to have them clipped from the original tiles. I believe I have successfully removed the points within the polygons from the individual files, but can't seem to make the file with the interior clipped points.I'm not sure what I'm doing wrong.

Thanks for you help!

Vivian

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe

--
Vivian Queija

605-212-6496

[Welborn, Toby]

Vivian,

Saw your post on the LASTools thread.  Just curious if your office has a license or if you are using the open source version?

Sorry to bug you, I am sure you have other work to do.

Toby Welborn

775.887.7671

U.S. Geological Survey

Nevada Water Science Center

2730 N. Deer Run Rd.

Carson City, Nevada, 89701

[Queija, Vivian]

No bother! We DO have a license - multiple licenses - of LAStools.

[Martin Isenburg]

Hello Vivian,

first download the newest version of LAStools from today (141017) as bugs have been fixed in lasclip/lascanopy (for -i *.laz -merged inputs) and lasground (rare case). Then try the following:

create a height normalized fusa.laz file:

>> lasheight -v -i ..\data\fusa.laz -replace_z -o fn.laz 

create pseudo plot polygons from the buildings:

>> lasboundary -v -i fn.laz -keep_class 6 -disjoint -concavity 2 -oshp

tile the normalized fusa.laz file into 4 tiles:

>> lastile -v -i fn.laz -tile_size 125 -o ft.laz

compute the avg height of the points of each pseudo plot polygon

>> lascanopy -i ft_*.laz -lop fn.shp -height_cutoff 0 -avg -o heights1.csv

split the points of each pseudo plot polygon into separate files

>> lasclip -v -i ft_*.laz -merged -poly fn.shp -split -o plots.laz

compute the avg height of the points in each plot file

>> lascanopy -i plots*.laz -files_are_plots -height_cutoff 0 -names -avg -o heights2.csv

Regards,

Martin @rapidlasso

[Queija, Vivian]

Thanks Martin, I will update to the latest version of LAStools.

Here is some more information on the data I have. 

There are 991 tiles of lidar that I have already run lasheight on and replaced the z. The polygon plots I have have been created/digitized from a raster in ArcMap. The lasheight points within the polygon plots are not classified to a specific classification like Buildings. There are multiple classifications with the plots and I need all the point classification within these plots to analyze. I cannot call out a specific classification alone to pull out of the plots.Is this going to be a problem if I don't follow your exact workflow? 

I am trying to simply clip the points within the polygons out of the tiles after lasheight has been run. Ideally, I would like to have two separate files for each tile, but I would be happy to have the original tiles minus the points within the polygons and a file with all the points from the polygons. 

Thanks,

Vivian

[Martin Isenburg]

Hello Vivian,

The only reason i was using the lasboundary step in the example that i gave was to generate *some* (obviously meaningless) shapefile that anyone would be able to reproduce from the available "fusa.laz" sample data set without me having to attach data to my email. Off course both - your shapefile as well as your (normalized?) LIDAR tiles - will come from elsewhere. The calls to lasclip and lascanopy should, however, have the same structure ...

The workflow you describe next can be realized with the '-interior' instead of the the '-split' flag.

 lasclip -v -i ft_*.laz -poly fn.shp -odix _poly -olaz -cores 4
 lasclip -v -i ft_*.laz -poly fn.shp -interior -odix _no_poly -olaz -cores 4

Regards,

Martin