Dense photogrammetric point cloud processing

[Matt]

Hi Martin,

I have been experimenting with lastools for the extraction of various types of terrain from photogrammetrically derived point clouds and I am having some issues processing the large point clouds. I have read previously about some of the issues processing these files but have had success with smaller sparse clouds and was hoping to scale up.

The full files are reasonably large at around 20 GB with a very close point spacing of around 10 cm (photo resolution). From memory I think the full file was around 680 million points.

I have been tiling these datasets into 1km by 1km tiles and have been getting the below error on the lasground tool and a similar one on the lasdem command.

lasground -lof file_list.txt -odir "E:\2013\OUTPUTS\POINTS\5Mile_Tiled" -olas -utm 55M

using projection UTM 'UTM 55 southern hemisphere'

processing file 'E:\2013\OUTPUTS\POINTS\5Mile_Tiled\5Mile_-1000_-1000.las'.

horizontal units are meters and vertical units are meters. nature mode.

reading 44527137 points. step is 5 m, spike is 1+10 m, and offset is 0.05 m ...

took 135.596 sec. finding initial ground points ...

took 1.657 sec. generating initial ground estimate ...

took 0.779 sec. refining ground ...

took 1.06 sec. adding terrain features ...

took 210.717 sec. integrating points 0.05 above ground ...

ERROR: failed malloc for buffer 9 of 4,194,304 TINtriangles.

I will try halving the tiles again and see if that helps.

Many Thanks

Matt

[Terje Mathisen]

Matt wrote:
> Hi Martin,
>
> I have been experimenting with lastools for the extraction of various
> types of terrain from photogrammetrically derived point clouds and I am
> having some issues processing the large point clouds. I have read
> previously about some of the issues processing these files but have had
> success with smaller sparse clouds and was hoping to scale up.
>
> The full files are reasonably large at around 20 GB with a very close
> point spacing of around 10 cm (photo resolution). From memory I think
> the full file was around 680 million points.

You really don't want 10 cm triangles for your ground cathegorization!

I would use the "-keep_random_fraction 0.1" (or even 0.01) option to
keep the number of points in each tile well below the 4G limit!

Terje

>
> I have been tiling these datasets into 1km by 1km tiles and have been
> getting the below error on the lasground tool and a similar one on the
> lasdem command.
>
> lasground -lof file_list.txt -odir "E:\2013\OUTPUTS\POINTS\5Mile_Tiled"
> -olas -utm 55M
> using projection UTM 'UTM 55 southern hemisphere'
> processing file 'E:\2013\OUTPUTS\POINTS\5Mile_Tiled\5Mile_-1000_-1000.las'.
> horizontal units are meters and vertical units are meters. nature mode.
> reading 44527137 points. step is 5 m, spike is 1+10 m, and offset is
> 0.05 m ...
> took 135.596 sec. finding initial ground points ...
> took 1.657 sec. generating initial ground estimate ...
> took 0.779 sec. refining ground ...
> took 1.06 sec. adding terrain features ...
> took 210.717 sec. integrating points 0.05 above ground ...
> ERROR: failed malloc for buffer 9 of 4,194,304 TINtriangles.
>
> I will try halving the tiles again and see if that helps.
>
> Many Thanks
>
> Matt
>

> --
> Download LAStools at
> http://lastools.org
> http://rapidlasso.com
> Be social with LAStools at
> http://facebook.com/LAStools
> http://twitter.com/LAStools
> http://linkedin.com/groups/LAStools-4408378
> Manage your settings at
> http://groups.google.com/group/lastools/subscribe
>
>


--
- <Terje.M...@tmsw.no>
"almost all programming can be viewed as an exercise in caching"

[Martin Isenburg]

Hello Matt,

44,527,137 points are too many for lasground as a quick search in the LAStools user forum with the keywords "lasground" and "malloc" will tell you. Simply use a tile size of 500 by 500 meters instead (don't forget to use a buffer) and you should be fine. You should try to keep the number of points below 20 million. There is a batch script in the batch script example folder that tells you how to ground classify single files that exceed that limit but since you are using lastile the easiest solution is a smaller tile size ...

Regards,

Martin @rapidlasso

--

http://rapidlasso.com - fast tools to ground your LiDARs

Matt

--

[Robin Poot]

Hi Matt,

 

I tried this to no avail....

 

Tiling will permit the processing to happen, and it is super-awesome and fast, but I think lastools relies on first-last return extraction to get the points considered bare earth. 

 

With photogrammetric point clouds all points, top of trees and on ground, are first return.  If I am wrong please lets share.  I was very impressed by the scriptability of lastools and the performance too, but just simply did not get good results.

 

The photogrammetric software I use relies on shape and point variance in a hierarchical fashion, and therefore I believe a lot more complicated, and I could not recreate the quality with lastools,

 

I do have datasets of simultaneous lidar and photo so it is easy to compare.

 

Kind Regards,

 

Robin

[Martin Isenburg]

Matt and Robin.

indeed. lasground does not like "too many bomb craters" (e.g. clusters of negative outliers) and needs at least one ground point per coarse cell (e.g. the cell that is used for the initial ground surface guess). the parameter that influences the latter is the '-step n' which uses n=5 in default (natural) mode and other pre-set values for '-city' or '-metro' (see README). You can directly set it without using the those switches by specifying  '-step 12.5' or  '-step 30' or else. But setting it too high will "chop off" the mountain tops if you have steep terrains ... 

Cheers,

Martin

--

[Matt]

Hi Martin and Robin,

I have had very good success in the past with ground extraction from the point clouds just not lately with the reasonably large ones. I tiled the master 20 gb file down to tiles with sizes ranging from 50 mb to 1.8 gig. Lastools seems to struggle with the tiles above 750 mb.

I have not found too many places where I cant extract good ground returns. I have had success in most of the forest types in my area and most of the sites are coastal so not too much issue with steep terrain. The point clouds that I generate do have overhanging/under hanging points as the models are not strictly planimetric so I can 'see the ground under the trees' in numerous places. 

Thanks for all the help. I will keep on trying and let you know once I have sorted out a technique of dealing with them.

Matt

[Martin Isenburg]

Matt,

lasground does not "struggle" with files containing more than 30 million points. It is simply not designed to handle such large files. For larger input is meant to be used in a tile-based processing pipeline with up to 20, maybe 25 million points per tile as outlines in the example batch scripts "huge_las_file_classify_ground.bat" or "typical_lidar_preparation_pipeline.bat" that are part of the lastools.zip distribution. So it seems you will have to use smaller tiles. 

Regards,

Martin @rapidlasso

[Umair Asghar]

Hi Matt, 

I've also been using LASground to extract bare earth from a photogrammetrically derived point cloud for a large landslide. I'm getting same error. Have you been able to figure a way around this problem?

 I'm very new to point cloud processing (less than 3 months) and so far used qCANUPO and VR-Mesh to extract bare earth models. I request your suggestion for other software, preferably open-source, for bare earth extraction. 

Regards,

Umair.

[Terje Mathisen]

Umair Asghar wrote:
> Hi Matt,
>
> I've also been using LASground to extract bare earth from a
> photogrammetrically derived point cloud for a large landslide. I'm
> getting same error. Have you been able to figure a way around this
> problem?

This is relatively easy:

1) Either you reduce the size of each tile: With 1000x1000 m and 10 cm
spacing you have 100 M points/tile, the error message is due to running
out of memory.

2) Or you subsample (random reduction should work fine) so that you get
about 1-10 points/square meter, this is still sufficient for ground
determination.

I would suggest doing both, as well as using tiles with a suitable
buffer. Personally I use 250x250m tiles with a 32m buffer, and I still
have to subsample tiles with very dense pulse spacing.

Terje

> --
> Download LAStools at
> http://lastools.org
> http://rapidlasso.com
> Be social with LAStools at
> http://facebook.com/LAStools
> http://twitter.com/LAStools
> http://linkedin.com/groups/LAStools-4408378
> Manage your settings at
> http://groups.google.com/group/lastools/subscribe


--

[Umair Asghar]

Terje, thanks a lot for your response. 

I'm able to use 500 x 500 m tiles and the results are good for most of the area. However, for slopes with some sharp undulations and some debris, LASground classifies the edges and debris as vegetation. For slope stability analysis I need those undulations and debris to be classified into ground as well. I've used custom parameters (step =10, 15, 20, 25m; Bulge= 1, 2; spike=1; down spike=1, offset= 0.35, Hyper fine to extra fine), but all parameter combinations from these, classify plenty of points (sharp undulations and debris) as vegetation. I used 0.35 and even 0.4, because I wanted the little/smaller grass to be a part of bare-earth.

Figure 1 shows the actual point cloud, Figure 2 shows the bare earth model extracted using custom parameters mentioned above, and Figure 3 shows the vegetation/trees that also contains some white/pale/light brown earth. 

Figure 1

Figure 2

Figure 3

Can someone suggest something so that the points corresponding to earth in Picture 3 should be classified as bare-earth and not vegetation?

Also, what's the purpose of bulge, spike and down spike? I haven't seen any difference in classification by using values of 1, 2 and 3. Do I need to go further up e.g. 5 or even 10 to see a significant difference?

I'll be much thankful for a comprehensive respone. 

--

 Think Green & be Environmental Friendly! 

[Terje Mathisen]

Umair Asghar wrote:
> Terje, thanks a lot for your response.
>
> I'm able to use 500 x 500 m tiles and the results are good for most of
> the area. However, for slopes with some sharp undulations and some
> debris, LASground classifies the edges and debris as vegetation. For
> slope stability analysis I need those undulations and debris to be
> classified into ground as well. I've used custom parameters (step =10,
> 15, 20, 25m; Bulge= 1, 2; spike=1; down spike=1, offset= 0.35, Hyper
> fine to extra fine), but all parameter combinations from these,
> classify plenty of points (sharp undulations and debris) as
> vegetation. I used 0.35 and even 0.4, because I wanted the
> little/smaller grass to be a part of bare-earth.

Looking at your images the area does look like it should be possible to
get a near-perfect ground TIN, the problem comes down to the sampling
theorem:

You stated earlier that you have approximately 10x10cm points, this
would the mean that the smallest feature you could hope to extract would
be about 20 cm, right?

lasground (and most/all other ground classification algorithms assume
you are looking for human-scale and larger features, you just need to
scale this down from meter-level to decimeter-level resolution, i.e.
what happens if you start with "wilderness" and then divide all the
parameters by 10?

Terje

>
> Figure 1 shows the actual point cloud, Figure 2 shows the bare earth
> model extracted using custom parameters mentioned above, and Figure 3
> shows the vegetation/trees that also contains some white/pale/light
> brown earth.
>
> Figure 1

> Inline image 1
>
>
>
> Figure 2

> Inline image 2
>
>
>
> Figure 3

> - <Terje.M...@tmsw.no <mailto:Terje.M...@tmsw.no>>

> "almost all programming can be viewed as an exercise in caching"
>
>
> --
> Download LAStools at
> http://lastools.org
> http://rapidlasso.com
> Be social with LAStools at
> http://facebook.com/LAStools
> http://twitter.com/LAStools
> http://linkedin.com/groups/LAStools-4408378

> <http://linkedin.com/groups/LAStools-4408378>
> Manage your settings at
> http://groups.google.com/group/lastools/subscribe

> <http://groups.google.com/group/lastools/subscribe>
>
>
>
>
> --
> --
> *Think Green & be Environmental Friendly! *

[Umair Asghar]

Dear Terje, 

My point cloud resolution is resolution is about 6 cm per pixel/point. 

When I use wilderness or nature, the results are worst than custom used parameters mentioned above, i.e.  more ground points get classified into vegetation. I'm not sure but I think the step parameter for wilderness is less than 15m. 

Also, is there any publication or manual that mentions the use or importance of bulge and spike? Or could you please just lightly explain these two parameters to me?

Regards,

Umair.

--

[Martin Isenburg]

Hello,

When I use wilderness or nature, the results are worst than custom used parameters mentioned above, i.e.  more ground points get classified into vegetation. I'm not sure but I think the step parameter for wilderness is less than 15m. 

Correct. If you look at the console output you can see what step parameter '-wilderness' translates to. Here an example:

>> lasground -i ..\data\fusa.laz -wilderness -o mist.laz

processing file '..\data\fusa.laz'.

horizontal units are meter and vertical units are meter. wilderness mode.

reading 277573 points. step is 3 m, sub is 4, bulge is 0.6 m, spike is 1+10 m, and offset is 0.05 m ...

took 0.836 sec. finding initial ground points ...

took 0.908 sec. generating initial ground estimate ...

took 0.187 sec. refining ground ...

took 0.893 sec. adding terrain features ...

took 0.915 sec. integrating points 0.05 above ground ...

took 0.979 sec. outputting ...

took 0.444 sec. 204902 points classified as ground.

done with 'mist.laz'. total time 5.162 sec.

>> lasground -i ..\data\fusa.laz -town -o mist.laz

processing file '..\data\fusa.laz'.

horizontal units are meter and vertical units are meter. town mode.

reading 277573 points. step is 10 m, sub is 6, bulge is 1 m, spike is 1+1.5 m, and offset is 0.05 m ...

took 1.181 sec. finding initial ground points ...

took 0.15 sec. generating initial ground estimate ...

took 0.079 sec. refining ground ...

took 0.366 sec. adding terrain features ...

took 0.957 sec. integrating points 0.05 above ground ...

took 0.998 sec. outputting ...

took 0.564 sec. 182306 points classified as ground.

done with 'mist.laz'. total time 7.315 sec.

Also, is there any publication or manual that mentions the use or importance of bulge and spike? Or could you please just lightly explain these two parameters to me?

Initially lasground creates a ground estimate by using the lowest point in each cell of a coarse grid whose cell size is determined by the step parameter. The '-wilderness' is just a short-hand for a step of 3 meter. Just like '-town' is a short-hand for a step of 10 meter. Also some of the other parameters change as you can see above. 

The bulge parameter defines how "high" coarse triangles of the coarse TIN are allowed to "bulge up" when deciding about whether to insert the currently considered last return into the slowly refining ground TIN or not. The lower you set it the less the TIN is bulging up and the more features (little hills) are potentially chopped off.

The spike parameter is used between refinement passes. We check the current TIN if any of the inserted points forms too much of a "spike" with its immediate one-triangle-ring of other points. A larger spike is useful if you expect your ground TIN to have great variation.

Regards,

Martin @rapidlasso