No performance gains with multi-core

[Dave Munge]

Good morning,
I do treatments on blocks of around 300 tiles of 1km². When I use a lastools tool without multi-cores the processing time is about 45 minutes and the CPU usage is around 3 or 4%.
What I find odd is that when I use multi-core on all 16 logical cores the processing time is similar and the overall CPU utilization is still around 3 or 4%. I see that the 16 cores are working, but overall there are no performance gains.

The only time I get a performance boost is if I combine lastools work directly (1 combined output instead of recreating 300 other outputs), the processing time drops to around 60 minutes.

Is it normal not to have performance gains in multi-core?
If not, why and is there a way to improve it?

THANKS !

[Michael Stimson]

Hi,

 

I don’t think in this case you could blame LASTools without investigation of other factors. Which tool are you using in particular? What is your disc/network doing at the same time? Working with LAS files often bottleneck at HDD access, worse over USB and even worse over network connections… are you using a decent internal (SATA) SSD to eliminate that potential bottleneck.

 

As for multiple output files that could be a situation like disc trashing, often seen in processes that write to many small files as the read/write heads flick between multiple locations (is your seek time over 1000ms?), is your write case turned on?

 

 

 

From: last...@googlegroups.com <last...@googlegroups.com> On Behalf Of Dave Munge
Sent: Sunday, 19 March 2023 12:10 AM
To: LAStools - efficient tools for LiDAR processing <last...@googlegroups.com>
Subject: [LAStools] No performance gains with multi-core

 

CAUTION: This email originated from outside of RPS.

Good morning,
I do treatments on blocks of around 300 tiles of 1km². When I use a lastools tool without multi-cores the processing time is about 45 minutes and the CPU usage is around 3 or 4%.
What I find odd is that when I use multi-core on all 16 logical cores the processing time is similar and the overall CPU utilization is still around 3 or 4%. I see that the 16 cores are working, but overall there are no performance gains.

The only time I get a performance boost is if I combine lastools work directly (1 combined output instead of recreating 300 other outputs), the processing time drops to around 60 minutes.

Is it normal not to have performance gains in multi-core?
If not, why and is there a way to improve it?

THANKS !

--
Download LAStools at
https://rapidlasso.de
Manage your settings at
https://groups.google.com/g/lastools/membership
---
You received this message because you are subscribed to the Google Groups "LAStools - efficient tools for LiDAR processing" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lastools+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lastools/b19a2896-dbc1-44a3-b616-a9e9df8395bcn%40googlegroups.com.

This email and its attachments may contain confidential and/or privileged information and is for the sole use of the intended recipient(s). The contents of this email must not be disclosed to or used by or copied in any way by anyone other than the intended recipient(s). If you are not the intended recipient, any use, distribution or copying of the information contained in this email and its attachments is strictly prohibited. Confidentiality and/or privilege in the content of this email is not waived. If you have received this email in error, please email the sender by replying to this message and immediately delete and destroy any copies of this email and any attachments. Please note that neither RPS Consultants Pty Ltd, any subsidiary, related entity ('RPS') nor the sender accepts any responsibility for viruses and it is your responsibility to scan or otherwise check this email and any attachments. The views or opinions expressed are the author's own and may not reflect the views or opinions of RPS

[Jochen Rapidlasso]

Hi Dave,

yes, most likely it it a file-IO bottleneck.

Best configuration:

2 local SSD drives, SATA or USB3++ connected,

read from one drive, write to another drive.
So check the IO features / bandwidth on your configuration and may report about it and we will see how to improve your speed,

Thanks,

Jochen @rapidlasso

[Terje Mathisen]

Are you using LAS or LAZ input/output data?

Unless you have extremely fast disk arrays, it is very typical for LAS to completely saturate the IO interface, and here we have the additional knowledge that -merged is making it faster, most likely due to avoiding 15 of the 16 output streams.

If you are currently processing LAS files I strongly encourage you to move everything to LAZ, if you already do so then you should at least try to write your output tiles to a separate (preferably SSD-based) disk target.

Terje

Dave Munge wrote:

--
Download LAStools at
https://rapidlasso.de
Manage your settings at
https://groups.google.com/g/lastools/membership
---
You received this message because you are subscribed to the Google Groups "LAStools - efficient tools for LiDAR processing" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lastools+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lastools/b19a2896-dbc1-44a3-b616-a9e9df8395bcn%40googlegroups.com.

-- 
- <Terje.M...@tmsw.no>
"almost all programming can be viewed as an exercise in caching"