hardware recommendations for faster processing

[Seabird]

 

Hello,

 

I'm looking for hardware recommendations for “faster” processing.

 

We are currently running a Lenovo s30:

                Xeon E5-1620-v2 (4 cores-2 logical cores per physical),

                16GB RAM,

                250GB SSD,

                1TB SATA.

               

When set to utilize all cores, cpu usage goes to 100%, but the process is then I/O bound. 

Read/write is being done to the SATA drive (not the SSD).

Memory usage sits at about 60%.

 

Given that the process seems to be I/O bound, I'm thinkng that upgrading to

SSD drives would improve the situation.

 

I could add more memory, but since I’m not seeing a high level of memory utilization,

is increasing memory going to increase performance?

 

I could also swap the processor for one with more cores, but at some point the interaction

with I/O will still bind the process.

 

Any recommendations/thoughts?

Cheers!

 

[Kaebisch, Tyler (DNR)]

You need more cores and more ram

 

Start with at least 8 cores and 32 gb RAM

 

This is what I run for statewide Minnesota lidar data:

 

Precision Workstation T5810 specs

Dell Precision Tower 5810 CTO Base (210-ACQM) 
32GB (4x8GB) 2133MHz DDR4 RDIMM ECC (370-ABUO) 
AMD FirePro W4100 2GB (4 mDP) (4 mDP-DP adapters) (490-BCCE) 
500GB 3.5inch Serial ATA (7,200 Rpm) Hard Drive (400-AAWR) 
Integrated Intel AHCI chipset SATA controller (6 x 6.0Gb/s) - SW RAID 0/1/5/10 (403-BBGV) 
8x Slimline DVD+/-RW Drive (429-AAPE) 
Intel Xeon Processor E5-1620 v3 (Four Core HT, 10MB Cache, 3.5GHz Turbo) (338-BFJW) 
Dell Precision Tower 5810 685W Chassis (329-BCFX) 
C1 SATA 3.5 Inch, 1-2 Hard Drives (449-BBEF) 
Non RAID (780-BBCJ) 
Additional Drive: 3.5 inch 2TB SATA 7.2k RPM HDD (401-AAMX) 
Boot drive or boot volume is less than 2TB (411-XXXY) 

--
Download LAStools at
http://lastools.org
http://rapidlasso.com
Be social with LAStools at
http://facebook.com/LAStools
http://twitter.com/LAStools
http://linkedin.com/groups/LAStools-4408378
Manage your settings at
http://groups.google.com/group/lastools/subscribe

[Evon Silvia]

If you're i/o bound, you need to figure out what kind of i/o you're doing at the bottleneck.

Case 1: If you're just waiting for a single tile to load, your slowdown may be from low RPM HDD, but more likely you're hitting the SATA bus limits. You won't see much improvement by switching to SSD unless it's a PCI express m2 SSD, which has a much faster data bus.

Case 2: If you're sampling pieces of multiple files (e.g., processing buffered tiles with neighbors, processing sections of tiles), then you'll see a big improvement processing from SSD because of faster seek times. Consider a SSD RAID array.

Case 3: If you're running multithreaded/multiprocess and multiple threads are reading/writing at the same time to the same drive (e.g., multi-process spawning, the -cores N option for some LAStools, etc), you might be running into thrashing. This can be reduced by running on SSD arrays, but the better option is to modify your process to ensure only one thread/process is reading or writing to a specific drive. For example, if running multiple instances of LAStools, dedicate a drive to each instance.

Evon

[Martin Isenburg]

Hello,

if you are I/O bound (and you likely are if you run LAStools on 4 or more cores) then it is imperative to use LAZ as your input and output format. 

To push this further assure that your point clouds are nicely ordered for maximum compression (first by flightline, then by gpstime, and then by return number) which can be achieved with lassort.

If you do repeated area-of-interest queries or on-the-fly buffering you should furthermore assure that your files have - after the initial order just described - are sorted into a space-filling curve and then lasindexed.

For advanced LAStools users the ultimate reduction of the "O" from the heavy disk "I/O" comes is form of LASlayers. Read up on it and try to convert one of your LiDAR preparation batch script that go from LAZ to LAZ in each and every step into a new LiDAR preparation batch script that instead only keeps the original LAZ tiles, create a tiny LAY file for each tile, and then just keeps appending to that LAY file until all steps are done.

http://rapidlasso.com/LASlayers

http://groups.google.com/forum/#!searchin/lastools/LASlayers

Regards,

Martin @rapidlasso

[Seabird]

Hi,

Great! Thanks to Tyler, Evon for the comments. Martin, I've been using LAZ but not sorted and i'll read up on the LAY files. Very useful information!

Martin - for hardware, is there a minimum memory per core recommendation for running multiple cores?

Cheers!

[Martin Isenburg]

Hi,

Each LAStools process will use no more than 2 GB. Some big processes will do their own out-of-core data handling (lasgrid, lascanopy, las2dem, blast2dem). The more of your file system that you can keep in the "cache" (assuming the same data tiles are frequently re-used) the better.

Regards,

Martin @rapidlasso