[LAStools] Lastile performance

58 views
Skip to first unread message

Fabio Picchi

unread,
Oct 27, 2022, 10:48:42 AM10/27/22
to last...@googlegroups.com
Hi

I am currently trying to speed up tiling using lastile with the -cores option. The data is indexed and prepared for lastile, but when I use all my available cores (48), the system just idles for roughly 90% of the time.

iostat -m                                                                                                                                                                                              fabio-debugging: Thu Oct 27 14:37:08 2022

Linux 5.4.209-116.367.amzn2.x86_64 (fabio-debugging)    10/27/22        _x86_64_        (48 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.49    0.00    1.68    0.09    0.00   90.73

Device             tps    MB_read/s    MB_wrtn/s    MB_dscd/s    MB_read    MB_wrtn    MB_dscd
md0            1344.79        33.73       104.42       407.90     160189     495860    1936983
nvme0n1           7.01         0.09         0.20         0.00        445        969          0
nvme1n1         626.39        15.05        50.53       203.98      71479     239967     968628
nvme2n1         627.31        14.99        50.40       203.92      71175     239311     968354


nvme1n1 is my input disk and nvme2n1, the output one. I can't help but wonder why I have roughly the same amount of reads and writes from both input and output disks. Is there something wrong with my setup?

In any case, is there a recommended number of cores for IO using separate input and output NVMEs. Regardless of the possible problem with my setup I would be interested to hear if someone has tried that out already.

Best,

--
Fabio Picchi I Backend Developer

Connect with us on LinkedIn | Visit our Site
Tillinmäentie 3B, 02330 Espoo, Finland

Jochen Rapidlasso

unread,
Nov 2, 2022, 9:09:32 AM11/2/22
to LAStools - efficient tools for LiDAR processing
Hi Fabio,
what is your setup?
How does your commandline/script/batchfile look like?
lastile with multiple cores only comes to effect if you have >1 input files.
Tell us more!
Thanks and best regards,

Jochen @rapidlasso

Fabio Picchi

unread,
Nov 3, 2022, 2:41:11 PM11/3/22
to last...@googlegroups.com
I'm running lastile in a docker container running Ubuntu Jammy. It has wine installed.

This is how my CLI command looks like:

wine lastile.exe -i data/input/*.laz -odir data/output -tile_size 100 -buffer 10 -olaz -cores 48

There are hundreds of large swaths (roughly one minute long) in input and I expect the tiles in output. Both paths are mounted to different NVME disks, so IO should be as fast as possible.

In my silly implementation of a tiler the CPU was the bottleneck because of the compressed inputs and outputs. And so it seems to be the case for lastile when running it in windows, natively. I wonder if wine is doing something seriously wrong here...

My Wine version is 7.0 by the way...

Best,
Fabio

--
Download LAStools at
https://rapidlasso.de
Manage your settings at
https://groups.google.com/g/lastools/membership
---
You received this message because you are subscribed to the Google Groups "LAStools - efficient tools for LiDAR processing" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lastools+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lastools/5ec3b5b3-8f9f-4c79-ad64-a26d93ea14b2n%40googlegroups.com.

Jochen Rapidlasso

unread,
Nov 3, 2022, 2:45:37 PM11/3/22
to LAStools - efficient tools for LiDAR processing
Hi Fabio,
this should run as you expect - so the problem may is in wine. 
Unfortunately I can not give much advice about wine. 
Hopefully another user may have experience with this.
One try: Does it make a difference if you run the sub-processes with 64 bit?
So this would be 
wine lastile.exe -i data/input/*.laz -odir data/output -tile_size 100 -buffer 10 -olaz -cores 48 -cpu64
Another one: Does the program tells you something of interest if you add the -v option?

Best regards,

Jochen @rapidlasso

Fabio Picchi

unread,
Nov 4, 2022, 4:50:50 AM11/4/22
to last...@googlegroups.com
I'll try it!

But actually, running with -cpu64 in wine gave me a faulty output as quite a few tiles were missing :|

Idan Tene

unread,
Nov 8, 2022, 1:34:01 PM11/8/22
to last...@googlegroups.com
Fabio,

We use lastools on Ubuntu as well. I'm not sure if this is what you're experiencing, but it tells a cautionary tale about emulators and wine specifically.
One built-in limitation of wine is that it only works with one CPU group. For example, we have a machine with 128 cores, and we can *ask* lastools to use 128 cores, but since wine only sees one CPU group, we're limited in our case to 64 cores.
Lastools and wine would still generate many additional workers, but they're competing for resources, leading to threading and overall slower processing.
We've since learned to use 63 cores (+1 for wineserver running it all) to minimize such competition.

I suggest you check how many CPU groups your setup has, and/or check out the lastools linux beta, once it supports multiprocessing natively.

Best,
Idan


Reply all
Reply to author
Forward
0 new messages