Run in parallel

[Ozias Hounkpatin]

Dear Whitebox Geospatial team,

This question is more related to python than for whitetool box. I am new to python and I am trying to look for ways to run in parallel the impoundment index for many tiled rasters. The code below is for only one core which takes too long. I read in the manual that it is possible to run this in parallel but I could not find while searching a way to implement this. Any help or direction (may be another platform) will be appreciated.

#---1@ Import path of all the dem files

path=r"L:\Gotland_Ditch\Dem_Tiles"

search_criteria = "*.tif"

q = os.path.join(path, search_criteria)

dem_fps = glob.glob(q)

dem_fps

#filename = os.path.split(dem_fps)

#---1@ Define output path

#output=r"L:\Gotland_Ditch\python_Imp\filename.tif"

#---2@ Build loop to compute impoundment index and save

for i in dem_fps:

  filename = os.path.basename(i)

  filepath =os.path.join(r"L:\Gotland_Ditch\python_Imp", filename)

  wbt.impoundment_size_index(

  i,

  output=filepath,

  damlength=4,

  out_type="mean depth",

  )

Best Regards,

Ozias Hounkpatin

[Whitebox Geospatial]

Hi Ozias,

Thank you for your question.

The impoundment size index (ISI) tool is an extremely computationally intensive tool, especially during the flow accumulation process. As a result, it is likely that the parts of the tool and your python script will run slow.

The ISI tool is designed to be as efficient as possible. In the tool itself, every part of the tool that can run in parallel already does. However, there are some parts in the code that cannot run in parallel and as a result run time increases. In addition to this, the tool is extremely memory intensive.  

Unfortunately, this is just the nature of the algorithm. Unless you are using a very powerful super computer, it is very likely that you will want to run this tool sequentially on your list of input DEMs.

The only other thing I can think of is the Multiprocessing python package. This package will allow you to send tasks to the various cores on your machine. I have had some luck with this package before, but I don’t know how useable it will be in this scenario. It is likely if you try run this tool on multiple DEMs at a time using multiple cores, you will either overload your cores or run out of memory quickly.

Best,

Whitebox Geospatial