Calculating memory requirements of fill_burn

[Richard Lumb]

Can anyone help me with calculating how much memory the fill_burn tool requires. I'm using the free python API.

For example,  I know from talking to John Lindsay that the peak memory usage of  breach_depressions_least_cost is 16 bytes per pixel of input raster. From this I am able to calculate the maximum size of DEM that I can process, and I tile my large DEM accordingly.

I leave quite a large margin - I assume that only 75% of the free memory will be available throughout the process. However, despite this safety margin I'm unable to run fill_burn with the same DEM and a 75mb shapefile. So it seems to me that peak memory requirement for fill_burn must be a lot higher than for breach_depressions_least_cost, but I don't know how to calculate the requirement myself.

Also, as a side note, I intend to add flatten_lakes into the workflow soon, so if this has higher peak memory usage than the other two tools I'll need to be able to calculate that as well.

Any help very much appreciated. Thanks in advance.

Rick

[Whitebox Geospatial Inc]

Hello Rick,

 

That tool is a bit tricky to accurate estimate the memory requirements for. Looking at the code, the main memory demands come from:

 

• `dem`, a raster with requirements of 8 bytes per pixel
• `streams`, a vector with varying memory requirements. Will largely depend on the number of features, but again the size of it on disc is not necessarily an indication of its memory requirements.
• `raster_streams`, an Array2D raster representation of the streams, 1 byte per pixel.
• `output`, a raster with requirements of 8 bytes per pixel
• `in_queue`, an Array2D with 1 byte per pixel
• `queue`, a VecDeque, 16 bytes per pixel
• `minheap`, a BinaryHeap, 8+8+8 = 24 bytes per pixel

 

So, at a minimum, you are looking at 58 bytes per pixel plus the size of the vector in memory. That does make this tool rather memory hungry. Now admittedly, I have not been particularly careful about memory usage when I wrote this tool and I could have done a better job of releasing bits of memory when they were no longer needed. For example, the streams vector is not used after the vector is rasterized, early on in the process. I could have dropped it at that point, but instead, it stays in memory for the whole run. I have just pushed a revised version of this tool to the Github repo that does a better job of this, which should lessen the peak memory usage of the tool, but it will still be a somewhat memory hungry tool, just due to the nature of what it is doing.

 

Regards,

 

John Lindsay

--
You received this message because you are subscribed to the Google Groups "WhiteboxTools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to whiteboxtool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/whiteboxtools/725514ca-03c4-4156-9c6e-9149ef2169dan%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.