warning cache size.... problems

[James Hollingsworth]

Hi Oleg,

Sorry, just clearing through a bunch on (hopefully) simple questions...

I'm processing a WV stereo-pair on my server. I'm using a session with 24 cores. However, when I run certain processes, like point2dem, using --threads 24, I keep getting these messages:

DEM: [********************************....................................] 47%Warning: Cache size (806 MB) is larger than the requested maximum cache size (805 MB). Consider increasing the cache in ~/.vwrc.
DEM: [********************************....................................] 48%Warning: Cache size (1209 MB) is larger than the requested maximum cache size (805 MB). Consider increasing the cache in ~/.vwrc.
DEM: [**********************************..................................] 50%Warning: Cache size (1815 MB) is larger than the requested maximum cache size (805 MB). Consider increasing the cache in ~/.vwrc.
DEM: [************************************................................] 53%Warning: Cache size (2723 MB) is larger than the requested maximum cache size (805 MB). Consider increasing the cache in ~/.vwrc.
DEM: [**************************************..............................] 57%Warning: Cache size (4086 MB) is larger than the requested maximum cache size (805 MB). Consider increasing the cache in ~/.vwrc.
DEM: [*******************************************.........................] 64%Warning: Cache size (6129 MB) is larger than the requested maximum cache size (805 MB). Consider increasing the cache in ~/.vwrc.
DEM: [**************************************************..................] 74%Warning: Cache size (9193 MB) is larger than the requested maximum cache size (805 MB). Consider increasing the cache in ~/.vwrc.

etc, etc... and the process takes AGES to run. Also, I don't seem to have a ~/.vwrc file, so I can't edit that. However, I have PLENTY of memory, so I don't see where the problem comes from, or at least how to fix it.

On a similar note, when I run parallel_stereo, it also took AGES to generate the -L.tif and -R.tif, and masks, etc (in stereo_pprc), also due to this caching hold-up. However, I can't see which options I can give to parallel_stereo that can help it better manage these issues. I remember in the past that if I reduce the number of cores, it actually speeds things up (e.g. with point2dem this seems to be the case), because you don't get these caching problems. However, parallel_stereo now gives an error with the option --threads. And the --processes and  --threads-multiprocess only seem to apply for the correlation process. 

When I run parallel_stereo, I'm going the mapprojected route, and the two files are big (~9GB tifs with LZW). The -F.tif is big too (16GB LZW).

ANyhow, if you have any thoughts on how to manage these sorts of problems (I'm sure you've encountered them), that would be great.

Cheers,

James

 

[Oleg Alexandrov]

This is a well-known problem. It shows up because your images are likely stored line-by-line, rather than in tiles. Here's an explanation and a work-around: https://github.com/NeoGeographyToolkit/StereoPipeline/issues/355

You can get a .vwrc and set a big cache. Here's the doc. https://stereopipeline.readthedocs.io/en/latest/installation.html#settings-optimization

I hope to come up with a fix at some point, but it will be tricky. The only thing I can think about is rewrite the image under the hood, which will take time from user's processing, and likely there's no good way of doing it just once and all subsequent processing uses that. It may be better for the user to just re-create their images with proper tiling, and increase the cache. 

Let me know if these work.

--
You received this message because you are subscribed to the Google Groups "Ames Stereo Pipeline Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ames-stereo-pipeline...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/54cf5c52-acf4-46ad-b699-575569f3ced9n%40googlegroups.com.

[Oleg Alexandrov]

To add to the previous email I wrote.

>  However, parallel_stereo now gives an error with the option --threads.

This was fixed in the latest version, 3.1.0, now it is just a warning. Also, that error says you should use --threads-multi and --threads-single, so that error is on purpose. 

[Oleg Alexandrov]

I increased the cache size to 1 GB, from 800 MB, and added to each tool the option --cache-size-mb, to be able to set it. The warning message now refers to to this option. So one need not get .vwrc, using which is a bit awkward because one has to carry it around on any new system. All this is documented for each tool. 

It is a bit tricky to decide on a default value for the system cache, as it is hard to know what each user's system's resources are. Also, if a user starts running say 20 processes, with each consuming 1 GB, or even 2 GB, that can result in memory running out, especially that MGM has its own separate memory buffer (the system cache is only for reading images from disk).

A long-term solution may be for the tools, after much complaining, to exceed the cache size set by the user. This may result in a crash though, rather than likely now, just very slow performance.

I also don't know what to do about the fact that some input .TIF and .NTF files are formatted not in blocks but row-wise or column-wise, and when they are huge, traversing them in an order which does not agree with how they are stored on disk messes up with the cache system, so the image portions are repeatedly read in and freed up as the cache fills up. Likely a lot of code may need to be modified to use some logic which respects the structure on disk, which would be a lot of case-by-case work.