SOLR memory use and excess paging

[Nick Lauland]

My organization runs over a dozen completely separate DSpace instances of wildly varying sizes, all of them 9.2 (and tracking), with Postgres, SOLR, node and tomcat all together on AWS ec2 running Amazon Linux.

Most of these run fine with 8g of ram, but they are "smaller" and their SOLR indexes are definitely smaller than that, so memory use does not seem to be a problem.

But a few seem to be running far slower than expected, and show problems with excessive paging (via "sar -B").  These are the ones where the SOLR index has grown to be twice or even four times the size of physical memory. (In the 64g to 128g ranges)

I'm wondering if anyone else has similar sized DSpace systems, and what they've seen with memory use and SOLR, and if they've been able to do any tuning to help reduce paging.

It's hard to gauge actual (or what kind) SOLR memory use is going on, as it (as we have it configured) uses memory mapped files (so it's virtual footprint is larger than physical plus swap), and very large buff/cache.

"Throwing more memory at SOLR" seems to be an obvious thing to do, but is that the real solution, and, is there a way to do that other than more physical memory that'd be efficient?  (And not to the detriment of Tomcat, postgres or node)

Thanks for any ideas!