Please kindly advise on the use of LocalTempSwap as Ehcache persistence strategy

[BK BK]

Hi, I would be grateful for your advice on the following Ehcache-Terracotta persistence problem. 

Scenario:

Our application runs on the ColdFusion 2018 application server. ColdFusion, itself a Java app, uses Ehcache 2.10.6 and Terracotta Runtime Toolkit 4.3.6.

Ehcache timeToLive and timeToIdle are both 86400 seconds (1 day). 

Java version: 11.0.10. 

Application's maximum heap size (xmx): 16 GB.

The problem:

In-memory caching was intensive in the application. As a result, heap usage rose to 95% within several hours of starting the application. As more and more users came online, heap use frequently rose to 99% during the day. The result was frequent garbage collection, accompanied by high CPU usage. 

Our solution:

We applied <persistence strategy="localTempSwap"/> for most of the cache elements in ehcache.xml. We hoped that caching on disk would reduce heap usage. Which it in fact did. But only somewhat. 

Questions we need your help with:

1. As a result of the  localTempSwap setting, 10 to 20 GB of cache data gets swapped to disk. However, heap usage remains high in spite of the large cache allocation to disk. Why?
2. Using JMX Bean, we see that, on some days, the number of cached items in memory at the end of the day is the same as the number of cached items on disk! This suggests duplicate caching. Is that possible? If so, how? If not, what could be going on?
3. We began using LocalTempSwap last week. We noticed that the diskStore directory contains remnants of cache data for each and every day since. Even though we restart the application, hence Ehcache, everyday. How do we ensure caching starts afresh daily, with a clean slate? Do we have to restart Terracotta daily as well?