Issues with Cachebox not expiring items when using ehCache replication

Dan G. Switzer, II

unread,

Oct 19, 2016, 3:50:50 PM10/19/16

to ColdBox Platform

We're seeing a few issues using the CFProvider in Cachebox. We're running our servers using ehCache's replication using the RMICacheReplicatorFactory. I've attached the ehcache.xml file we're using. We're using CF10u21 on all nodes.

Here are the issues we're seeing:

We're seeing issues where expired items are still holding on to memory and are still showing up in the cache count. We started noticing that that the JVM memory on the servers wasn't reducing the heap as expected when objects should have expired from cache. When looking into things, we noticed that the calling getSize() was showing object counts much higher than expected. If we go into the Cachebox monitor, on the first run I'll see a much higher object count than expected. If I immediately refresh the monitor, the object count will drop down to expected levels. I thought it strange that the first run of the Cachebox Monitor would show a high object count and immediately running again, would show an object count close to what we'd expect (usually about 1/4 of the size). I did a little testing and it seems that it's when it loops through all the objects to display them, that the expired objects are really getting evicted. After I refresh the Cachebox Monitor for the second time, I can immediately do a garbage collection to reclaim the memory. It seems like something is holding on to the cache objects, even though they're supposed to be expired.
We're seeing issues where calling the clear() command on an item to expire it, is not always propagating the command to all nodes. After an objects is updated, we fire the clear() command to flush the item from the cache pool. However, it seems like this isn't always working to expire the item. What we're seeing are cases were a node will sometimes have an outdated version of the cache, even though we're called the clear() on it's cache key. What's really strange, is if we use the Clear All button in the Cachebox Monitor, it will properly clear all the items from the cache pool on all the nodes. I noticed under the hood that the clearAll() uses getObjectStore().removeAll(), where as clear() uses CF's cacheRemove().

Any thoughts on what might be causing our issues?

I think it may ultimately be related to CF's cacheRemove() maybe not actually properly flushing items from cache.

On a related note, does anyone know ehCache's replication works under the hood? If you remove an item named "cachebox-1" from cache, should it remove any item in a node with the same cache key (i.e. "cachebox-1") or is it trying to match some additional metadata along with the key name? I'm just curious if the objects are somehow getting out-of-sync and that's why there' not flushing properly.

Thanks!

-Dan

ehcache_cf10_replicate.xml

Luis Majano

unread,

Oct 20, 2016, 5:05:01 AM10/20/16

to ColdBox Platform

Dan

CacheBox passes through to cf for issuing commands. So it most likely a cf issue.

I have not had good expwreince with ehcache in replication mode. How many nodes do you have?

Luis Majano
CEO
Ortus Solutions, Corp
www.ortussolutions.com
P/F: 1-888-557-8057

--
--
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
For News, visit http://blog.coldbox.org
For Documentation, visit http://wiki.coldbox.org
For Bug Reports, visit https://ortussolutions.atlassian.net/browse/COLDBOX
---
You received this message because you are subscribed to the Google Groups "ColdBox Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coldbox+u...@googlegroups.com.
To post to this group, send email to col...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coldbox/c4661cad-83f0-4874-b01c-b4931a3de3f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.