java.io.EOFException for disk persistent cache for removeAll operation

[Ceyhan Kasap]

Hi,

I happened to open an issue before writing to this group (https://github.com/ehcache/ehcache3/issues/2619) , sorry about that , but I wanted to replicate the issue we are seeing in production .

It would be great if you could comment since this is a prod issue 

below is the environment info and issue desription.

Regards

What version of Ehcache you are currently using;     -> 3.2.5

Paste the configuration for the Cache/CacheManager you have an issue with; -> pasted

Add any name and version of other library or framework you use Ehcache with (e.g. Hibernate); -> spring boot 2.0.3.RELEASE , spring-boot-starter-cache 2.0.3.RELEASE, hibernate 5.2.17

Providing JDK and OS versions maybe useful as well.  -> jdk: Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode) os: Red Hat Enterprise Linux Server release 7.5 (Maipo) 

We are using ehcache in our spring boot application. Our spring boot version is 2.0.3.RELEASE and spring-boot-starter-cache 2.0.3.RELEASE uses ehcache 3.5.2.

Our motivation to use ehcache was that it is both jsr107 compliant and provides offheap support.

Below is our spring config:

@Configuration
@ConditionalOnWebApplication
@EnableCaching
public class CacheConfig {
    @Autowired
    private ApplicationContext context;

    @Bean
    public JCacheManagerFactoryBean jCacheManagerFactoryBean() throws IOException {
        JCacheManagerFactoryBean jCacheManagerFactoryBean = new JCacheManagerFactoryBean();
        Resource resource = context.getResource("classpath:mts/ehcache.xml");
        jCacheManagerFactoryBean.setCacheManagerUri(resource.getURI());
        return jCacheManagerFactoryBean;
    }

    @Bean
    public JCacheCacheManager ehCacheCacheManager() throws IOException {
        Properties props = System.getProperties();
        props.setProperty(Caching.JAVAX_CACHE_CACHING_PROVIDER, "org.ehcache.jsr107.EhcacheCachingProvider");
        JCacheCacheManager jCacheCacheManager = new JCacheCacheManager();
        jCacheCacheManager.setCacheManager(jCacheManagerFactoryBean().getObject());
        jCacheCacheManager.setTransactionAware(true);
        return jCacheCacheManager;

    }

}

The problem we face in production is, for a disk persistent cache with moderately big size, we have the below java.io.EOFException error in removeAll operation :

Error : RuntimeException: java.io.EOFException 
java.lang.RuntimeException: java.io.EOFException
        at org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine$FileChunk.readKeyBuffer(FileBackedStorageEngine.java:541) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine.readKeyBuffer(FileBackedStorageEngine.java:265) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at org.terracotta.offheapstore.storage.PortabilityBasedStorageEngine.readKey(PortabilityBasedStorageEngine.java:119) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at org.terracotta.offheapstore.OffHeapHashMap$DirectEntry.<init>(OffHeapHashMap.java:1540) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at org.terracotta.offheapstore.OffHeapHashMap$EntryIterator.create(OffHeapHashMap.java:1518) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at org.terracotta.offheapstore.OffHeapHashMap$EntryIterator.create(OffHeapHashMap.java:1511) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at org.terracotta.offheapstore.OffHeapHashMap$HashIterator.next(OffHeapHashMap.java:1407) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at org.terracotta.offheapstore.AbstractLockedOffHeapHashMap$LockedEntryIterator.next(AbstractLockedOffHeapHashMap.java:399) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at org.terracotta.offheapstore.AbstractLockedOffHeapHashMap$LockedEntryIterator.next(AbstractLockedOffHeapHashMap.java:392) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at org.terracotta.offheapstore.concurrent.AbstractConcurrentOffHeapMap$AggregateIterator.next(AbstractConcurrentOffHeapMap.java:553) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore$1.next(AbstractOffHeapStore.java:499) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore$1.next(AbstractOffHeapStore.java:489) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at org.ehcache.core.EhcacheBase$Jsr107CacheBase.removeAll(EhcacheBase.java:708) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at org.ehcache.jsr107.Eh107Cache.removeAll(Eh107Cache.java:304) ~[ehcache-3.5.2.jar!/:3.5.2 7941fa2573343b31ae56a12564404552c6d6eff0]
        at com.mycompany.myproject.services.cache.service.impl.CacheService.doClearCacheWithName(CacheService.java:56) ~[MyProjectServicesCache_classes.jar!/:?]  

There is nothing special in the code calling the removeAll operation. Just gets the cache with name and calls clear all:

private void doClearCacheWithName(String cacheName) {
    Cache<Object, Object> cache = cacheManager.getCache(cacheName);
    if (cache == null) {
        throw new MyProjectException(String.format("Cache with name : %s does not exist!", cacheName));
    }
    logger.info(String.format("Clearing cache with name : %s", cacheName));
    cache.removeAll();
}

Here is the production config for ourBigCache:

   <cache alias="ourBigCache">
        <expiry>
            <ttl unit="seconds">21600</ttl> 
        </expiry>
        <resources>
            <heap unit="entries">1000</heap>
            <disk unit="MB">4096</disk>
        </resources>
    </cache>

We could not reproduce this neither locally nor for test environment.

Please note that this cache has very a high usage (read count is very high in production), but I guess that should not make any difference .

I could not find any similar reported issue. There are some very old disk problems menitoned but they are too old and not similar:

https://sourceforge.net/p/ehcache/discussion/322278/thread/e7a62df3/

http://forums.terracotta.org/forums/posts/list/2694.page

Any help would be greatly appreciated.

Regards

[Chris Dennis]

Most likely this is just another symptom of https://github.com/ehcache/ehcache3/issues/2542 which was fixed in 74239a93e14eb7477841fffa36c971ef9e930686

Unfortunately this fix hasn’t been merged anywhere yet. If you want to pick this up you could cut your own build of master (which would probably be a good thing to verify). Otherwise you'll have to wait for the first dot-release on the 3.7 line (timeline unknown at this point).

Chris

--
You received this message because you are subscribed to the Google Groups "ehcache-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ehcache-user...@googlegroups.com.
To post to this group, send email to ehcach...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ehcache-users/c6803689-5ae3-4888-a765-b96ba2d873e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ceyhan Kasap]

Hi @chrisdennis ,

Is it possible to perform a build on commit 74239a93e14eb7477841fffa36c971ef9e930686 and upload the artifact to an acccessable path? The reason I a mrequesting is that our company policy disables code downloads and buıld from external sources.

Regards

11 Mart 2019 Pazartesi 16:38:13 UTC+3 tarihinde Chris Dennis yazdı:

[Chris Dennis]

You should be able to pull a 3.7.1 internal release via Terracotta’s maven repo: (for example) http://repo.terracotta.org/maven2/org/ehcache/ehcache/3.7.1-internal2/ehcache-3.7.1-internal2.pom

These are untested. They are just builds of master that we use to help with integration work internally.  Use at your own risk!

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/ehcache-users/5a199a3e-4e1e-4677-9c28-11127ac8f4a2%40googlegroups.com.

[Ceyhan Kasap]

Hi Chris ,

First of all, many thanks for your support. I managed to perform a buıld on commit 74239a93e14eb7477841fffa36c971ef9e930686.  But something is wrong (most probably due to me, gradle config or etc.) , I have the below error in my application with the output jar:

Attribute 'enable-management' is not allowed to appear in element 'jsr107:defaults'.

When I checked, I saw that this attribute is indeed missing in ehcache-107ext.xsd in the output jar ( I do not have much experience with gradle and I do not know why this happened)

On the other hand, the internal jar you mentioned, ehcache-3.7.1-internal2.jar seems to include your commit ( I checked after decompliation ) and this does not have the above problem.

My question is : Is this the latest stable release candidate for 3.7.1? If not, where can I find this candidate? Directory browsing seems to be disabled for http://repo.terracotta.org and I do not know if there is a newer stable version.

By the way, Initial tests in the test environment did not show any  major problem  yet.

Best regards

11 Mart 2019 Pazartesi 18:35:40 UTC+3 tarihinde Chris Dennis yazdı:

[Chris Dennis]

Right, that's because you’ve checked out the commit hash and not the result of the merge of that commit in to master.  Ehcache 3 uses “daggy fixes” (https://wiki.monotone.ca/DaggyFixes/ apologies for the broken HTTPS) for all of it’s bug fixes.  Instead you probably want to build e3aa12a7d which is the result of merging 74239a93e and the related 9a57a12df in to master.  Regarding the internal release, yes right now internal2 is the latest version (it’s a build of fb7ddb321).

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/ehcache-users/425d4678-f114-4d4a-b857-0be524f1b610%40googlegroups.com.

[Ceyhan Kasap]

Thanks for the explanation. 

Today we changed the jar in production and did not encounter error since than:) Many thanks again..

Regards

12 Mart 2019 Salı 15:03:37 UTC+3 tarihinde Chris Dennis yazdı: