Ehcache active removal of stale data from disk. Disk Optimization

[dakks...@gmail.com]

Hi,

Ehcache Version: 3.8

-XX:+UseG1GC: Heap + OffHeap +Disk, multiple cache buckets(regions)

JVM conf

-Xms2g
-Xmx2g
-XX:MaxDirectMemorySize=10g

-XX:+UseG1GC

Due to large disk allocation, stale data keep living on the disk. Hence if a bucket reaches max capacity it always remain full.

Current data size is around 175GB per node.

Current config is giving ~1.5 ms response, but intermittently it shoots up to ~10ms.

Any tuning with respect to theadpools, persistence=true or false, linux optimization is welcome. Even any troubleshooting tips are welcome.

Or Any advice on actively evicting data without affecting the performance of get/put/delete operations.

Also I read about TickingTimeSource can improve performance ~20-30%. Any practical experience regarding that

Config XML:

<?xml version="1.0" encoding="UTF-8"?>

<ehcache:config
    xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
    xmlns:ehcache='http://www.ehcache.org/v3'
    xmlns:jsr107='http://www.ehcache.org/v3/jsr107'
    xsi:schemaLocation="
        http://www.ehcache.org/v3 http://www.ehcache.org/schema/ehcache-core-3.0.xsd
        http://www.ehcache.org/v3/jsr107 http://www.ehcache.org/schema/ehcache-107-ext-3.0.xsd">



    <ehcache:persistence directory="/opt/ehcache/data" />


    <ehcache:cache alias="basicCache">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">2</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:disk unit="MB" persistent="true">10</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>


    <ehcache:cache alias="1">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">300</ehcache:offheap>
            <ehcache:disk unit="MB" persistent="true">500</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="2">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">300</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">500</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="3">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">300</ehcache:offheap>
            <ehcache:disk unit="MB" persistent="true">500</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>


    <ehcache:cache alias="4">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">300</ehcache:offheap>
            <ehcache:disk unit="MB" persistent="true">500</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="5">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">400</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">15</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>


    <ehcache:cache alias="7">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">400</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">2</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>


    <ehcache:cache alias="10">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">400</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">5</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>


    <ehcache:cache alias="12">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">400</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">2</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>


    <ehcache:cache alias="15">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">700</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">5</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="20">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">700</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">5</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>


    <ehcache:cache alias="30">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">700</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="false">60</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>


    <ehcache:cache alias="40">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">700</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">3</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>


    <ehcache:cache alias="60">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">500</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">20</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="90">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">500</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">2</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="120">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">500</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">20</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="180">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">500</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">10</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="240">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">500</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">1</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>


    <ehcache:cache alias="360">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">500</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">10</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="720">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">500</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">4</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="1440">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:offheap unit="MB">500</ehcache:offheap>
            <ehcache:disk unit="GB" persistent="true">2</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="2880">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">172800</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:disk unit="GB" persistent="true">1</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="7200">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">432000</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:disk unit="GB" persistent="true">20</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="8640">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">518400</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:disk unit="GB" persistent="true">1</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="14400">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">864000</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:disk unit="GB" persistent="true">10</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="43200">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">2592000</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:disk unit="GB" persistent="true">1</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="144000">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">8640000</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:disk unit="GB" persistent="true">1</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="432000">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">25920000</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:disk unit="GB" persistent="true">1</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="525600">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">31536000</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:disk unit="GB" persistent="true">1</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="1440000">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">86400000</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:disk unit="GB" persistent="true">1</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

    <ehcache:cache alias="4320000">
        <ehcache:key-type>java.lang.String</ehcache:key-type>
        <ehcache:value-type>byte[]</ehcache:value-type>
        <ehcache:expiry>
            <ehcache:ttl unit="seconds">259200000</ehcache:ttl>
        </ehcache:expiry>
        <ehcache:resources>
            <ehcache:heap unit="MB">1</ehcache:heap>
            <ehcache:disk unit="GB" persistent="true">60</ehcache:disk>
        </ehcache:resources>
    </ehcache:cache>

....

Looking forward for some great advice from the community.

[dkode]

Is this group active ? Or If anybody knows about a more active ehcache group. It will be helpful.

[Chris Dennis]

Some things that jump out at me…

 

basicCache: I suspect the 2 second TTL here is just causing disk churn… I’d consider leaving this as heap only if at all possible. In particular even if your disk persistent… with a 2 second TTL everything is going to be expired when the cache comes back from a re-deploy/reboot.

 

Why are the heap tiers so small? 1MB is pretty tiny. If you’re looking to run some kind of tightly scoped session cache I would consider sticking to a count based cache… they are cheaper especially when your capacity constraint is so tight.

 

For the caches where the disk and offheap tiers are close in capacity you should consider whether you need persistence… normal cache operation will be much more performant if you just have heap and offheap.

 

Anything beyond this would require way more information about the what the application does with the caches (the access pattern), what the underlying hardware looks like, and what exactly your target metrics/SLAs are. It’s probably also going to require a bunch of performance testing and experimentation.

 

Chris

--
You received this message because you are subscribed to the Google Groups "ehcache-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ehcache-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ehcache-users/75dcdd20-fcd5-4306-af2a-b0f374cb94acn%40googlegroups.com.

[dakks...@gmail.com]

Thanks a lot Chris for the response

basicCache: I suspect the 2 second TTL here is just causing disk churn

Please ignore basicCache, we are not using this bucket. Buckets are named as per the ttl, although for smaller buckets ttl has been increased to 24hr to support some fail-over use cases and serve stale data.

Why are the heap tiers so small?

Since we are using byte arrays as value for caching. (string -> gzipped_byte_array) i suppose offheap and onheap should perform equally as there is no serialization overhead. This is also to avoid GC related issues. Since we can't create a cache without heap tier it is necessary.

For the caches where the disk and offheap tiers are close in capacity you should consider whether you need persistence… normal cache operation will be much more performant if you just have heap and offheap.

I got your point, but due to resource constrains we can't have a lot of RAM. So idea is to use disk while sacrificing a little on SLAs. Per node ~150GB cache is present. Although a lot might be stale due to lazy expiry

The access pattern

Cache is centralized and app servers access it over HTTP. Ehcache is deployed as a webapp in tomcat container. And other app servers access the cache webapp

Hardware:

8CPU, 16GB ram (No balooning)

Per Node Traffic (GET +PUT) : 60k RPM (40K Get, 20K put)

On an average system is working as per the excepted SLA of 1ms resp time. But intermittently it is giving spikes. I think these are related to evictions which is getting triggered as a lot a stale data is present.

Can a background task be run to iterate over a sample of cache entries and delete the expired one's or would it be inviting more troubles.As i have a lot of cache i suppose it may cause OOM in iteration.

Also how to properly configure the thread pools, i have not configured any at the moment. What is the default configuration.

[dakks...@gmail.com]

HI Chris,

Hope i have cleared the doubts related to config and usage. If not please correct me, i will be happy to provide more details.

[bakiaba...@gmail.com]

I don't think you should be using UseG1GC with an Xmx heap setting of 2GB. UseG1GC is optimal for heap sizes of 4GB or higher.

If you insist on UseG1GC, then increase the max heap to -Xmx4g.    If you insist on -Xmx2g, then switch from UseG1GC to, say, UseParallelOldGC.

[dkode]

@Chris

Any advice on actively evicting data without affecting the performance of get/put/delete operations much because as per my knowledge ehcache is only evicting the stale data on get events or if storage is full.

Is there a safe way to iterate over the cache keys  and check if it is expired  and possibly delete and free up disk space. Can it cause fragmentation issues ?

I can thinking of running a background thread to remove expired data from the cache. But i am not sure how should i iterate over the cache safely. Any suggestion from you side ?

[dkode]

@Chris Any suggestions for this. Any help will be deeply appreciated.