Eviction policy is not working

[karim slaimi]

Hello Hazelcast team,

I installed the hazelcast open-source version.

I am trying the eviction policy on my "test" map but it's not working.

Below my map conf:

<map name="test">    

<time-to-live-seconds>600</time-to-live-seconds>

        <max-idle-seconds>180</max-idle-seconds>

<eviction eviction-policy="LRU" max-size-policy="USED_HEAP_SIZE" size="100"/>

</map>

<map name="default">    

...

</map>

When it reached 100MB it stopped adding any new entry but it did not do the eviction part.

So I am not able to add new entries since the eviction is not working.

Can you please help me fix it to get the right eviction behaviour.

Best Regards

Karim SLAIMI

[Tom OConnell]

Hi -

How are you measuring the 100MB? The eviction configuration is set to "100MB per member"
. Have a look at the eviction configuration docs.

Tom

[karim slaimi]

Hi Tom,

Actually, I have one member only.

I am using hazelcast man-center so then I can check the entries and size of my "test" map.
Am I doing something wrong with the configuration?

Because it stopped adding any new entry but it did not do the eviction.

Any help please.

Best regards

Karim SLAIMI

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/7d61271a-6638-4d4e-90ae-61aa40349359n%40googlegroups.com.

[Tom OConnell]

Hi Karim –

 

The config looks ok. You have configured both expiration and eviction. Map entries should be removed after either 10 minutes (600 s) overall, or after 3 minutes of no activity on a key. The 100MB is calculated to the per-partition size, so it’s possible that this might be a little deceiving.

So, a couple of questions –

• What data are you loading?
  How large are the objects you’re caching?
• What are your keys?
  Is there any chance they’re not unique?
  How large are the keys?

 

You say that when it reached 100MB, it stopped adding data – how? Did your client throw exceptions? Terminate?

Could your keys have wrapped, so you’re doing updates to existing data?

 

What are you seeing in the management center? On the map, you should see the entry and backup counts and the size. You should also see the rate of ‘put’s and the latency. Does this change when you hit 100MB?

 

Tom

[karim slaimi]

Hi Tom,

let me explain to you the context: actually I would like to set a maximum heap size to a specific map. 
if this maximum heap size is reached, I would like that the least recently used entry be evicted in order to put the new entry.

So how should I do it please?

(by the way I removed the expiration configuration. I am focused on the eviction policy one only for now)

Best regards

Karim

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/8cf2e7a7-88f2-40fd-9273-10e8accd4cdbn%40googlegroups.com.

[Tom OConnell]

Hi -

I'd still like to know how you're populating the map. I don't understand if the client threw an exception or stopped running. If it didn't stop and there were net, no new objects added to the map, that's surprising - unless the keys wrapped and the client were overwriting older objects.

Understanding that is key to figuring this out.

You can add a map event listener, as shown here: https://docs.hazelcast.com/imdg/4.2/events/object-events.html#listening-for-map-events.

If you're using java, please take the "Listen" class from that page and update it so that the map name in the 'main' method matches the map you're using. Let that run while your existing client is running. 

It would be useful to add some logging to your existing client - log the keys and the size (yourMap.getSize()) before and after each operation. Please make sure you're catching all exceptions and do not allow the client to exit (i.e. System.exit(nnn)).

You've said that the client stopped populating the map, but not how it stopped or how it terminated - please let me know that.

Tom

[karim slaimi]

Hi Tom,

so I am caching business data. The amount size is between 10-100 MB for each entry.

a map key is a combination of userID+a timestamp <userID+a timestamp, business object>

My hazelcast eviction policy is as below:

<eviction eviction-policy="LRU" max-size-policy="USED_HEAP_SIZE" size="100"/>

Below is a screen-shot of my map using man-center.

I put 3 entries with a total size of 74.42MB.

When I tried to add a 4e entry nothing happened. This 4e entry size is about 40MB.

There is no exception neither on the client side nor on the server log side.

So I tried with a another entry (about 16MB) and it was able to add it into the map

I hope this helps to figure out my issue.

Kind regards

karim SLAIMI

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/840b2a39-e9f9-4a1c-9629-a0a67bee61d2n%40googlegroups.com.

[karim slaimi]

Hi Tom,

I did add an entry map listener as you suggested.

There is a weird behaviour: actually the eviction is happening. 

However, whatever the eviction policy I set (LFU or LRU), when the max-size is reached (max-size-policy="USED_HEAP_SIZE"), it is always evicting the last inserted entry!!!!

I am not expecting this behaviour. Is it a normal one?

Best regards

Karim SLAIMI

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/840b2a39-e9f9-4a1c-9629-a0a67bee61d2n%40googlegroups.com.

[Tom OConnell]

Hi -

I believe the issue is that, if you're not fetching the objects you insert, they're not counted as used. Can you add a thread that dreads from the map? That should result in changed behavior. Make sure that you're not reading all the keys equally.

Tom

[Bence Eros]

Hello Karim,

On Tue, Oct 19, 2021 at 1:11 PM karim slaimi <karim....@gmail.com> wrote:

Hi Tom,

I did add an entry map listener as you suggested.

There is a weird behaviour: actually the eviction is happening. 

However, whatever the eviction policy I set (LFU or LRU), when the max-size is reached (max-size-policy="USED_HEAP_SIZE"), it is always evicting the last inserted entry!!!!

Small note here: if you enable backup reads in your map config, then those backup reads won't be taken into account by the eviction algorithm. So if you do read the entry, but the read returns the value from a backup partition, the entry from the primary partition can still get evicted.

I don't know if this is your case.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAHSeLL1Cn5Bnex4uEW6aexiCHG%3DgYJqqVTsUWNTHrafAyQu-Aw%40mail.gmail.com.

--

Bence Erős

dev@management-center

er...@hazelcast.com

2 W 5th Ave, Ste 300 | San Mateo, CA 94402 | USA +1 (650) 521-5453 | hazelcast.com

[karim slaimi]

Hello Bens,

Thank you for your help.
However I did not enable backup reads.

The issue is definitely linked to the algorithm sampling entries during the eviction process. And it's always evicting the last entry.
Each entry I add into the map is used right away (get from the map). so then when eviction happened it is evicting last entry which causing an issue on my application since I am not able to get back the entry from the map (the entry was just evicted).

So,
1- I did my custom eviction comparator as explained here Managing Map Memory (hazelcast.com)
2- Then, I change my hazelcast eviction conf as below:
<eviction comparator-class-name="com.XXX.CustomMapEvictionPolicy"/>

3- I added the jar with CustomMapEvictionPolicy class into hazelcast lib (and also into user-lib).

4- However when the eviction stage is reached (no more heap space), the CustomMapEvictionPolicy is not called. and the hazelcast cluster is destroying my client connection.

Am I missing something? Am I doing something wrong?

Thank you in advance for your help.

Best regards

Karim SLAIMI

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAN%2Bc%3DYU3rch_ji2iLtfwFDS6LEv8UKrwTze2c4D-iea5cheiyQ%40mail.gmail.com.

[karim slaimi]

Hello Bence and Tom, 

Thank you for your help.
However I did not enable backup reads.

The issue is definitely linked to the algorithm sampling entries during the eviction process. And it's always evicting the last entry.

Each entry I add into the map is used right away (get from the map). So then when eviction happened it was evicting the last entry which caused an issue on my application since I am not able to get back the entry from the map (the entry was just evicted).

So,
1- I did my custom eviction comparator as explained here Managing Map Memory (hazelcast.com)

2- Then, I changed my hazelcast eviction conf as below:
<eviction comparator-class-name="com.XXX.CustomMapEvictionPolicy"/>

3- I added the jar with CustomMapEvictionPolicy class into hazelcast lib (and also into user-lib).

4- However when the eviction stage is reached (no more heap space), the CustomMapEvictionPolicy is not called. and the hazelcast cluster is destroying my client connection.

Am I missing something? Am I doing something wrong?

Thank you in advance for your help.

Best regards

Karim SLAIMI

[Neil Stevenson]

Same as this ? https://stackoverflow.com/questions/69672512/hazlecast-eviction-is-not-working-with-used-heap-size-with-hazlecast5-0

[karim slaimi]

Hello Neil,

actually it's not working as this hazelcast - Hazlecast eviction is not working with USED_HEAP_SIZE with hazlecast5.0 - Stack Overflow using USED_HEAP_SIZE. It is always removing the last added entry. it seems caused by the sampling eviction algorithm.

But it's also not working using my custom eviction policy as I explained in my previous email.

Any help please?

Best regards

Karim SLAIMI

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/4d93ab2e-9cdb-44c7-b7df-ba5f616ca42cn%40googlegroups.com.

[Neil Stevenson]

In order to check this further, would you be able to answer the request on StackOverflow for more detail please ?

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAHSeLL1g5nMWezchWKJU%3Dv6qc-NvwxQb6m6gj%3DzTkN%3DGV%2Bi98Q%40mail.gmail.com.

[Neil Stevenson]

See https://groups.google.com/d/msgid/hazelcast/CAHSeLL1eNSCf0%2ByYuvs4HBLCBc9BvWF9vp3eJs6XJOcPLQ-bGQ%40mail.gmail.com 

and https://github.com/hazelcast/hazelcast/issues/19862

May be the same issue, or at least related. A fix that went into 4.1.4 / 4.2.1 that may not be complete