eviction policy issue

[karim slaimi]

Hello all,

I am still fighting with my eviction policy on my map.

There is one thing that is weird:

the latest newly added entry into the map is taken into account by the eviction policy and it is evicted!!!

is it normal behaviour that the just added/put entry into a map is evicted.

I thought that when an eviction has to happen it's in order to free space to add the newly/freshly added entry?

So I did not get how it's possible the entry I am adding currently to be then right away used by clients is evicted.

my current configuration is:

<map name="businessMap">    

<in-memory-format>BINARY</in-memory-format>

<time-to-live-seconds>1200</time-to-live-seconds>

<eviction eviction-policy="LFU" max-size-policy="USED_HEAP_SIZE" size="100"/>

</map>

It is not working with both LFU/LRU policies.

I have one Hazelcast node only.

Please help me understanding the eviction algorithm behaviour. I still did not get how it's possible that the last added value is evicted. I didn't even get how it's taken into account by the eviction algorithm. 

Any help pls?

Best regards

Karim SAIMI

[Tom OConnell]

Hi Karim –

I have this config:

    <map name="some-map">

    <in-memory-format>BINARY</in-memory-format>

    <statistics-enabled>true</statistics-enabled>

    <time-to-live-seconds>600</time-to-live-seconds>

        <max-idle-seconds>180</max-idle-seconds>

<eviction eviction-policy="LRU" max-size-policy="USED_HEAP_SIZE" size="1"/>

<entry-listeners>

<entry-listener>com.tao.hz.mem.demo.FooEvictionListener</entry-listener>

</entry-listeners>

    </map>

I also have an entry-evicted listener that prints out the key that was evicted as well as the last key inserted. I tested with both LRU and LFU and in neither case does it evict the most recently inserted key.

I see this output –

           com.tao.hz.mem.demo.FooEvictionListener  : evicted key: 11393: last key: 877039

I tested with Hazelcast 5.0 and this is not the behavior that you describe.

 Try doing something like logging the last key and the key that was evicted.

Tom

[karim slaimi]

Hi Tom,

Thank you for your reply.

I am using Hazlecast 4.0.

my configuration is as below:

<map name="businessMap">

     <in-memory-format>BINARY</in-memory-format>

     <statistics-enabled>true</statistics-enabled>

     <time-to-live-seconds>600</time-to-live-seconds>

        <max-idle-seconds>180</max-idle-seconds>

<eviction eviction-policy="LRU" max-size-policy="USED_HEAP_SIZE" size="70"/>

    </map>

I am doing in the java code: businessMap.set(key, businessValue);

Below the map entry listener log:

Entry Added:EntryEvent{entryEventType=ADDED, member=Member [192.168.2.15]:5777 - 85391363-45bf-4450-93c8-f951d1da6db2, name='businessMap', key=147_Business.2021-10-27_13:00:33_1635333297527, 

Entry Evicted:EntryEvent{entryEventType=EVICTED, member=Member [192.168.2.15]:5777 - 85391363-45bf-4450-93c8-f951d1da6db2, name='businessMap', key=147_Business.2021-10-27_13:00:33_1635333297527, 

As you can see it is removing the entry just added !!!!

I may be doing something wrong! I do not get what is wrong yet.

Do you have any idea how I can fix it?

Best regards

Karim SLAIMI

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/dc5037b1-c489-4c2a-83f2-b011e0cd13den%40googlegroups.com.

[Neil Stevenson]

Are you able to retry with 4.2.2 ?

It could be this https://docs.hazelcast.org/docs/rn/#fixes-421 / https://github.com/hazelcast/hazelcast/pull/18624 / https://github.com/hazelcast/hazelcast/issues/18614 , fixed in 4.2.1

If that makes no difference, would you be able to post a *complete* example (that we can compile & run) that shows the issue please ?

Completeness is the important part. We can't figure it out from the code snippets given so far.

You can remove any domain model you have, and replace it with something simpler like a String of the same size.

That will simplify things and shouldn't make any difference to the behaviour (but if it does, please let us know)

Neil

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAHSeLL1sghR3bCfwV%3DnM8O3d-C-08PCsdwY2ydpSeyq_Gbp_SA%40mail.gmail.com.

[karim slaimi]

Hi Neil,

Thank you for your reply. 

It's definitely linked to the issue Fix LRU comparison algorithm by ahmetmircik · Pull Request #18624 · hazelcast/hazelcast · GitHub

I did a try with both 4.2.1 and 4.2.2 (using LRU and then LFU) and I still have the same behaviour which is evicting the last added entry as you can see below:

Entry Added:EntryEvent{entryEventType=ADDED, member=Member [192.168.2.15]:5777 - 36456203-d195-4288-b358-03b9cc6ca97a, name='businessMap', key=147_Business.2021-10-27_15:18:40_1635340737457, 

Entry Added:EntryEvent{entryEventType=ADDED, member=Member [192.168.2.15]:5777 - 36456203-d195-4288-b358-03b9cc6ca97a, name='businessMap', key=147_Business.2021-10-27_15:18:40_1635340767379, 

Entry Evicted:EntryEvent{entryEventType=EVICTED, member=Member [192.168.2.15]:5777 - 36456203-d195-4288-b358-03b9cc6ca97a, name='businessMap', key=147_Business.2021-10-27_15:18:40_1635340767379,

Best regards

Karim SLAIMI

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAPSE6Xwo_K3yn9scADa4Bno9y7PDAd1H5q5FCu1Voy3aceXFYQ%40mail.gmail.com.

[Neil Stevenson]

A complete example that we can compile & run will be the quickest route to locating the problem 

Is this something you can provide ?

It shouldn't run to more than 30 lines, and shouldn't need to include anything commercially sensitive such as a domain model

Neil

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAHSeLL210wBmLfJ8bpfQoYJaYtdUD6yrZbZqPFEvqZg9s2W60A%40mail.gmail.com.

[karim slaimi]

Hello Neil,

I think I finally figured out the issue.

I did a quick test with a basic springboot app for testing the 4.2.2 hazelcast version. The eviction algorithm is not evicting the last added entry. However, it is evicting some recent entries and not necessarily the oldest ones. So is this normal that is not evicting the oldest entries first.

I also noticed that when I have several hundreds of entries on the map it starts working better than having a few dozens of entries only. 
Actually what is happening in my business app is: I have less than 100 entries into the map so it's always evicting the last added entries even with hazelcast 4.2.2. However with my test app I am adding thousands entries and the eviction policy is working better (not evicting the last entry).

In short, I would like to know if it's normal behaviour that when the map is containing few entries (lets say between 100-200) it's always evicting last entries? If so, how can I change that? because actually the last entry will be consumed right away by a service.

Thank you for your help.

Best regards

Karim SLAIMI

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAPSE6XwyWgpQGKM6XkKioA9eSiKeAEDOD1dDSc%2BVbq54iqG-xA%40mail.gmail.com.

[karim slaimi]

Hello hazelcast team,

can you please tell me if my assumptions are corrects. If so, how can I manage to make it better working on my side please?

Best regards

SLAIMI Karim

[Neil Stevenson]

Can you provide a complete code sample that shows the problem please ?

This should not need much coding, and will help us diagnose what is going wrong

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAHSeLL03MbSBD_1sAYUbxtzyjyqyu3PAvMsXURMvE6H7414J-g%40mail.gmail.com.

[karim slaimi]

Hello,

I did a quick springboot app demo with a basic controller for map eviction test.

The controller is putting <Integer<List<Integer>> entries into the map.

I am trying to insert 4 entries. You will see that the last one is not inserted. It is actually evicted.

please find attached the controller java class file.

my hazelcast configuration is as below:

<map name="some-map">

        <time-to-live-seconds>0</time-to-live-seconds>

        <max-idle-seconds>0</max-idle-seconds>

        <eviction eviction-policy="LFU" max-size-policy="USED_HEAP_SIZE" size="3"/>

</map>

Please let me know how I can fix this behaviour.

Best regards

Karim SLAIMI

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAPSE6XwJASPTgas%3DjV4iqH5kbF7SvF4UtWNC_ktbNn3AWP5VFA%40mail.gmail.com.

[Neil Stevenson]

Thank you! This has helped uncover what I think is a bug in Hazelcast

I have simplified the coding and logged it as an issue here, https://github.com/hazelcast/hazelcast/issues/19862

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAHSeLL1eNSCf0%2ByYuvs4HBLCBc9BvWF9vp3eJs6XJOcPLQ-bGQ%40mail.gmail.com.

[Neil Stevenson]

Hi

 Actually this isn't a bug in Hazelcast, it's one of those cases where the right thing happens but it seems wrong without the detail. We'll get the documentation updated.

 The main design here is that eviction works per partition.

 As documented here, https://docs.hazelcast.com/hazelcast/5.1-snapshot/data-structures/managing-map-memory#understanding-map-eviction

each partition gets an equal share of the allowance.

 This algorithm is necessary for instance, because partitions can move from member to member on a rebalance.

 The example posted inserts 4 keys with large values.

 For the keys chosen, each entry is in a different behaviour. 

 The keys are integers 0, 1, 2 and 3, allocated to partitions 11, 31, 5 and 227.

 ("hazelcastInstance.getPartitionService().getPartition(i).getPartitionId()" will show this)

 What happens is when "put(3, v)" is added to partition 227, this is the point at which the memory threshold has been breached,

and partition 227 needs to find entries to evict to keep it's share of memory in check. Partition 227 only has one entry, key 3 just

inserted, so that's the only one it can delete.

 The behaviour is a consequence of the entry sizes being large in relation to the allowed memory.

 It's almost always a bad idea to reduce the partition count, but for *a test* you could repeat the example with "hazelcast.partition.count" set to "1".

 All entries go to the same partition, partition 0, and when "put(3, v)" happens there are other entries in partition 0 and key 0 can be deleted as it's oldest.

 For production, a better idea is to increase the allowed memory.

 Finally, on a real cluster with several nodes, remember that other apparent issues will arise.

 For example, in a 2 node cluster, it might happen that keys 0 and 2 are on the first node, and keys 1 and 3 are on the second node.

 If the second node is low on memory, it can only choose of it's keys to evict, so either key 1 or 3 will have to go, let's say key 1.

 This means the oldest key, key 0, remains.

 Key 0 is on the first node, evicting it won't help the memory on the second node.

 But if you listed the keys you'd now see keys 0, 2 and 3, and again this could cause confusion.

 

Neil