MemCached Evictions

[rahul_kcle]

Hi,

We are using memcached for almost an year in our production
environment.

From last 2 weeks i am seeing evictions happening on our memcached
boxes even though there is lot of memory left . Here are the stats
from memcached


STAT bytes_read 434627188758
STAT bytes_written 357821569260
STAT limit_maxbytes 23622320128
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 5
STAT conn_yields 0
STAT bytes 14573115225
STAT curr_items 1853350
STAT total_items 66439158
STAT evictions 7000591


Bytes is much lessser than limit_maxbytes.

I dont understand why the evictions are happening even though there is
space left.

We are using version 1.4.1

We rae using 4 memcached boxes using consistent hashing.


Any help is appreciated.

Thanks.

[Mikael Fridh]

On Nov 4, 6:32 pm, rahul_kcle <vishnudee...@gmail.com> wrote:
>
> From last 2 weeks i am seeing evictions happening on our memcached
> boxes even though there is lot of memory left . Here are the stats
> from memcached
>
> STAT bytes_read 434627188758
> STAT bytes_written 357821569260
> STAT limit_maxbytes 23622320128
> STAT accepting_conns 1
> STAT listen_disabled_num 0
> STAT threads 5
> STAT conn_yields 0
> STAT bytes 14573115225
> STAT curr_items 1853350
> STAT total_items 66439158
> STAT evictions 7000591
>
> Bytes is much lessser than limit_maxbytes.

See stats slabs, possibly your slab distribution profile have shifted
over time.

Mikael

[Kate Wang]

We are experiencing high reclaims instead of evictions. Could slab distribution shift cause that as well?

If the slab distribution shifted could cause high eviction rate, what's the best way to fix it or avoid it?

Thanks!

[dormando]

reclaims are good, evictions are bad

[vishnu]

What is slab distribution?

How can i resolve this issue?

Thanks.

[Artur Ejsmont]

Mikael can be right.

Each slab is 1mb and its designated to hold items of particualr size.
So if you had 5000 items over 500KB inserted into the cache at some
time ( at the beginning ) they would consume 5GB (they would take 1MB
each even if they were 501KB).

So after some time caching patterns change and you dont insert big
items any more. This 5GB is wasted if you are not using such big items
any more. So distribution changes and big items are not needed any
more but more smaller items cant fit into the cache as memcached cant
put small items into slabs designated for big items.

Sorry if my explanation is not super good :)

checkout this tool out though
http://artur.ejsmont.org/blog/content/first-version-of-memcache-stats-script-based-on-memcachephp

i took a open source stats scripts and added detailed view of slabs.
It should tell you how many slabs you have allocated per size and how
many items are there. I guess this should give you good idea what is
really happening.

:- )

art

--
Visit me at:
http://artur.ejsmont.org/blog/

[Kaiwang Chen]

Is there any way to shrink slab class so that space allocated at peak
time can be reused for other slab classes later?

Thanks,
kc

2010/11/5 Artur Ejsmont <ejsmon...@gmail.com>:

[Artur Ejsmont]

there was supposed to be a rebalancing functionality but i am not sure
if it went into stable already? lately i was busy doing other things.

can anyone confirm what is the status of reclaiming slabs please? I am
also interested is it available.

thanks

art

[dormando]

Not yet, no :/