Memcached flush_all not working as expected?

[Ryan LeCompte]

Has anyone run into the following scenario?

I have two memcached servers setup running with 1GB memory each. I've
maxed out the two servers memory-wise. Then, I go to each one and
issue a flush_all. Now, I then insert elements with an expiration
time of 24 hours. However, it seems like the inserts are not sticking
-- it seems as though memcached is still taking out even the new
elements when it does its evictions. I've checked 'stats' and see that
the eviction counts are increasing, but I didn't think that memcached
would throw away my fresh inserts _after_ the flush? Any ideas?

Thanks,
Ryan

[Ryan LeCompte]

I should also mention that if I manually restart the two memcached
daemons, the program works flawlessly and no evictions take place
(always get cache hits and no misses after fully first inserting all
of them).

Using memcached 1.2.5 on CentOS 4.6 (64 bit).

[Ryan LeCompte]

Okay, now I'm seeing some *very* strange behavior with memcached (at
this point I'm not sure if it's memcached or my cient library,
spymemcached):

I've isolated the test case to be the following:

1) Start up memcached servers 1 and 2 with 1024MB each.
2) Fill up the memcached instances to the point of overflow/eviction
with different keys all pointing to the same value (a 300K string of
characters)
3) Attempt to add 1000 times different types of objects to the
memcached. It looks like 1/4 of the items get stored into memcached,
but the others get evicted. How does the LRU/eviction work in this
case? I would have expected memcached to NOT evict the new items that
I'm inserting, but rather the older ones from step #2 above.
4) I issue a flush_all on both memcached servers.
5) I repeat step #3 above and I am still seeing the same behavior
where my newly inserted items are getting evicted.

Now, here's the VERY VERY strange part:

If I repeat step #2 above but use the same or event different keys, no
evictions occur! I can insert 1000 new entries using the same 300K
string, and none of the newly inserted items gets evicted. Why would
memcache behave in this way? I didn't think it treated any of the
existing values in a "special" way when executing the LRU algorithm?

As you can imagine, I'm a little lost here. Can anyone shed any light?

Thanks,
Ryan

[Ryan LeCompte]

Okay, to add to the madness I made sure that the different objects
that I inserted in step #3 also included the 300K string as part of
its internal data structure, and guess what? No more evictions for
those newly inserted items! Wow, does Memcached treat very large
objects in a special way when it comes to eviction?

Ryan

[Dustin]

On Jul 11, 11:57 am, "Ryan LeCompte" <lecom...@gmail.com> wrote:
> Okay, to add to the madness I made sure that the different objects
> that I inserted in step #3 also included the 300K string as part of
> its internal data structure, and guess what? No more evictions for
> those newly inserted items! Wow, does Memcached treat very large
> objects in a special way when it comes to eviction?

Oh, well those would fall into different slab classes. I'm not
entirely sure if that's a real problem or not, but it's an area that
could probably use further testing.

[Henrik Schröder]

Hi Ryan,

Memcached is internally segmented depending on the size of the data you store, these are called slab classes. If you only store objects of one size, only slabs of the corresponding class will be allocated, and memcached is unable to change this allocation runtime.

Your test fills it with 300k size objects which means that all memory will be allocated to slabs that can hold those objects. When you later try to store smaller objects, there is no memory left to allocate slabs of a smaller size, so they get evicted immediately.

For your test to be accurate you have to fill up memcached with objects of varying sizes, to ensure that a varied mix of slab classes are allocated.


/Henrik Schröder

[Ryan LeCompte]

Thanks! This makes perfect sense.