How to know when memcached is full?

[Jitendra]

Hi,
We are planning to implement memcached as part of our project.As the
data is huge, we are expecting it will go beyond the size of cache. So
how can we know that the cache is full, hence we can start another
instance of memcache.

Regards
Jitendra

[Josef Finsel]

First, when memcached gets full, it will start removing items from the cache using the Least Recently Used algorithm. 

Second, you can have multiple instances of memcached running, adding them gets complicated because it will change the hashing algorithm used to determine which cache the data is in. Think of it this way. If you have one cache with items 1-100 in it and add a second cache, most clients are going to spread the items out between the instances. As this is a very crude example, let's assume that every even numbered item is placed in cache 2 and every odd numbered item in cache 1. When you add a second cache and notify the client that it now has two caches, every request for an even numbered item will fail the first time. Even though the item may exist in Cache 1, the client is looking for it in cache 2.

Does that help?

Josef

--
"If you see a whole thing - it seems that it's always beautiful. Planets, lives... But up close a world's all dirt and rocks. And day to day, life's a hard job, you get tired, you lose the pattern."
Ursula K. Le Guin

http://www.finsel.com/words,-words,-words.aspx (My blog) - http://www.finsel.com/photo-gallery.aspx (My Photogallery)  -http://www.reluctantdba.com/dbas-and-programmers/blog.aspx (My Professional Blog)

[Syed Ali]

Hi
I dont think its a good idea to add memcache servers on the fly. This
will almost certainly induce hashing problems.
You should assess the memory requirements and then either start
memcache with appropriate memory or start multiple ones on the same
box or on separate ones.

And to answer your question memcache will probably do LIFO kind of
cleanup to make space for newer items.

By the way, how long you intend to store data items? It is not
persistent ..you know that right?

Hope this helps.

Regards,
Syed

--
Sent from my mobile device

Best,
- Ali

[Jose Celestino]

Words by Jitendra [Wed, Nov 19, 2008 at 01:42:28AM -0800]:

Look for the number of evictions growing when you to stats.
Evicions happening means memcached is removing active objects from cache
to make space for new ones.

--
Jose Celestino | http://japc.uncovering.org/files/japc-pgpkey.asc
----------------------------------------------------------------
"One man’s theology is another man’s belly laugh." -- Robert A. Heinlein

[Vipin Kalra]

I'll second with the opinion that other people have posted and will not recommend to add Memcache on the fly. Rather best practice would be forecast your capacity and plan accordingly for the number of Memcached server which will provide better scalability.  

 

To answer to your original question of how to check if memcahced is full is by checking memcahed status and compare "bytes" with "limit_maxbytes". Pls note that doing it this way can have performance implications depending on how often you write into memcache.

 

Displaying the cache stats:
 pid: 17476
 uptime: 2950218
 time: 1225921019
 version: 1.2.5
 pointer_size: 32
 rusage_user: 0.185971
 rusage_system: 5.735128
 curr_items: 1321
 total_items: 2964
 bytes: 29910319
 curr_connections: 6
 total_connections: 59
 connection_structures: 20
 cmd_get: 3487
 cmd_set: 2964
 get_hits: 2535
 get_misses: 952
 evictions: 0
 bytes_read: 33846167
 bytes_written: 30810647
 limit_maxbytes: 1073741824
 threads: 1

[Jitendra]

Thanks to all for the suggestions.
I didn’t have the idea that memcache uses the clean-up algorithm to
clean the items from the cache.

Any way we are planning to store some historic data in the cache for
faster access.
I agree, we have to plan for the no of instances required and start
them based on the data type that we want to store.

I know that using the memcache status binary (memstat) we can see the
bytes written and limit of max bytes.
To test this condition I executed one small program which will keep on
adding the items to memcache. Deliberately I started memcache server
to assign small amount of memory (./memcached -d -m 1 -l localhost -p
11211).
The server allocate only 1048576 bytes of memory. But I found that the
bytes_written it not going above 910610 and it is deleting the items
from the cache.
So I can’t know what is the actual limit where I have to stop.

Appreciate any suggestions on this regard.
Thanks in advance.

[Clint Webb]

Unfortunately the test you are doing there is not likely going to give you results you are expecting.   Memcached allocated 'slabs' of memory 1mb at a time.  It allocates slabs for different sized data.  The '-m 1' option you have specified will only allow it to create one slab of the data size of the first set you are giving it.  In other words if the first set you do has a value that is 240 bytes long, a slab of 1mb will be created that can hold 4096 items of that size or smaller.  If you tried to do a set with more than 256 bytes of data, then that set would fail because memcached wont be able to allocated any more memory.

Memcached is not built to work very well with a very low amount of memory allocated.  I doubt anyone would recommend you use it with less than 32mb entered.

I'm not sure what you are trying to determine here. 

--
"Be excellent to each other"

[Aaron Stone]

Using the RRD graphs generated by this script:
http://dealnews.com/developers/cacti/memcached.html

Shouldn't be too hard to add a graph for evictions. Then you know,
between the memory used graph and the evictions graph, when your cache
is full and how much it is overflowing and churning.

Aaron

[Jitendra]

Hi,
Thanks for the updates.

I can't just predict the amount of data to be stored in the cache. So
I wan't to start another instance when one is full.
I found that if I make (evict_to_free=0) then I get (SERVER ERROR)
when the cache is full.
So I guess if I will get the (SERVER ERROR) then I can push the
further data to another cache.

Appreciate any suggestions on this regard.

I saw the graphs.
They are really good. We will try to implement those.

Regards
Jitendra

[Josef Finsel]

Jitendra,

What are you looking to get out of memcached? I ask because starting up another instance is a bit contrary to how memcached should be used if you are looking for a cache. Remember, a cache is a copy of data placed in a quickly accessible place, it's not the primary or only copy of the data. The purpose of a cache is to stick data that is accessed frequently in a place where it can be gotten more quickly than going to the source.

The logic behind using a cache is that it takes longer to get something off of disk so you place it in memory.

So, here's how you implement caching, at a very, very basic level. Pre-caching, every time a client wants data, you go to the database and get it. After caching, you look in the cache and return it from there if it's found. If it's not found in the cache, you get it out of the database, stick it in the cache and then deliver it to the client. That way, the next person that goes looking for it can find it in the cache.

memcached implements an LRU algorithm to remove items from the cache when it's getting full. So an item that's been cached but never accessed again might be removed from the cache and, the next time someone goes to get data from the cache the extra step needs to be taken to get it from the database and put it in the cache again. But, since the cache has dramatically lowered your database hits, this takes a lot less time than it would have if all of the cached data wasn't in the cache.

If you're looking to use memcached for some purpose other that this (a full store of all of your data, a queue), then there's a good chance memcached won't do what you want or it will not provide the benefits you're looking for.

Josef

[Clint Webb]

I'm not sure you understand the concept of the cache.  I highly recommend that you do further reading.  What you are suggesting is entirely not the typical way of using it.

Although there are many ways to use memcached, by trying to get it to do something it was not designed to do is likely to just end up with frustration.

For starters, memcache is NOT designed to be able to cache everything.
It is not designed to add more cache servers to the cluster when one gets full.

By doing things the way you are implying, you are invalidating the actual useful purpose of memcache which is a DISTRIBUTED HASH TABLE.

Memcache IS designed to be run with multiple instances on multiple servers, which the clients splitting up requests predictably over all those servers.  Adding an extra server invalidates a good chunk of the data that is already stored... which although not generally a problem, is the opposite of what you seem to be trying to do.

[Henrik Schröder]

On Thu, Nov 20, 2008 at 14:00, Jitendra <jkn...@gmail.com> wrote:

Hi,
Thanks for the updates.

I can't just predict the amount of data to be stored in the cache.

That's usually pretty tough, but that shouldn't really matter, what's important with a cache is the hit ratio, i.e. how often your application can get the data from the cache instead of from your data storage. If your hit ratio is too low, then it might be a good idea to increase the size of your cache.
 

So I wan't to start another instance when one is full.

This makes no sense. When your have filled your memory, memcached will start throwing out the least recently used items first. This is good, these items are probably not needed right now by your application anyway, so when they are discarded, they are least likely to cause a cache miss. How much of the cache is used or how many evictions have happened is a bad metric for measuring cache performance, you can have a great hit ratio, and still have lots of evictions, and it's only your hit ratio that is the important metric for the performance of your application. Only if your hit ratio is going down might it be useful to increase your cache size or go over your expiration times.

Also, adding a memcached instance during runtime is pretty bad since it will change the key distribution. Some clients are less disruptive than others, specifically the "ketama" method minimizes the impact, but if you have n servers and add one, then at least 1/(n+1) of all your items will be moved to the new server, and since that is empty, all those items need to be fetched from your data storage again. You also have no real idea which specific items will have to be re-fetched. Your application must be able to handle that *ANY* get of an item from memcached may fail and return nothing. Therefore, using memcached as some sort of primary data storage is a really bad idea.


/Henrik