memcached not freeing up ram?

[tongueroo]

Hi guys,

I was wondering if memcached frees up unused ram after it expires or
if it just keeps using the ram in there but invalidates the key.

Im hearing from people that it does free up the ram when caches
expire, but the behavior that we see on our site is this.

Here's the pattern I've been seeing with the memcached. Memcached for
us doesnt seem to release memory, it just keeps growing until
eventually slowness shows up on our site and then we are required
restart the memcached instances and mongrels afterwards because
newrelic starts showing that memcached reads are slow. The reason I
think that memcached doesnt release memory for us is that it takes us
about a week before the 1GB memcached on a slice gets full and our
longest expire is 6 hours...

When we start up our memcached servers and leave them running, they
dont get full right away. It takes them about 1 week to eventually
get full and run out of RAM. Our highest expiry is 6 hours, so
memcached should free space up after that. So, since the memcached
doesnt gets full within 6 hours it seems like memcached isnt freeing
up the expired caches.. If it did get full within 6 hours, then
obviously we need to allocate more RAM.


Thanks in advance for any responses,
Tung

[Dustin]

On Mar 17, 10:27 am, tongueroo <tongue...@gmail.com> wrote:
> Hi guys,
>
> I was wondering if memcached frees up unused ram after it expires or
> if it just keeps using the ram in there but invalidates the key.

No, it should never free RAM.

> Here's the pattern I've been seeing with the memcached. Memcached for
> us doesnt seem to release memory, it just keeps growing until
> eventually slowness shows up on our site and then we are required
> restart the memcached instances and mongrels afterwards because
> newrelic starts showing that memcached reads are slow. The reason I
> think that memcached doesnt release memory for us is that it takes us
> about a week before the 1GB memcached on a slice gets full and our
> longest expire is 6 hours...

It sounds like you're just giving it more memory than you can
afford. It won't grow (much) beyond what you tell it that it can. If
1GB is too big for you, tell it it can only have 512MB.

[Roch Delsalle]

what happens when it's full ?

--
☠
Roch DELSALLE
55, rue de Bellechasse.
75007 Paris
FRANCE
rdel...@gmail.com

[Dustin]

On Mar 17, 11:07 am, Roch Delsalle <rdelsa...@gmail.com> wrote:
> what happens when it's full ?

There's a distinction between the memory it's asked the system for
and the memory it's using internally.

It will continue to reuse the memory it allocates throughout its
lifetime. What it will not do is hand it back to the system for other
things to use.

[Perrin Harkins]

On Tue, Mar 17, 2009 at 2:07 PM, Roch Delsalle <rdel...@gmail.com> wrote:
> what happens when it's full ?

It silently drops data. This is one of the reasons people say not to
use it as a database.

- Perrin

[Tadeu Alves]

hello there guys, i was looking for some help to configurate my memcached api into a mysql server with innodb tables.

 

Server config

2x Quad Core Intel® Xeon® E5450, 2x6MB Cache, 3.0GHz, 1333MHz FSB
8GB 800MHz Memory
2x SAS 73GB 15000RPM in RAID 1

 

SO and API config

Red Hat Enterprise Linux 5 x64

MYSQL 5.0.45 and our database size is about 9gb (we don't clear mdl_log)  PHP 5.1.6

[Dustin]

If you would like to start a new topic, use the ``new post''
button. Nothing in your post has anything at all to do with memcached
not freeing RAM.

[Jose Celestino]

Words by Tadeu Alves [Tue, Mar 17, 2009 at 07:09:48PM -0300]:

> hello there guys, i was looking for some help to configurate my memcached
> api into a mysql server with innodb tables.
>

OMG, this must be the thread highjacking week.

--
Jose Celestino | http://japc.uncovering.org/files/japc-pgpkey.asc
----------------------------------------------------------------
"One man’s theology is another man’s belly laugh." -- Robert A. Heinlein

[Marc Bollinger]

Funny you should mention that! I've been having this problem, see...

--
Marc Bollinger
mboll...@gmail.com

[Roch Delsalle]

yes but if for instance I'm building dynamic SQL queries at somes point the memory usage will keep growing even if the data is released.

[Perrin Harkins]

On Wed, Mar 18, 2009 at 1:02 PM, Roch Delsalle <rdel...@gmail.com> wrote:
> yes but if for instance I'm building dynamic SQL queries at somes point the
> memory usage will keep growing even if the data is released.

How would that happen? It just drops old data and replaces it with
your new stuff. It doesn't increase the overall size.

- Perrin

[Dustin]

On Mar 18, 10:09 am, Perrin Harkins <phark...@gmail.com> wrote:

> On Wed, Mar 18, 2009 at 1:02 PM, Roch Delsalle <rdelsa...@gmail.com> wrote:
> > yes but if for instance I'm building dynamic SQL queries at somes point the
> > memory usage will keep growing even if the data is released.
>
> How would that happen?  It just drops old data and replaces it with
> your new stuff.  It doesn't increase the overall size.

It doesn't try very hard to reuse memory when it hasn't allocated as
much as you've given it. Trond's put in a change recently that will
allow it to take a quick pass to look for expired objects before
allocating or hitting the LRU.

The aggressiveness of this should also be configurable (working
through prerequisites to that).

[Perrin Harkins]

On Wed, Mar 18, 2009 at 1:26 PM, Dustin <dsal...@gmail.com> wrote:
>  It doesn't try very hard to reuse memory when it hasn't allocated as
> much as you've given it.

Sure, and I'm guessing it's not religious about making sure don't go
over the limit even a fraction, but there should be no significant
growth past the limit you set, right? And if you store more data at
that point, it should drop old data to make room for it rather than
growing.

- Perrin

[Dustin]

On Mar 18, 10:39 am, Perrin Harkins <phark...@gmail.com> wrote:

> On Wed, Mar 18, 2009 at 1:26 PM, Dustin <dsalli...@gmail.com> wrote:
> >  It doesn't try very hard to reuse memory when it hasn't allocated as
> > much as you've given it.
>
> Sure, and I'm guessing it's not religious about making sure don't go
> over the limit even a fraction, but there should be no significant
> growth past the limit you set, right?  And if you store more data at
> that point, it should drop old data to make room for it rather than
> growing.

The memory limits you give it are specifically how much memory it's
allowed to use for cache. It will use additional memory for
connections (in my tree on my system, that's 300 bytes each) and
miscellaneous buffers (e.g. when you do stats, the response is
buffered in a new allocation.

Some of that can actually reuse slab memory, but it doesn't
currently.

[tongueroo]

memcached reads are reported as very slow. 10+ seconds.

Tung

On Mar 17, 11:07 am, Roch Delsalle <rdelsa...@gmail.com> wrote:

> rdelsa...@gmail.com

[tongueroo]

there a way to check how much ram us being used for caching? Vs
overall. Were using ruby's memcache-client. MemCache#stats gives some
stats but it seems to be the same as what ps aux tells us.

[Dustin]

On Mar 18, 8:00 pm, tongueroo <tongue...@gmail.com> wrote:
> memcached reads are reported as very slow. 10+ seconds.

Are you giving it more RAM than you actually have? I would expect
that behavior if it were fetching from swap.

[tongueroo]

Thats a good point. Um. maybe thats swap is whaat is happening.
Though I dont think that we are swapping, that would makes the most
sense.
Below are our 9 memcached servers
1. how much free ram as listed by free -m
2. memcached version number
3. how much ram is being used by memcached right now, its near full
again (full at 1GB)

https://gist.github.com/e886d958a4bc8e103810

Right now we it doesnt appear that we have swapping.
However, we do run our memcached instances on the same slices as our
app servers where our mongrels live. Perhaps
spikes in mongrels is causing it to swap..

Do those free numbers look good?
From what Ive been told is that the second row of free is more
important..

-/+ buffers/cache: XXX XXX

Thats how much actual free ram we have before we start swapping A gig
on each slice seems o-plenty.

Thanks again for all the helpful feedback and responses thus far.

Tung

[Matt Ingenthron]

tongueroo wrote:
> Thats a good point. Um. maybe thats swap is whaat is happening.
>

Note there is a "-k" flag to memcached which, depending on underlying
OS, can lock down the pages so they won't page out to swap space. Since
your mongrel memory usage will be variable, you may be able to use that
flag to avoid a situation where memory pressure evicts memcached's pages.

Also, I don't know cruby's internals all that well, but running a quick
rails app here locally seems to show that all of the memory allocations
are in heap. Since those won't be given back until the process exits,
if you're running lots of mongrels without ever exiting you're likely
not as efficient about memory usage as you could be. I only see
a very small amount of memory mmap()'d. If things are evenly
distributed and cruby (and the underlying memory allocator you're using)
are efficient about memory allocation, it may not be too bad.

One solution for this would be to recycle the mongrels on a regular
basis. Maybe you're already doing that? There are some other solutions
too.