Limitation for Memcached items ?

[TopLess]

Hello,

it seems that there is a limitation concerning the Memcached Items
parameter.
On two differents servers, we observer a regular increasing of this
number (up 125.000) then it stays constant even if many news html
pages are generated on the websites.

How delete this limitation ?

Thanks

[Colin Pitrat]

Do you mean you reach max memory usage for your memcached server ? You
may want to increase your maximum memory allocated to memcached:

$ memcached -h
(...)
-m <num> max memory to use for items in megabytes, default is 64 MB
(...)

Be careful to not put it too high, as you certainly don't want to
swap. It is only memory reserved for items, so it means your memcached
daemon needs more memory, and you also need to keep some memory for
the system, plus some other services that may run on the same box.

2009/5/14 TopLess <0w9za2b...@jetable.com>:

[TopLess]

Hi Colin,

thx for your answer.
This is our configuration :
# memory
-m 10000

In fact, each server had 16 GB and only 3 GB is reserved by system, so
there is no problem concerning the free memory.
So we don't reach max memory usage, but memcached items stays strictly
constant.
An idea ?

On 14 mai, 11:19, Colin Pitrat <colin.pit...@gmail.com> wrote:
> Do you mean you reach max memory usage for your memcached server ? You
> may want to increase your maximum memory allocated to memcached:
>
> $ memcached -h
> (...)
> -m <num>      max memory to use for items in megabytes, default is 64 MB
> (...)
>
> Be careful to not put it too high, as you certainly don't want to
> swap. It is only memory reserved for items, so it means your memcached
> daemon needs more memory, and you also need to keep some memory for
> the system, plus some other services that may run on the same box.
>

> 2009/5/14 TopLess <0w9za2bm29ax...@jetable.com>:

[Colin Pitrat]

You can check the output of stats from telnet interface. You can also
have a look at stats slabs.

$ telnet <server> <port>
stats
(...)

you should have a look at evictions, bytes and limit_maxbytes.

$ telnet <server> <port>
stats slabs
(...)

for each slab, you should check free_chunks and free_chunks_end, and
at global level you should check total_malloced.

2009/5/14 TopLess <0w9za2b...@jetable.com>:

[Dustin]

On May 14, 2:37 am, TopLess <0w9za2bm29ax...@jetable.com> wrote:
> Hi Colin,
>
> thx for your answer.
> This is our configuration :
> # memory
> -m 10000
>
> In fact, each server had 16 GB and only 3 GB is reserved by system, so
> there is no problem concerning the free memory.
> So we don't reach max memory usage, but memcached items stays strictly
> constant.
> An idea ?

That does seem odd. The output of "stats" and "stats items" and
"stats slabs" would help.

What expiration are you setting for these items? What version of
memcached (stats will include that).

[luciano11]

My understanding of the current software is that an individual memory
bucket will grow in size as needed as long as there is free memory,
but that growth of one memory bucket cannot cause another bucket to
evict data.

So is it possible that the cache is partially filled by another memory
bucket that has data with no expiration?

[Colin Pitrat]

In fact, with default settings, memcached allocate by page of ~1 MB
(exact page size is the nearest multiple of chunk size). Each time a
page is full (i.e. no free chunk), if allocated memory does not reach
the memory limit, another page is allocated for this chunk size.

So if you allocate a lot of small chunks for small items, and they are
evicted, the memory cannot be reused for bigger chunks. The only
solution is to restart memcached, a flush will not help.

There is a mode that always allocate 1MB pages, which allows to
reallocate a page to another chunk size. However, I'm not sure that it
is the problem here. Having output of stats and stats slabs will help
to diagnose.

2009/5/14 luciano11 <luci...@gmail.com>:

[TopLess]

Hi,

to help us to diagnose this problem, this is our stats :

Memory usage (server) :
free : 11.7 GB
used : 3.98 GB

Memcached bytes used :
used 1.23 G (constant now)
max available : 1.9 G

memcached current connections :
52

memcached hits and misses :
hits/sec : 14
misses/sec : 2

memcached items :
items cached : 94814 (constant now)

memcached requests/s :
get/s : 16
set/s : 0

memcached network traffic :
output : 4 k
input : 1109


If memcached allocate by page 1MB, considering that we have 100.000
memcached items, it corresponds to 100 GB memory (impossible).
Thanks for your help !

On 14 mai, 23:12, Colin Pitrat <colin.pit...@gmail.com> wrote:
> In fact, with default settings, memcached allocate by page of ~1 MB
> (exact page size is the nearest multiple of chunk size). Each time a
> page is full (i.e. no free chunk), if allocated memory does not reach
> the memory limit, another page is allocated for this chunk size.
>
> So if you allocate a lot of small chunks for small items, and they are
> evicted, the memory cannot be reused for bigger chunks. The only
> solution is to restart memcached, a flush will not help.
>
> There is a mode that always allocate 1MB pages, which allows to
> reallocate a page to another chunk size. However, I'm not sure that it
> is the problem here. Having output of stats and stats slabs will help
> to diagnose.
>

> 2009/5/14 luciano11 <lucian...@gmail.com>:

[Colin Pitrat]

One page contains many chunks, and a chunk can contain one item.

Here is an example on my server:
$ echo "stats slabs" | netcat <host> <port> | grep chunk_size
STAT 1:chunk_size 104
STAT 4:chunk_size 224
STAT 5:chunk_size 280
STAT 6:chunk_size 352
STAT 7:chunk_size 440
STAT 8:chunk_size 552
STAT 9:chunk_size 696
STAT 10:chunk_size 872
STAT 11:chunk_size 1096
STAT 12:chunk_size 1376
STAT 13:chunk_size 1720
STAT 14:chunk_size 2152
STAT 15:chunk_size 2696
STAT 16:chunk_size 3376
STAT 17:chunk_size 4224
STAT 18:chunk_size 5280
STAT 19:chunk_size 6600
STAT 20:chunk_size 8256
STAT 21:chunk_size 10320
STAT 22:chunk_size 12904
STAT 23:chunk_size 16136
STAT 24:chunk_size 20176

The number after stat is the slabclassid. The chunk_size is in bytes.
If we look at the slabclassid 1:
$ echo "stats slabs" | netcat obeat103 33799 | grep "STAT 1:"
STAT 1:chunk_size 104
STAT 1:chunks_per_page 10082
STAT 1:total_pages 4
STAT 1:total_chunks 40328
STAT 1:used_chunks 10504
STAT 1:free_chunks 29824
STAT 1:free_chunks_end 5154

So chunks in this page are 104 bytes, and there are 10082 chunks by
page, i.e the page is 0.999954 MB. On my server, 4 pages have been
allocated for this slabclassid, resulting in 40328 chunks of 104 bytes
availables. But in fact, only 10504 of these chunks are used.

2009/5/15 TopLess <0w9za2b...@jetable.com>:

[Clint Webb]

On Fri, May 15, 2009 at 3:54 PM, TopLess <0w9za2b...@jetable.com> wrote:

Hi,

to help us to diagnose this problem, this is our stats :

Memory usage (server) :
free : 11.7 GB
used : 3.98 GB

Its kind of odd that your total memory usage is kind of maxed out at about 4gb.  4gb is kind of a magic number on some operating systems, Especially on 32 bit systems.

What are you running this on?

--
"Be excellent to each other"

[Dustin]

On May 15, 12:54 am, TopLess <0w9za2bm29ax...@jetable.com> wrote:
> Hi,
>
> to help us to diagnose this problem, this is our stats :

``stats'' is a memcached protocol command. If you show us the
output of the three stats variations I listed before, we can tell you
what's happening in your server. If you don't, we can only guess.

[TopLess]

we have used the last memcached version (1.2.8) on your linux server
(64bit for information) and now all is ok....very strange, perhaps a
bug with this previously version.