Tuning memcached on ubuntu x86_64 (2.6.35)

[David Mitchell]

Is it normal to have a 16 percent virtual memory overhead in memcached
on x86_64 linux? memcached STAT bytes is reporting 3219 megabytes of
data, but virtual memory is 16 percent higher at 3834. Resident memory
is 14 percent higher at 3763 megabytes.

Is there a way to tune linux/memcached to get memcached to consume
less virtual memory?

At the moment, my 4GB system is full with 3219 megabtyes of data
loaded in memcached. I am seeing lots of evictions when I try to load
more data.

Below is my configuration and stats.

Ubuntu SMP x86_64 2.6.35-22-server
memcached version 1.4.5-1ubuntu1
libevent-1.4-2 version 1.4.13-stable-1

I am using the default settings for memcached, namely, chunk_size of
48 and 4 threads.

Stack size on my linux system is set to 8192 kilobytes. Should I
reduce the stack size?

I am using the default slab page size of 1MB. Should I reduce this
amount?

system: using free
3954 megabytes total memory
3928 megabytes used
26 megabytes free

memcached process: using ps
3763 megabytes RSS
3834 megabytes VSZ
3219 megabtyes reported by STAT bytes

settings:
STAT maxbytes 3879731200
STAT maxconns 1024
STAT tcpport 11211
STAT udpport 11211
STAT inter NULL
STAT verbosity 0
STAT oldest 0
STAT evictions on
STAT domain_socket NULL
STAT umask 700
STAT growth_factor 1.25
STAT chunk_size 48
STAT num_threads 4
STAT stat_key_prefix :
STAT detail_enabled no
STAT reqs_per_event 20
STAT cas_enabled yes
STAT tcp_backlog 1024
STAT binding_protocol auto-negotiate
STAT auth_enabled_sasl no
STAT item_size_max 1048576

STAT pid 716
STAT uptime 10271
STAT time 1311199316
STAT version 1.4.5
STAT pointer_size 64
STAT rusage_user 175.140000
STAT rusage_system 325.200000
STAT curr_connections 11
STAT total_connections 73
STAT connection_structures 30
STAT cmd_get 9137902
STAT cmd_set 9067050
STAT cmd_flush 0
STAT get_hits 337524
STAT get_misses 8800378
STAT delete_misses 0
STAT delete_hits 266674
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 3468460589
STAT bytes_written 244250132
STAT limit_maxbytes 3879731200
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 0
STAT bytes 3376314121
STAT curr_items 8080800
STAT total_items 9067050
STAT evictions 719576
STAT reclaimed 0

[dormando]

> Is it normal to have a 16 percent virtual memory overhead in memcached
> on x86_64 linux? memcached STAT bytes is reporting 3219 megabytes of
> data, but virtual memory is 16 percent higher at 3834. Resident memory
> is 14 percent higher at 3763 megabytes.
>
> Is there a way to tune linux/memcached to get memcached to consume
> less virtual memory?
>

Are you using some bizarre VM system where virtual memory actually
matters? I can start up apps with terabytes of VM "allocated" just fine.

The overhead in RSS is normal. you lose some memory to buffers, pointers,
the hash table structure, etc.

[Trond Norbye]

Are you running out of virtual memory ;)

Please note that you may also tune the slab classes if your object size doesn't match the default slabclasses causing a poor memory utilization...

Trond

[David Mitchell]

On Jul 21, 1:41 pm, Trond Norbye <trond.nor...@gmail.com> wrote:
> Are you running out of virtual memory ;)
>

Well, the problem is that memcached will use swap, when it runs out of
resident memory. When swap space fills up, memcached will crash under
load.

Yesterday, I had the -m option set to 3700 (or 3700 megabtyes), since
I have a 4GB system. But, I started getting evictions when the
dataset reached the size of 3219.9 megabytes. As I mentioned above, I
started getting evictions at 3763 megabytes of RSS and 3834 megabytes
of VSZ. Is the -m option the size of the dataset or is it the size of
resident memory?

Today, I increased the -m option to 8000 (or 8000 megabytes) to see
what would happen. I only have 3954 megabytes total memory in the
system. Now, memcached is filling up the swap space. I assume that I
will start getting evictions when the virtual memory is full.

It seems to me that I should avoid touching the swap space, since
memcached can become unstable when using swap space. But, last week,
I got into trouble because I set the -m option close to the total
available memory on the system, and I guess that I had the value set
too high, since the swap space filled up and memcached crashed.
Today, I am trying to duplicate the issue that I saw last week.

David

[dormando]

-m is the limit of the internal slabber. your stored data will be some
amount smaller than that due to overhead. As is stated in a lot of places,
there're a few hundred megs of extra things going on, -m isn't a global
limit.

You seem to have confused virtual memory and resident initially; the
virtual memory doesn't matter at all. just keep reducing -m until your
system is happy.