Redis overhead

[Richard Hesse]

How much overhead is to be expected for each key and connection in redis? I ask because while the configured memory limit does a great job at limiting the data set size, the RSS footprint continues to grow (and possibly leak) without bounds. Is this expected behavior when trying to use redis as a LRU cache?

-richard

[Salvatore Sanfilippo]

RSS growing is not expected behavior, please could you send your INFO
output and the output of CONFIG GET * ?

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Salvatore 'antirez' Sanfilippo
open source developer - GoPivotal
http://invece.org

We suspect that trading off implementation flexibility for
understandability makes sense for most system designs.
— Diego Ongaro and John Ousterhout (from Raft paper)

[Richard Hesse]

Sure thing. Here you go. info:

# Server
redis_version:2.6.16
redis_git_sha1:00000000
redis_git_dirty:0
redis_mode:standalone
os:Linux 2.6.32-358.18.1.el6.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.7
process_id:15608
run_id:4779dfd51b36d1ec874eb7372f2943ecc42e3257
tcp_port:6379
uptime_in_seconds:263665
uptime_in_days:3
hz:10
lru_clock:1922906

# Clients
connected_clients:130
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:136364642752
used_memory_human:127.00G
used_memory_rss:145208041472
used_memory_peak:136366014088
used_memory_peak_human:127.00G
used_memory_lua:31744
mem_fragmentation_ratio:1.06
mem_allocator:jemalloc-3.2.0

# Persistence
loading:0
rdb_changes_since_last_save:10363931
rdb_bgsave_in_progress:0
rdb_last_save_time:1382114196
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok

# Stats
total_connections_received:112908369
total_commands_processed:244043189
instantaneous_ops_per_sec:1257
rejected_connections:0
expired_keys:0
evicted_keys:523595
keyspace_hits:98056974
keyspace_misses:22595439
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0

# Replication
role:master
connected_slaves:0

# CPU
used_cpu_sys:10971.89
used_cpu_user:6274.10
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Keyspace
db0:keys=6489878,expires=0,avg_ttl=0

CONFIG GET *:
CONFIG GET *
*100
$10
dbfilename
$8
dump.rdb
$11
requirepass
$0

$10
masterauth
$0

$4
bind
$0

$10
unixsocket
$29
/foo/data/redis/redis.sock
$7
logfile
$0

$7
pidfile
$18
/var/run/redis.pid
$9
maxmemory
$12
136365211648
$17
maxmemory-samples
$1
3
$7
timeout
$3
600
$13
tcp-keepalive
$1
0
$27
auto-aof-rewrite-percentage
$2
50
$25
auto-aof-rewrite-min-size
$8
67108864
$24
hash-max-ziplist-entries
$3
512
$22
hash-max-ziplist-value
$2
64
$24
list-max-ziplist-entries
$3
512
$22
list-max-ziplist-value
$2
64
$22
set-max-intset-entries
$3
512
$24
zset-max-ziplist-entries
$3
128
$22
zset-max-ziplist-value
$2
64
$14
lua-time-limit
$4
5000
$23
slowlog-log-slower-than
$5
10000
$15
slowlog-max-len
$4
1024
$4
port
$4
6379
$9
databases
$1
1
$22
repl-ping-slave-period
$2
10
$12
repl-timeout
$2
60
$10
maxclients
$5
15000
$15
watchdog-period
$1
0
$14
slave-priority
$3
100
$2
hz
$2
10
$25
no-appendfsync-on-rewrite
$2
no
$22
slave-serve-stale-data
$3
yes
$15
slave-read-only
$3
yes
$27
stop-writes-on-bgsave-error
$3
yes
$9
daemonize
$2
no
$14
rdbcompression
$3
yes
$11
rdbchecksum
$3
yes
$15
activerehashing
$3
yes
$24
repl-disable-tcp-nodelay
$2
no
$29
aof-rewrite-incremental-fsync
$3
yes
$10
appendonly
$2
no
$3
dir
$18
/foo/data/redis
$16
maxmemory-policy
$11
allkeys-lru
$11
appendfsync
$2
no
$4
save
$0

$8
loglevel
$6
notice
$26
client-output-buffer-limit
$67
normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60
$14
unixsocketperm
$3
777
$7
slaveof
$0

TIA!

-richard

[Didier Spezia]

I take my calculator.

According to the info output:

RSS = 145 208 041 472

Let's remove the memory used by Redis.

It is used_memory_peak, multiplied by mem_fragmentation_ratio,

since we need to account also the fragmentation overhead (here: 6%)

RSS - used_memory_peak * mem_fragmentation_ratio

= 660 066 539

Now if you remove memory managed by the kernel itself, like the

page allocation table (at least 256 MB), or the socket buffers

(up to 2*128K per connection if you use pipelining a bit aggressively),

I don't think a huge amount of unexplained memory remains.

Supposing the memory fragmentation is stable, the RSS should not

increase much more than its current level.

Regards,

Didier.

[Richard Hesse]

And from this morning when I posted those stats, RSS has increased by another 2GB.
used_memory_rss:147863035904

It is indeed increasing and will continue to increase until it uses all of the "spare" RAM on the machine (which is 16GB right now).

If fragmention is a problem here, what is the expected fragmentation when running redis as a LRU? 2% 5% 10%?

-richard

[Salvatore Sanfilippo]

Hello again Richard, I can't see nothing wrong in your Redis instance
memory usage:

used_memory_peak_human:127.00G with a fragmentation ratio of 1.06. The
fragmentation ratio can raise a little bit in the future because of
how the allocator works, but it is not unbound, at some point it no
longer increases.
If you are near to the physical memory limit I suggest lowering the
maxmemory to a very safe value, see with your workload what is the
peak fragmentation ratio you get, and then raise it again.

It will be great if you can give us some information about the
workload you are using (INFO commandstats) and what is the
fragmentation ratio that you'll eventually reach when it stops growing
more.

Cheers,
Salvatore

[Salvatore Sanfilippo]

p.s. do you have monitoring systems sampling the memory usage / RSS
and so forth? Would be great to see a graph.
If you don't have anything like that I strongly suggest you to install
similar tools as it is really great to have this kind of metrics, for
example from the shape of the curve you could already guess when it is
going to stabilize at a given fragmentation ratio.

[Richard Hesse]

Oh I have lots of graphs. ;)








Thanks for looking into the issue. I'm guessing that the memory fragmentation needs to stabilize more? I'll try dropping the max data set size and see how that does.

-richard

[Richard Hesse]

My bad, it looks like my inline graphs were not actually posted. Why does Google give us the option to insert graphs but not actually send them?

Anyway, fragmentation is at 1.29 and still climbing.

-richard