Memory fragmentation with Redis hashes

[Aaron Pfeifer]

Hey folks -

I was hoping I might be able to find some suggestions or thoughts on how to reduce memory fragmentation given our workload on Redis...

Our steady-state memory fragmentation is 1.51.

The data that we write is as follows:

* 300 keys written per second

* Keys are ascii strings of varying length between 12 and 14 characters

* Each key is written using hmset with between 70 and 120 fields

* 95% of the fields are always 16 byte binary strings (byte representations of UDIDs, e.g. "07\xeb\x95\xbf0BQ\xabO\x89mY\xeb\xd9\x96")

* The remaining 5% of the fields are always 87 character ascii strings

* The value for every field is an 8 byte binary string

* Every key is stored as a ziplist (the write strategy is based on http://instagram-engineering.tumblr.com/post/12202313862/storing-hundreds-of-millions-of-simple-key-value)

Some more information about the write load:

* We never delete fields from keys or manually delete keys

* Occasionally we'll make 2 hmset calls on the same key -- more often than not we make a single hmset call per key

* Writes all occur within a Lua script

The read load is as follows:

* We rarely read existing keys -- more often than not we're doing a bunch of hmget calls on non-existent keys

Some information about the cluster:

* Server is an AWS Elasticache instance (r3.large)

* Redis 2.8.24

* Available memory: 15GB

* Set up in a master / slave configuration

Relevant information on Redis configuration (I think the rest of it is pretty standard on Elasticache):

* backups are disabled

* appendonly: no

* appendfsync: no

* hash-max-ziplist-entries: 1024

* maxmemory: 9.5GB

* maxmemory-policy: allkeys-lru

* maxmemory-samples: 500 (I set this really high in order to make sure Redis was as close to an LRU as possible and avoid situations where memory couldn't be freed up because a few old keys were not yet evicted)

Output from info (on the master):

# Server

redis_version:2.8.24

redis_git_sha1:0

redis_git_dirty:0

redis_build_id:0

redis_mode:standalone

os:Amazon ElastiCache

arch_bits:64

multiplexing_api:epoll

gcc_version:0.0.0

process_id:1

run_id:23ddbf31b50f7c767068a541bb257f4f4fc5ee26

tcp_port:6379

uptime_in_seconds:164837

uptime_in_days:1

hz:10

lru_clock:16064304

config_file:/etc/redis.conf

# Clients

connected_clients:16

client_longest_output_list:0

client_biggest_input_buf:0

blocked_clients:0

# Memory

used_memory:10175284344

used_memory_human:9.48G

used_memory_rss:15500750848

used_memory_peak:11644166944

used_memory_peak_human:10.84G

used_memory_lua:88064

mem_fragmentation_ratio:1.52

mem_allocator:jemalloc-3.6.0

# Persistence

loading:0

rdb_changes_since_last_save:125969942

rdb_bgsave_in_progress:0

rdb_last_save_time:1475517259

rdb_last_bgsave_status:ok

rdb_last_bgsave_time_sec:-1

rdb_current_bgsave_time_sec:-1

aof_enabled:0

aof_rewrite_in_progress:0

aof_rewrite_scheduled:0

aof_last_rewrite_time_sec:-1

aof_current_rewrite_time_sec:-1

aof_last_bgrewrite_status:ok

aof_last_write_status:ok

# Stats

total_connections_received:5295

total_commands_processed:377181517

instantaneous_ops_per_sec:2

total_net_input_bytes:56548141889

total_net_output_bytes:192080162732

instantaneous_input_kbps:0.10

instantaneous_output_kbps:0.07

rejected_connections:0

sync_full:3

sync_partial_ok:0

sync_partial_err:0

expired_keys:0

evicted_keys:15578425

keyspace_hits:109016257

keyspace_misses:16482044

pubsub_channels:0

pubsub_patterns:0

latest_fork_usec:0

# Replication

role:master

connected_slaves:1

slave0:ip=...,port=6379,state=online,offset=89741866396,lag=1

master_repl_offset:89741866498

repl_backlog_active:1

repl_backlog_size:1048576

repl_backlog_first_byte_offset:89740817923

repl_backlog_histlen:1048576

# CPU

used_cpu_sys:373.55

used_cpu_user:6510.59

used_cpu_sys_children:0.00

used_cpu_user_children:0.00

# Commandstats

cmdstat_get:calls=63,usec=191,usec_per_call=3.03

cmdstat_set:calls=164534,usec=863853,usec_per_call=5.25

cmdstat_del:calls=457375,usec=637683,usec_per_call=1.39

cmdstat_hmset:calls=125085386,usec=1033968688,usec_per_call=8.27

cmdstat_hmget:calls=125270623,usec=707389654,usec_per_call=5.65

cmdstat_hkeys:calls=10,usec=1454,usec_per_call=145.40

cmdstat_hgetall:calls=1,usec=25,usec_per_call=25.00

cmdstat_select:calls=1,usec=1,usec_per_call=1.00

cmdstat_keys:calls=22,usec=18792252,usec_per_call=854193.25

cmdstat_ping:calls=270292,usec=599703,usec_per_call=2.22

cmdstat_psync:calls=3,usec=787,usec_per_call=262.33

cmdstat_replconf:calls=158933,usec=369350,usec_per_call=2.32

cmdstat_info:calls=40245,usec=2743122,usec_per_call=68.16

cmdstat_slaveof:calls=1,usec=181,usec_per_call=181.00

cmdstat_config:calls=5424,usec=113733,usec_per_call=20.97

cmdstat_client:calls=2720,usec=131661,usec_per_call=48.40

cmdstat_eval:calls=461,usec=33321,usec_per_call=72.28

cmdstat_evalsha:calls=125270162,usec=4469330550,usec_per_call=35.68

cmdstat_script:calls=53,usec=1741,usec_per_call=32.85

# Keyspace

db0:keys=4156552,expires=0,avg_ttl=0

I'm not sure if this workload would have improved memory fragmentation in Redis 3.2 since Elasticache does not yet support that version.

Happy to provide additional information, graphs, stats, etc.  The only thing I can't provide is details on anything that requires SSH access since Elasticache doesn't provide that ability.

Given all that, any thoughts on things we could change to improve fragmentation?  Is there reason to believe that Redis 3.2 would provide significant improvements here?

Thanks for any guidance!

-Aaron

[jose]

the memory fragmentation ratio is calculated as the amount of memory currently in use divided by physical mmeory actually used.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Aaron Pfeifer]

Hi Jose -

Sorry, re-reading my post I can see I may not have been clear with my question.

I know how the memory fragmentation ratio is calculated -- I'm actually looking for how I might be able to update configurations in Redis or change the design of my write workload in order to reduce fragmentation and get closer to 1.0.  A ratio of 1.5 is an indication to me that I'm not making effective use of the memory available on the server.  Without adding more memory to the server (and noting that Redis has a ratio of 1.5 while at it's max memory), I'd like to see if there's anything I can do to improve effectiveness of memory use.  Perhaps the answer is that 1.5 is the expected fragmentation given my workload -- certainly an acceptable answer.

For example, one of the notes in the 3.2 release notes was this: "Redis is now more memory efficient thanks to changes operated by Oran Agra to SDS and the Jemalloc size classes setup" -- so I could imagine we might see an improvement in Redis 3.2.  I'm curious if, beyond finding a way to use a newer version of Redis, whether there's anything else I can do (functional, configuration, etc.) to make improvements.

Thanks,

Aaron

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.