Redis high fragmentation ratio

[har...@snapchat.com]

Hi Folks,

I have been running redis 2.6.16 on our servers since like 3 days. The traffic pattern is most of our keys expire in two days. 

Initially the mem frag ratio is 1.02 but after 3 days, the frag ratio reached 7.02, because of the expired keys after two days.

This is the redis cli info output after three days

# Server

redis_version:2.6.16

redis_git_sha1:00000000

redis_git_dirty:0

redis_mode:standalone

os:Linux 3.3.8-gcg-201305291443 x86_64

arch_bits:64

multiplexing_api:epoll

gcc_version:4.7.2

process_id:17085

run_id:81f5b5ca19e6e99c0d0f3e2b60192bab00ed772b

tcp_port:6379

uptime_in_seconds:456765

uptime_in_days:5

hz:10

lru_clock:1701442

# Clients

connected_clients:86

client_longest_output_list:0

client_biggest_input_buf:0

blocked_clients:0

# Memory

used_memory:658886608

used_memory_human:628.36M

used_memory_rss:4721012736

used_memory_peak:6895495360

used_memory_peak_human:6.42G

used_memory_lua:34816

mem_fragmentation_ratio:7.17

mem_allocator:jemalloc-3.2.0

# Persistence

loading:0

rdb_changes_since_last_save:164204689

rdb_bgsave_in_progress:0

rdb_last_save_time:1379706464

rdb_last_bgsave_status:ok

rdb_last_bgsave_time_sec:-1

rdb_current_bgsave_time_sec:-1

aof_enabled:1

aof_rewrite_in_progress:0

aof_rewrite_scheduled:0

aof_last_rewrite_time_sec:63

aof_current_rewrite_time_sec:-1

aof_current_rewrite_time_sec:-1

aof_last_bgrewrite_status:ok

aof_current_size:3868700873

aof_base_size:3606817570

aof_pending_rewrite:0

aof_buffer_length:0

aof_rewrite_buffer_length:0

aof_pending_bio_fsync:0

aof_delayed_fsync:0

# Stats

total_connections_received:29001

total_commands_processed:258293499

instantaneous_ops_per_sec:0

rejected_connections:0

expired_keys:0

evicted_keys:0

keyspace_hits:81726729

keyspace_misses:43755703

pubsub_channels:0

pubsub_patterns:0

latest_fork_usec:114570

# Replication

role:slave

master_host:prod-master101-0

master_port:6379

master_link_status:up

master_last_io_seconds_ago:6

master_sync_in_progress:0

slave_priority:100

slave_read_only:1

connected_slaves:0

# CPU

used_cpu_sys:5664.82

used_cpu_user:3601.12

used_cpu_sys_children:33.14

used_cpu_user_children:117.06

# Keyspace

db0:keys=92630,expires=0,avg_ttl=0

But after the server restart, the frag ratio dropped to 1.02

This is the redis-cli info afterwards

# Server

redis_version:2.6.16

redis_git_sha1:00000000

redis_git_dirty:0

redis_mode:standalone

os:Linux 3.3.8-gcg-201305291443 x86_64

arch_bits:64

multiplexing_api:epoll

gcc_version:4.7.2

process_id:22209

run_id:c7b52fa584c6a2b742d1b1b40373bb4b9591edd8

tcp_port:6379

uptime_in_seconds:2040

uptime_in_days:0

hz:10

lru_clock:1701442

# Clients

connected_clients:2

client_longest_output_list:0

client_biggest_input_buf:0

blocked_clients:0

# Memory

used_memory:759104656

used_memory_human:723.94M

used_memory_rss:775901184

used_memory_peak:8046033248

used_memory_peak_human:7.49G

used_memory_lua:32768

mem_fragmentation_ratio:1.02

mem_allocator:jemalloc-3.2.0

# Persistence

loading:0

rdb_changes_since_last_save:110356534

rdb_bgsave_in_progress:0

rdb_last_save_time:1380161183

rdb_last_bgsave_status:ok

rdb_last_bgsave_time_sec:-1

rdb_current_bgsave_time_sec:-1

aof_enabled:1

aof_rewrite_in_progress:0

aof_rewrite_scheduled:0

aof_last_rewrite_time_sec:4

aof_current_rewrite_time_sec:-1

aof_last_bgrewrite_status:ok

aof_current_size:363839461

aof_base_size:363839461

aof_pending_rewrite:0

aof_buffer_length:0

aof_rewrite_buffer_length:0

aof_pending_bio_fsync:0

aof_delayed_fsync:0

# Stats

total_connections_received:110

total_commands_processed:360110

instantaneous_ops_per_sec:0

rejected_connections:0

expired_keys:0

evicted_keys:0

keyspace_hits:1697712

keyspace_misses:181

pubsub_channels:0

pubsub_patterns:0

latest_fork_usec:20292

# Replication

role:slave

master_host:prod-master100-0

master_port:6379

master_link_status:up

master_last_io_seconds_ago:2

master_sync_in_progress:0

slave_priority:100

slave_read_only:1

connected_slaves:0

# CPU

used_cpu_sys:8.51

used_cpu_user:72.65

used_cpu_sys_children:0.43

used_cpu_user_children:1.29

# Keyspace

db0:keys=107261,expires=0,avg_ttl=0

I understand that the memory consumed by rss is not released to the OS by redis, but will be reused for additional data. But I am wondering if there are any disadvantages with high frag ratio ? 

Is this something common where ppl restart redis servers once in a while? 

Thanks in advance for your help!

Thanks,
Harish

[Salvatore Sanfilippo]

Hello, see these fields:

used_memory_human:628.36M
used_memory_rss:4721012736
used_memory_peak:6895495360
used_memory_peak_human:6.42G

You have a peak memory usage of 6.4 GB but you are currently using 628
MB, this is why you see a big fragmentation, as the RSS will stay more
near to the peak.

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Salvatore 'antirez' Sanfilippo
open source developer - GoPivotal
http://invece.org

We suspect that trading off implementation flexibility for
understandability makes sense for most system designs.
— Diego Ongaro and John Ousterhout (from Raft paper)

[har...@snapchat.com]

Thanks Antirez!

I understand the cause of higher fragmentation ratio as our peak usage is high. I am trying to understand if there are any side effects of letting the redis servers run with high fragmentation ratio's ?

Is it a common practice for people to restart redis servers once in a while to overcome the high fragmentation ratio?

Thanks,
Harish

[Josiah Carlson]

The only reason that you'd experience any sort of harm is if you have other processes that need to use that memory. Then parts of the memory that Redis isn't interested in (but which haven't been released to the OS) can be paged out. If Redis ever tried to use that memory again, it would stall to wait for the memory to be paged back in by the OS.

As long as you don't have other memory pressure on the machine, you should be fine.

 - Josiah

[Harish Doddi]

Thanks Josiah!

Harish

--
You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/XuiwF3SNMpI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.

To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

--

Harish Doddi,

Snapchat Inc,

Venice, CA - 90291