redis expriry problem

[c.z]

I have an application where multiple clients that 

a) tries to get from redis;

b) if it's not there, then it get the value from db and set it to expire in 10 seconds;

the clients run the above logic every 11 seconds. There are only two keys that are used, each with less than 200k bytes value.

there are about 50 clients. However, after running the clients for 30 mins, Redis memory usage went way up to exceed the memory limit.

It seems that redis doesn't handle expiry gracefully -- is it because it doesn't do de-fragment aggressively or is there a bug?

thanks.

[Josiah Carlson]

When Redis is using a lot of memory, can you send us the output of INFO?

Regards,
- Josiah

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/mcOTtRjZDwoJ.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.

[c.z]

here is the info output. in this case, we stopped using expiry, just 6 keys, 4 of which gets updated every 10 seconds with new value, the max memory is 256M

redis_version:2.4.17

redis_git_sha1:00000000

redis_git_dirty:0

arch_bits:64

multiplexing_api:epoll

gcc_version:4.1.2

process_id:983

run_id:78751cae18887cf8c34d5e9340f84eb3263cb9b6

uptime_in_seconds:3145

uptime_in_days:0

lru_clock:903948

used_cpu_sys:5.42

used_cpu_user:1.35

used_cpu_sys_children:0.10

used_cpu_user_children:0.35

connected_clients:38

connected_slaves:0

client_longest_output_list:0

client_biggest_input_buf:0

blocked_clients:0

used_memory:250491456

used_memory_human:238.89M

used_memory_rss:140718080

used_memory_peak:284084592

used_memory_peak_human:270.92M

mem_fragmentation_ratio:0.56

mem_allocator:jemalloc-3.0.0

loading:0

aof_enabled:0

changes_since_last_save:84

bgsave_in_progress:0

last_save_time:1351216632

bgrewriteaof_in_progress:0

total_connections_received:120

total_commands_processed:47998

expired_keys:4

evicted_keys:2

keyspace_hits:41708

keyspace_misses:46

pubsub_channels:0

pubsub_patterns:0

latest_fork_usec:2781

vm_enabled:0

role:master

db0:keys=6,expires=0

[Josiah Carlson]

That is odd. If you were to execute a BGSAVE, how large is the resulting .rdb?

- Josiah

> https://groups.google.com/d/msg/redis-db/-/tnABAQuCl60J.

[c.z]

dump size is 4658467. Redis has some serious problems.

[Josiah Carlson]

If you have 6 keys/values and a 4.6M dump, that means that each
key/value pair averages about 760k compressed. Your original claim
that each key/value is less than 200k is incorrect.

Could you do us another favor and call STRLEN on each value, and also
calculate the length of the related key? I suspect that you will find
that your <200kbyte key/value pairs are actually substantially larger
than what you expect, as no one has previously experienced the bug you
seem to be experiencing.

Regards,
- Josiah

> https://groups.google.com/d/msg/redis-db/-/rTeRKjqjZJ8J.

[c.z]

-rw-r--r-- 1 root root 4658580 Oct 26 12:51 dump.rdb

strlen k1 :128894

strlen k2: 379216

k5,k7 are 13 bytes length each

k1 and k2, k5, k7 are updated every 10 seconds, these are what caused the trouble.

k3 and k4s every 10 minutes 

strlen k3: 4172372

strlen k4: 95078