Re: Possible redis bug?

[Josiah Carlson]

On Mon, Sep 17, 2012 at 10:40 PM, zs <zsl...@gmail.com> wrote:
> We have used redis for about 6+ month in production. There are a couple of
> issues that I've observed with redis-2.4.4

There are a few known issues with Redis 2.4.4, which is why 2.4.17 exists ;)

> a) is there a possible memory leak? we've seen memory gone from 200M to 10G
> in 6 month, but the data size hasn't changed. We are using mostly hash data
> set, using jedis java library. Restarting the server seems to drop the
> memory back down.

There are a few potential causes of this, all of which are fixed in
releases up to and including Redis 2.4.17 . I would suggest you
upgrade.

> b) keys command and expire command doesn't seem to work with specific keys?
> say i have a key a
> set a 10
> expire a 10
>
> however, key a is expired, a get still returns 10, though documentation says
> that after a key expired, it will be deleted automatically
>
> are these bugs in 2.4.4?

I do not remember such a bug, but my memory may be flawed. Regardless,
you should upgrade to 2.4.17, it should fix your memory bug.

> thx, and btw, is redis cluster going to be released anytime soon?

Sadly, no. Redis 2.6 and sentinel are first, and while cluster has a
lot of work done for it, there is still enough work to be done that
Salvatore isn't even recommending people run it for testing. On the
other hand, Redis 2.6 has been in RC mode for a few months, and
sentinel has been available for several weeks.

Regards,
- Josiah

[M. Edward (Ed) Borasky]

On Tue, Sep 18, 2012 at 6:04 AM, Josiah Carlson
<josiah....@gmail.com> wrote:

[snip]

>> thx, and btw, is redis cluster going to be released anytime soon?
>
> Sadly, no. Redis 2.6 and sentinel are first, and while cluster has a
> lot of work done for it, there is still enough work to be done that
> Salvatore isn't even recommending people run it for testing. On the
> other hand, Redis 2.6 has been in RC mode for a few months, and
> sentinel has been available for several weeks.
>
> Regards,
> - Josiah

Is Sentinel in the 2.6.0 RC7 tarball, or do I need to go to Github for it?
--
Twitter: http://twitter.com/znmeb; Computational Journalism Publishers
Workbench: http://j.mp/QCsXOr

How the Hell can the lion sleep with all those people singing "A weem
oh way!" at the top of their lungs?

[Yifeng Jiang]

Hi,

Sentinel is only available in the unstable branch on github.

See: http://redis.io/topics/sentinel

- Yifeng

[zs]

Using 2.4.17 at least solved the expiration problem.

However, I've observed another issue with 2.4.17.

I have maxmemory set to 256M. However, I've seen the log showed that redis used the memory
proportional to the number of clients (each might be writing updating the same key).  There are only
3 keys used for this cache, total memory for these should be around 75M. And occationally, i get ERR memory
exceeding maxmemory, but not always.

So my question is:
what's maxmemory? how is it observed? and how can total memory used be bigger than maxmemory?



[14749] 21 Sep 11:34:31 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:34:31 - 17 clients connected (0 slaves), 554518168 bytes in use
[14749] 21 Sep 11:34:36 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:34:36 - 17 clients connected (0 slaves), 638404152 bytes in use
[14749] 21 Sep 11:34:41 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:34:41 - 17 clients connected (0 slaves), 667764216 bytes in use
[14749] 21 Sep 11:34:46 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:34:46 - 17 clients connected (0 slaves), 667764176 bytes in use
[14749] 21 Sep 11:34:51 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:34:51 - 17 clients connected (0 slaves), 667764144 bytes in use
[14749] 21 Sep 11:34:56 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:34:56 - 17 clients connected (0 slaves), 667764144 bytes in use
[14749] 21 Sep 11:35:01 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:35:01 - 17 clients connected (0 slaves), 667764144 bytes in use
[14749] 21 Sep 11:35:06 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:35:06 - 16 clients connected (0 slaves), 630006888 bytes in use
[14749] 21 Sep 11:35:11 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:35:11 - 13 clients connected (0 slaves), 516735128 bytes in use
[14749] 21 Sep 11:35:16 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:35:16 - 12 clients connected (0 slaves), 478977856 bytes in use
[14749] 21 Sep 11:35:21 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:35:21 - 10 clients connected (0 slaves), 403463328 bytes in use
[14749] 21 Sep 11:35:26 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:35:26 - 9 clients connected (0 slaves), 365706072 bytes in use
[14749] 21 Sep 11:35:31 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:35:31 - 9 clients connected (0 slaves), 365706064 bytes in use
[14749] 21 Sep 11:35:36 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:35:36 - 8 clients connected (0 slaves), 327948848 bytes in use
[14749] 21 Sep 11:35:41 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:35:41 - 8 clients connected (0 slaves), 327948848 bytes in use
[14749] 21 Sep 11:35:46 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:35:46 - 8 clients connected (0 slaves), 327948840 bytes in use
[14749] 21 Sep 11:35:51 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:35:51 - 7 clients connected (0 slaves), 290191560 bytes in use
[14749] 21 Sep 11:35:56 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:35:56 - 6 clients connected (0 slaves), 252434280 bytes in use
[14749] 21 Sep 11:36:01 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:36:01 - 6 clients connected (0 slaves), 252434304 bytes in use
[14749] 21 Sep 11:36:06 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[14749] 21 Sep 11:36:06 - 6 clients connected (0 slaves), 252434304 bytes in use
[14749] 21 Sep 11:36:11 - DB 0: 2 keys (0 volatile) in 4 slots HT.

[Josiah Carlson]

If your clients are using that much memory, then they likely either
have very large outgoing buffers associated with them, or had very
large outgoing buffers associated with them in the past that were not
released.

Can you check INFO and CLIENT LIST output? That may tell us if it is
related to outgoing buffers, or previous outgoing buffers.

- Josiah

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/RoyKzS4vFkoJ.
>
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.

[zs]

here is the output of info:

redis_version:2.4.17
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.1.2
process_id:14749
run_id:4ae6f29c8b9a38b15b77c9998cb93090efc2b53b
uptime_in_seconds:172217
uptime_in_days:1
lru_clock:606641
used_cpu_sys:201.53
used_cpu_user:121.91
used_cpu_sys_children:2.73
used_cpu_user_children:14.85
connected_clients:1
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:63648160
used_memory_human:60.70M
used_memory_rss:62668800
used_memory_peak:743278872
used_memory_peak_human:708.85M
mem_fragmentation_ratio:0.98
mem_allocator:jemalloc-3.0.0
loading:0
aof_enabled:0
changes_since_last_save:5
bgsave_in_progress:0
last_save_time:1348243230
bgrewriteaof_in_progress:0
total_connections_received:2918
total_commands_processed:12261
expired_keys:49
evicted_keys:141
keyspace_hits:2771
keyspace_misses:1189
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:7307
vm_enabled:0
role:master
db0:keys=3,expires=1

CLIENT LIST show a bunch of
fd=8 idle=9 flags=N db=0 sub=0 psub=0 qbuf=0 obl=0 oll=0 events=r cmd=get

not sure how to read them

[Josiah Carlson]

I'm sorry I wasn't more explicit.

The information is really only useful when your server is having the
issue. From the INFO output, you only seem to have 1 client... yet you
have many lines from CLIENT LIST? That seems strange...

But specifically, CLIENT LIST shows lines with "oll=" information,
that is the length of the output buffer list in items. Redis 2.6 gives
you the actual bytes used, but knowing the number of items in the
output buffer is a good start for 2.4 .

Regards,
- Josiah

> https://groups.google.com/d/msg/redis-db/-/KsW2N6Dt96kJ.

[c.z]

I did a bunch of CLIENT LIST calls.

I see up to 30 clients, but all most all of them have 0 (the same output i pasted).

I did on two instances where I see qbuff = 6047336 and qbuf = 1779140. 

My two keys are of size 25M and 35M, and those are the only two keys I use. Clients 

either get or set on these keys.

The info command showed that we have used up to 500M, how could that happen?

redis_version:2.4.17redis_git_sha1:00000000redis_git_dirty:0arch_bits:64multiplexing_api:epoll

gcc_version:4.1.2process_id:14749run_id:4ae6f29c8b9a38b15b77c9998cb93090efc2b53buptime_in_seconds:182677uptime_in_days:2lru_clock:607687used_cpu_sys:241.47used_cpu_user:145.08used_cpu_sys_children:3.29used_cpu_user_children:17.65connected_clients:16connected_slaves:0client_longest_output_list:0client_biggest_input_buf:20015164blocked_clients:0used_memory:520955400used_memory_human:496.82Mused_memory_rss:494592000used_memory_peak:743278872used_memory_peak_human:708.85Mmem_fragmentation_ratio:0.95mem_allocator:jemalloc-3.0.0loading:0aof_enabled:0changes_since_last_save:9bgsave_in_progress:0last_save_time:1348253731bgrewriteaof_in_progress:0total_connections_received:3437

total_commands_processed:14899expired_keys:57evicted_keys:162keyspace_hits:3477keyspace_misses:1400pubsub_channels:0

pubsub_patterns:0

latest_fork_usec:8873

vm_enabled:0

role:master

db0:keys=2,expires=0

[Josiah Carlson]

Wait...

You have 2 keys in your database. One is 25 megabytes, the other is 35
megabytes. And all you are doing is reading and writing those two
values?

Your memory use is caused by one of three things:
1. incoming buffers
2. outgoing buffers
3. memory fragmentation

Aside from changing your access patterns (breaking your data up into
smaller pieces that *won't* lead to fragmentation), forcing Redis to
pre-allocate fixed-size blocks for big allocations like this to
prevent fragmentation when you SET/GET values, or a few other things,
I am out of "simple" solutions.

More specifically, you have access patterns that the current memory
allocator (and Redis) was not designed to handle. I'm not surprised
that you're having a bad time.

If you can explain the problem you are attempting to solve, we might
be able to offer a better solution that doesn't involve these
operations, that would give you better memory behavior.

Regards,
- Josiah

> https://groups.google.com/d/msg/redis-db/-/QRNJCqdUPJ8J.

[c.z]

The use case is to cache a couple of big data structure. 

I can use compression to reduce the size of the data, but is that the real problem? redis documents says that value can be 512M. 

The other thing is that one of the key (with 35M-length value) has a expiry. I know expiry didn't quite work in 2.4.4, could there be a problem with 2.4.17 as well? (i.e., key expired, but memory didn't really get de-allocated?).

How do you avoid memory fragmentation? (mem_fragmentation_ratio:0.98, does it mean that it's 98percent fragmented?)

[Josiah Carlson]

On Sun, Sep 23, 2012 at 6:23 PM, c.z <cz...@rocketfuelinc.com> wrote:
> The use case is to cache a couple of big data structure.

Does the entire structure change every time it is written? Only a
small part? Can you encode the changes as a series of deltas, which
are rolled up occasionally?

> I can use compression to reduce the size of the data, but is that the real
> problem? redis documents says that value can be 512M.

*Can be* is not the same as *should be*.

Redis is limited by what the underlying memory allocator is able to
do. Redis currently uses Jemalloc, which does a few things to
reduce/minimize memory fragmentation, and which has solved the issue
for the majority of use-cases. But your particular use-case seems like
it is stressing the memory allocator in ways that it wasn't intended
(a few large allocations that are repeated and freed).

> The other thing is that one of the key (with 35M-length value) has a expiry.
> I know expiry didn't quite work in 2.4.4, could there be a problem with
> 2.4.17 as well? (i.e., key expired, but memory didn't really get
> de-allocated?).

When data "expires" in Redis, it is not an active expiration. That is,
if a key passes expiration, that doesn't mean it is immediately
deallocated. Typically, Redis will make a handful of random probes to
clean out a few keys if it finds any that should be expired. If too
many are found, it performs a scan to clean out as many expired keys
as possible.

Looking at your most recent INFO output:
db0:keys=2,expires=0

You don't have any keys that should be expired.

That said, if you only have a few keys, they should be expired almost
as though they were actively expired (unless I'm missing something).

> How do you avoid memory fragmentation? (mem_fragmentation_ratio:0.98, does
> it mean that it's 98percent fragmented?)

No. 1.0 means that Redis is using exactly as much memory as is
expected. < 1.0 means that it is using less memory than is expected. >
1.0 means it is using more. If you are seeing < 1.0, that is usually
a sign of a bug somewhere. Typical fragmentation ratios with the
current jemalloc allocator are usually in the 1.0 to 1.15 range, which
means up to 15% memory is used that is not accounted for in other
parts.

Also, "memory fragmentation ratio" is an attempt to measure an issue
that occurs in all processes that run on computers, specifically a
combination of "internal fragmentation" and "external fragmentation"
as described: http://en.wikipedia.org/wiki/Fragmentation_(computing) .
Though the ratio that is calculated by Redis is different than the
description on the wikipedia page.

Regards,
- Josiah

> https://groups.google.com/d/msg/redis-db/-/vCj_uCgToagJ.

[c.z]

>That said, if you only have a few keys, they should be expired almost 
> as though they were actively expired (unless I'm missing something).

by 'actively expired', do you mean that the client should delete the keys itself?

expired_keys:103

evicted_keys:1040

keyspace_hits:10368

keyspace_misses:33897

showed that keys did get expired. But based on your explanation, it might be possible that key-value

stays in memory for a much longer period, as they don't get removed right away...

[Josiah Carlson]

On Mon, Sep 24, 2012 at 10:11 AM, c.z <cz...@rocketfuelinc.com> wrote:
>
>>That said, if you only have a few keys, they should be expired almost
>> as though they were actively expired (unless I'm missing something).
>
> by 'actively expired', do you mean that the client should delete the keys
> itself?

Some systems with expiration keep a priority queue of items to be
expired. Redis doesn't. Redis will randomly check keys to be expired,
and expire them.

> expired_keys:103
> evicted_keys:1040
> keyspace_hits:10368
> keyspace_misses:33897
>
> showed that keys did get expired. But based on your explanation, it might be
> possible that key-value
> stays in memory for a much longer period, as they don't get removed right
> away...

That is not the case. It is either fragmentation, or it might be kept
around to finish the transfer to clients that requested it before it
had expired (especially for slow clients).

- Josiah

[Dcshi]

发自我的 iPad

To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/QRNJCqdUPJ8J.