Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Redis DB
Home
+ new post
About this group
Join this group
Message from discussion Regressions in 2.6 when out of memory & writing during multi-exec?
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Salvatore Sanfilippo  
View profile  
 More options Nov 15 2012, 1:27 pm
From: Salvatore Sanfilippo <anti...@gmail.com>
Date: Thu, 15 Nov 2012 19:26:42 +0100
Local: Thurs, Nov 15 2012 1:26 pm
Subject: Re: Regressions in 2.6 when out of memory & writing during multi-exec?
Print | View thread | Show original | Report this message | Find messages by this author
First observation, the reality is that while 2.6 changed behavior and
made it worse, this was broken in Redis 2.4 as well.

1) If you set maxmemory to 1, then to 0 again during a transaction,
only a random number of commands will be committed.
2) If any event during a transaction frees some memory, only some
command can be committed.

Now "broken" is not probably the right term as actually Redis issues
an error when you try to queue the command, but well, it is not a
desirable behavior probably. However to skip the check client side is
still a risk because of syntax errors.

Now, how to fix this to have safety & simple behavior from the point
of view of client library implementations?

IMHO the best thing to do is:

1) If we detect an OOM error, syntax error, or any other error while
queueing a command (another error is that RDB dump is failing, and we
no longer accept writes), we flag the client internally and the EXEC
will always fail when called.
2) In such a case we return a generic error in EXEC, like: "Error
while queueing one or more command. EXEC aborted.".
3) I believe it is ways better if when this happens we automatically
DISCARD the transaction, for the sake of client simplicity.
4) A client should always read, once EXEC was called, the reply from
all the commands, and where QUEUE was expected but was not returned,
an error should be raised together with the EXEC error itself.

Does this makes sense?

Note that the alternative design where we simply always reply QUEUED
to commands and fail on EXEC is not possible as we dont' want to queue
an huge transaction while there is an OOM condition.

Cheers,
Salvatore

On Thu, Nov 15, 2012 at 6:56 PM, Martyn Loughran <mar...@pusher.com> wrote:
> While upgrading from 2.4 to 2.6 I noticed some changes which I haven't seen
> documented, and look like regressions to me.

> Scenario: out of memory & writing during multi-exec

> # Change 1 - exec succeeds even if one of commands was an error

> On redis 2.4 (tested 2.4.17)

>     redis 127.0.0.1:6379> config get maxmemory
>     1) "maxmemory"
>     2) "0"
>     redis 127.0.0.1:6379> multi
>     OK
>     redis 127.0.0.1:6379> set foo yyy
>     QUEUED

> In another client set memory limit `config set maxmemory 1`

>     redis 127.0.0.1:6379> set bar uuu
>     (error) ERR command not allowed when used memory > 'maxmemory'
>     redis 127.0.0.1:6379> exec
>     (error) ERR command not allowed when used memory > 'maxmemory'

>     Transaction still open, need to call discard

> On redis 2.6 (tested 2.6.4)

>     redis 127.0.0.1:6379> config get maxmemory
>     1) "maxmemory"
>     2) "0"
>     redis 127.0.0.1:6379> multi
>     OK
>     redis 127.0.0.1:6379> set foo yyy
>     QUEUED

> In another client set memory limit `config set maxmemory 1`

>     redis 127.0.0.1:6379> set bar uuu
>     (error) OOM command not allowed when used memory > 'maxmemory'.
>     redis 127.0.0.1:6379> exec
>     1) OK
>     redis 127.0.0.1:6379> get foo
>     "yyy"
>     redis 127.0.0.1:6379> get bar
>     (nil)

> This seems like a pretty serious issue. foo and bar were set during a
> transaction, but one one of the keys was written!

> The only way to code around this is to check the response from each of the
> commands during a multi, and to only exec if there were no errors. This is
> problematic for all sorts of reasons. Previously it was easy to see whether
> the transaction succeeded by reading the exec reply.

> The OOM error is also inconsistent with this line in the transaction docs
> "When a Redis connection is in the context of a MULTI request, all commands
> will reply with the string QUEUED unless they are syntactically incorrect".

> # Change 2 - memory limit enforced during queuing rather than exec

> This is related to the first change, but I just wanted to be explicit

> On redis 2.4 (tested 2.4.17) the memory limit is enforced when exec is
> called

>     redis 127.0.0.1:6379> multi
>     OK
>     redis 127.0.0.1:6379> set foo asdf
>     QUEUED

> In another client set memory limit `config set maxmemory 1`, then exec the
> transaction

>     redis 127.0.0.1:6379> exec
>     (error) ERR command not allowed when used memory > 'maxmemory'

> As expected the set fails.

> For redis 2.6 (tested 2.6.4), the set does not fail

>     redis 127.0.0.1:6379> multi
>     OK
>     redis 127.0.0.1:6379> set foo asdf
>     QUEUED

> In another client set memory limit `config set maxmemory 1`, then exec the
> transaction

>     redis 127.0.0.1:6379> exec
>     1) OK

> This feels like a bug to me.

> Thanks very much for all your work on redis guys :)

> Martyn

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To post to this group, send email to redis-db@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
       — David Gelernter


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2013 Google