Issue 199 in redis: Redis loses data when data is pushed after blocking (BRPOP/BLPOP) client terminates

[re...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 199 by bartosz.blimke: Redis loses data when data is pushed after
blocking (BRPOP/BLPOP) client terminates
http://code.google.com/p/redis/issues/detail?id=199

Description:

When I create a new Redis connection and I send `BRPOP` command I
terminate the client. Redis server is never told that the client
disconnected.
Redis still thinks the connection is active and the client is waiting for
data
from list.
When I connect again and I push data to this list, Redis gives this data to
disconnected client, and this data is not available anymore. It's lost.

What steps will reproduce the problem?

We first connect to Redis, we send BRPOP command and we disconnect
after that.

telnet localhost 6379
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
BRPOP foobar_list 0
^]
telnet> quit
Connection closed.

Then we connect again to Redis and we can see that Redis still sees old
connection:

telnet localhost 6379
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
...
connected_clients:2
...
blocked_clients:1
...

Now lets push data to a list

LPUSH foobar_list 1
1
:1

And when we try to get this data

lrange foobar_list 0 -1
*0

it's not there. It's actually lost.

What is the expected output?

Data should not be lost!

What do you see instead?
Pushed data is not available anymore.

What version of the product are you using?
Redis 1.3.6 from trunk

On what operating system?
Snow Leopard

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings

[re...@googlecode.com]

Comment #1 on issue 199 by bartosz.blimke: Redis loses data when data is

pushed after blocking (BRPOP/BLPOP) client terminates
http://code.google.com/p/redis/issues/detail?id=199

Is there a chance this will be fixed any time soon or is there a version of
Redis I could
use, without this issue?

[re...@googlecode.com]

Comment #2 on issue 199 by nytram: Redis loses data when data is pushed

after blocking (BRPOP/BLPOP) client terminates
http://code.google.com/p/redis/issues/detail?id=199

The issue here seems to be that data is sent to a blocking client with no
acknowledgement, which may well be by
design. It would be great to hear from someone familiar with the internals
on whether this is really considered to a
problem or whether it is by design.

Unfortunately for a user implementing a system where data cannot be lost
one it has entered a queue, the
blocking operations in their current state cannot be used. A blocking
RPOPLPUSH would allow implementing a
reserved state which could be cleaned up - is that command planned. For now
is polling the only safe option, or is
there another workaround?

Thanks!

[Demis Bellot]

I'm encountering similar issues when implementing a Redis-based queue. 

To get around the 'lost message' issues we should be using 'server-side message traversal'. I've identified a couple of atomic commands that would be useful to have for Redis-Queing here:

http://groups.google.com/group/redis-db/browse_thread/thread/a8165717ea2c1f60

ZPOPLPUSH - 'POP highest priority message from Priority Queue (Sorted Set by Priority) and PUSH into worker queue (List)' 
LREMLPUSH - 'REM from worker queue and PUSH into a) success -> 'out/processed queue' b) failure -> 'dlq' 

MULTI/EXEC (i.e. Redis Transactions) with the ability to provide 'atomic compound operations' provides some solutions in this area.

Unfortunately looking at its design it doesn't look like it supports using the results for other operations within the 'same transaction', which means stuff like 'ZPOPLPUSH' will be hard to implement.

At the moment the approach that I'm taking is to ensure that a message is always in a server list/set before popping it off another one.

1. LPUSH Message => In Queue

2. RPOPLPUSH Message from In Queue => Worker Request Queue

3. Worker thread processes message

4 a) On Success LPUSH message on rolling LTRIM'ed Out Queue than LREM from Worker Request Queue

   b) On Failure:

      i) Increment message retry count, LPUSH back onto => In Queue, LREM from Worker Request Queue

      ii) LPUSH onto request DLQ, LREM from Worker Request Queue

    

In essence, the message always exists in a Redis LIST before it's popped off another one. The problem with this approach of course is that on service failure you risk processing the message multiple times, however I believe this is a lesser evil then losing the message entirely. It means you should ensure your messages are idempotent so it can be processed multiple times without side-effects.

- Demis

http://code.google.com/p/servicestack/wiki/ServiceStackRedis

You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[re...@googlecode.com]

Comment #3 on issue 199 by antirez: Redis loses data when data is pushed

after blocking (BRPOP/BLPOP) client terminates
http://code.google.com/p/redis/issues/detail?id=199

That's very strange as I remember that I fixed the bug a few months ago.
Are you sure
this still is the same on Redis master? Can't replicate.

[re...@googlecode.com]

Comment #4 on issue 199 by bartosz.blimke: Redis loses data when data is

pushed after blocking (BRPOP/BLPOP) client terminates
http://code.google.com/p/redis/issues/detail?id=199

I tried to reproduce it today again using telnet sessions and I'm not able
to. Very
strange.

I'm still getting problems when I do it using redis gem. Here is the spec:

--- CUT HERE----
require 'rubygems'
require 'timeout'
require 'redis'

describe "Redis BRPOP" do
before(:each) do
Redis.new.flushdb
end

it "should block when there is no data available" do
lambda {
Timeout::timeout(1) { Redis.new.brpop("foobar_list", 0) }
}.should raise_error(Timeout::Error)
end

it "should return data when it's available" do
Redis.new.lpush("foobar_list", "my data")
lambda {
Timeout::timeout(1) {
Redis.new.brpop("foobar_list", 0).should == "my data"
}
}.should_not raise_error(Timeout::Error)
end

end
--- CUT HERE----

When I run examples separately, they work fine. When run them together,
second
example fails with TimeoutError raised.

[re...@googlecode.com]

Comment #5 on issue 199 by bartosz.blimke: Redis loses data when data is

pushed after blocking (BRPOP/BLPOP) client terminates
http://code.google.com/p/redis/issues/detail?id=199

Please let me know if you can reproduce the problem using code above. If
not I will try
to find some other example.

[re...@googlecode.com]

Updates:
Status: Verified

Comment #6 on issue 199 by antirez: Redis loses data when data is pushed

after blocking (BRPOP/BLPOP) client terminates
http://code.google.com/p/redis/issues/detail?id=199

This is now ok, closing