auto-reconnect

[Brian Hammond]

One or more commands are serialized and sent over the wire to Redis.

The client waits for a reply (async).

Redis is killed (SIGTERM).

Redis is restarted.

Some indeterminate number of commands from the original batch were
processed by Redis but without a reply sent by Redis or received by
the client.

What's a client to do?

http://gist.github.com/372038

http://github.com/fictorial/redis-node-client/blob/master/test/test_shutdown_reconnect.js



--
Subscription settings: http://groups.google.com/group/redis-clients-dev/subscribe?hl=it

[Salvatore Sanfilippo]

On Tue, Apr 20, 2010 at 7:26 AM, Brian Hammond <br...@fictorial.com> wrote:
> One or more commands are serialized and sent over the wire to Redis.
>
> The client waits for a reply (async).
>
> Redis is killed (SIGTERM).
>
> Redis is restarted.
>
> Some indeterminate number of commands from the original batch were
> processed by Redis but without a reply sent by Redis or received by
> the client.
>
> What's a client to do?

Hello Brian,

in an evented implementation like redis node client reconnecting is
going to be very hard...
Users that absolutely need to protect about this kind of stuff should
use MULTI/EXEC.
If the client is blocking it's very different (at least if the
pipeline is not used) because it's a request/reply protocol so no more
than a single command can fail. Resending the command is probably not
a good idea if the command is a write command, while it makes sense
for read-only commands. For write commands I would return an I/O error
and reconnect, so that the next command will work again.

In the evented implementation I would return an error for every
command that can't be processed, to the registered handler.
Maybe a way to mitigate this is not sending commands asap to the
socket if there are many pending requests we didn't still read, so
that on reconnection all the commands not yet queued can be issued
(and not re-issued). The trick of giving the client information about
what command is a write operation and what a read-only op can work
even in this scenario of the event driven programming, reissuing the
read only commands. But this fixes just part of the problem...

Cheers,
Salvatore

> http://gist.github.com/372038
>
> http://github.com/fictorial/redis-node-client/blob/master/test/test_shutdown_reconnect.js
>
>
>
> --
> Subscription settings: http://groups.google.com/group/redis-clients-dev/subscribe?hl=it
>

--
Salvatore 'antirez' Sanfilippo
http://invece.org

"Once you have something that grows faster than education grows,
you’re always going to get a pop culture.", Alan Kay

[Brian Hammond]

Yes, reconnecting is difficult actually if I wanted to guarantee that
commands submitted will be handled. We cannot know this.

Re: MULTI/EXEC. How is this different from my issue as per the gist
above? That is, Redis handled all the commands including the EXEC but
the client never received the reply because Redis was killed by some
16 year old system admin. Moreover, with the various durability
settings, I cannot guarantee to the client users that the command will
be called back with a reply since there are of course ways in which a
reply will not happen.

I agree with you. The best I can do is at least detect this situation
and let the client user figure out what they want to do (retry? give
up?).

Here's the language of some guidelines/docs for redis-node-client that
I will work off of. Thoughts?


Reconnections to Redis should be transparent to the user of redis-node-
client.

The user can just call command methods (e.g. lpush) even when not
connected to Redis. The commands will be submitted to Redis when a
connection to Redis is established. For commands that were submitted
and the connection to Redis lost before receiving a reply, the reply
will never come, and we have no knowledge of which submitted commands
were processed by Redis. In this case, submitted commands will have
their callbacks called with an err of an Error object whose .message
is "IO Error" and whose .originalCommand is an Array containing the
original command at index 0 and arguments thereafter. It is up to the
caller to attempt to retry the call but the original callback is
removed as the reply will never come for that submitted command.

The user should be notified when a connection cannot be established
(or a reconnection reestablished). If a connection to Redis fails and
cannot be reestablished, the client emits noconnection. No
reconnections will be attempted if the first connection attempt fails.

The user should be notified when an attempt is made to reconnect to
Redis after the initial or subsequent established connection is lost.
The client emits reconnecting upon a reconnection attempt, and
reconnected if the connection was reestablished.

The user should be notified when a connection is established. The
client emits a connected event when first connected, and reconnected
when a connection is later reestablished automatically.


Thanks,
Brian

> >http://github.com/fictorial/redis-node-client/blob/master/test/test_s...

>
> > --
> > Subscription settings:http://groups.google.com/group/redis-clients-dev/subscribe?hl=it
>
> --

> Salvatore 'antirez' Sanfilippohttp://invece.org

[Salvatore Sanfilippo]

On Tue, Apr 20, 2010 at 4:14 PM, Brian Hammond <br...@fictorial.com> wrote:
> Yes, reconnecting is difficult actually if I wanted to guarantee that
> commands submitted will be handled.  We cannot know this.

I wonder if actually there is some trick in order to know about it...

> Re: MULTI/EXEC.  How is this different from my issue as per the gist
> above?  That is, Redis handled all the commands including the EXEC but
> the client never received the reply because Redis was killed by some
> 16 year old system admin.  Moreover, with the various durability
> settings, I cannot guarantee to the client users that the command will
> be called back with a reply since there are of course ways in which a
> reply will not happen.

Ok but for instance, that's how it may work:

the client generates a random number, large enough to never collide.
Create a key name "verify:<large random>"

MULTI
... do your work ...
SET verify:<lrage random> YES!
EXPIRE verify:<lrage random> 300
EXEC

If something goes bad and we have to reconnect we can still be able to
check if our operation succeeded or not.

In other words, applications that need to do this can find ways to
work around the limitations of a socket oriented stuff without
"handles" for operations.

> I agree with you. The best I can do is at least detect this situation
> and let the client user figure out what they want to do (retry? give
> up?).

I think the client should just report the read error indeed, it will
be up to the application to mount a system like the above to verify if
the operation needs to be reissued...

> Here's the language of some guidelines/docs for redis-node-client that
> I will work off of.  Thoughts?
>
>
> Reconnections to Redis should be transparent to the user of redis-node-
> client.

Reconnection not in the middle of a command indeed can be that cool.
That is, you try to issue a command but there is no link. The client
reconnects and issue the command.

> The user can just call command methods (e.g. lpush) even when not
> connected to Redis. The commands will be submitted to Redis when a
> connection to Redis is established. For commands that were submitted
> and the connection to Redis lost before receiving a reply, the reply
> will never come, and we have no knowledge of which submitted commands
> were processed by Redis. In this case, submitted commands will have
> their callbacks called with an err of an Error object whose .message
> is "IO Error" and whose .originalCommand is an Array containing the
> original command at index 0 and arguments thereafter. It is up to the
> caller to attempt to retry the call but the original callback is
> removed as the reply will never come for that submitted command.

I think this is perfectly sane...

> The user should be notified when a connection cannot be established
> (or a reconnection reestablished). If a connection to Redis fails and
> cannot be reestablished, the client emits noconnection. No
> reconnections will be attempted if the first connection attempt fails.

Again sounds ok to me. But if the client calls a new command again,
the reconnection should already be attempted IMHO.

> The user should be notified when an attempt is made to reconnect to
> Redis after the initial or subsequent established connection is lost.
> The client emits reconnecting upon a reconnection attempt, and
> reconnected if the connection was reestablished.

Not sure how you would handle this at API level.
Calling a method given that your client is event-driven?

> The user should be notified when a connection is established. The
> client emits a connected event when first connected, and reconnected
> when a connection is later reestablished automatically.

I guess it's the same here.

Thanks for sharing!
Salvatore

--
Salvatore 'antirez' Sanfilippo

[Brian Hammond]

I understand what you mean now. I was hoping for a generic solution
wherein the client user (app) didn't need to do anything special. I
think that's a pipe dream now.

Thanks. OK.