Are the replies ordered when using async APIs?

[Sam Fang]

Hi Community,

I would like to ask a question about the use of asynchronous APIs,

for example the async APIs provided by hiredis.

In the following example, if the async APIs are called in the given order, is it guaranteed that:

1. the commands are sent in the order: "GET key1", "GET key2", "GET key3" to Redis server?

2. the replies are processed by the client with correct association with the callback funcs, 

    i.e. "reply for GET key1" must be processed by UserCallbackFn1,  

    "reply for GET key2" must be processed by UserCallbackFn2, etc.

How does the client co-relate a reply with its original request command correctly

and hence the corresponding callback func? What if the replies arrives in different order 

than the order that the commands and callback functions were inserted into redisCallbackList?

...
redisAsyncCommand(c, UserCallbackFn1, NULL, "GET key1");
redisAsyncCommand(c, UserCallbackFn2, NULL, "GET key2");
redisAsyncCommand(c, UserCallbackFn3, NULL, "GET key3");
...

// what if replies came in the order
//       <-- reply 3 to GET key3
//       <-- reply 1 to GET key1
//       <-- reply 2 to GET key2

Any help on this will be very much appreciated!

Thanks,

Sam

[Jan-Erik Rediger]

Hey,

Yes, it is guaranteed that commands sent in the right order and it is
also guaranteed that the replies are processed in the correct order.

With the async API calling `redisAsyncCommand` will append the command
to an internal buffer and will schedule a write operation.
It will also save the associated callback in a internal list.

At the same time the async API also schedules read operations, so
whenever one command returns, the first callback is popped from the list
and called with the received reply.

As Redis itself is single-threaded it will process the commands in the
order they are received.

Does this clear up everything or is something still left unclear?

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.

[Sam Fang]

Hi Janerik,

Many thanks for the answer. I realized that it is TCP that guarantees

the correct odering of the commands and replies.

I guess redis does not embed any sort of "operation id" into the command or

reply packet to specificaly identify a command-reply pair, does it?

Thanks,

Sam

[Josiah Carlson]

No, Redis does not use any sort of "operation id". On an individual client connection, Redis guarantees sequential execution. As such, async Redis clients typically use a FIFO queue or equivalent for callbacks.

 - Josiah

[Jan-Erik Rediger]

Thanks, Josiah. Again my reply was sent directly to the OP, instead of
to the List. I should definitely learn how to use my mail client
correctly. :D

[Josiah Carlson]

It happens :)

 - Josiah

[Sam Fang]

Hi,

Does this hold true for Redis Cluster as well?

Thanks,

Sam

On Thursday, April 2, 2015 at 5:13:47 PM UTC+8, janerik wrote:

[Itamar Haber]

Yes - Redis Cluster is made up of individual Redis shards that behave like standalone Redis in that sense.

--

Itamar Haber | Chief Developers Advocate
Redis Watch Newsletter - Curator and Janitor
Redis Labs - Enterprise-Class Redis for Developers

Mobile: +1 (415) 688 2443
Mobile (IL): +972 (54) 567 9692
Email: ita...@redislabs.com
Skype: itamar.haber

Blog  |  Twitter  |  LinkedIn

[Pavel Shaydo]

On Wed, 8 Apr 2015 11:04:24 +0300
Itamar Haber <ita...@redislabs.com> wrote:

> Yes - Redis Cluster is made up of individual Redis shards that behave
> like standalone Redis in that sense.

This is correct only when you are sure that no migration of failover
will happen. Imagine the following situation. Slot for key K was
managed by node A, but at some point has been migrated to node B.
Application asynchronously sends the following commands:

INCRBY K 1
INCRBY K 2
DEL K

Client don't know about migration yet, so it sends request 'INCRBY K 1'
to node A, next 'INCRBY K 2' it sends again to node A, then it receives
-MOVED response to the first sent command and notes that slot for K has
been moved to node B so it sends 'INCRBY K 1' to node B, then it gets
'DEL K' from application and sends it to node B, and finally it receives
-MOVED response for the second increment and sends it to B. In the end
B receives commands in the following order:

INCRBY K 1
DEL K
INCRBY K 2

--
Pavel Shaydo

[Josiah Carlson]

I have documented and discussed Redis cluster + async being funky a few different times: https://github.com/antirez/redis/issues/2151 .

If you want those 3 commands to complete sequentially, put them in a MULTI/EXEC transaction. Otherwise timing may affect command ordering and execution when confronted with cluster failover/data migration.

 - Josiah

[Josiah Carlson]

Actually, I think it would be good documentation to say "Given arbitrary delays on a network connection, and/or the vagaries of asynchronous calls/pipelining (which are equivalent in this sense), you can only guarantee sequential command execution of any series of commands if they are surrounded by MULTI/EXEC, and/or are just a single Lua scripting call."

You can force a multi-command-sequence to be arbitrarily reordered with failures at the right time. And I'll repeat myself again, the only way to guarantee sequential operation is with MULTI/EXEC or a Lua script. The failure is possible if you have a highly-delayed, but still-functioning network connection. Or if Redis were CPU-bound and the timeout parameters were set low, or an overloaded/broken/buggy proxy, etc. Wrap your stuff in MULTI/EXEC or use Lua scripts. Seriously. I'd be surprised if https://aphyr.com/ doesn't figure this out during his testing.

 - Josiah

[Pavel Shaydo]

On Thu, 9 Apr 2015 10:48:58 -0700
Josiah Carlson <josiah....@gmail.com> wrote:
> I have documented and discussed Redis cluster + async being funky a
> few different times: https://github.com/antirez/redis/issues/2151 .
>
> If you want those 3 commands to complete sequentially, put them in a
> MULTI/EXEC transaction. Otherwise timing may affect command ordering
> and execution when confronted with cluster failover/data migration.

Are there any guidelines on how redis cluster client should implement
MULTI/EXEC support? These commands don't have keys, so I can imagine
client can wait for the command following MULTI before deciding which
connection to use, and then it should keep using the same connection
for all the commands till EXEC and ignore redirections or fail as soon
as it got an error.

--
Pavel Shaydo

[Josiah Carlson]

Redis cluster absolutely has documentation on how MULTI/EXEC works. The server side of Redis cluster (and standard Redis for that matter) buffers all commands after the MULTI, then upon receiving EXEC, will check keys, make sure that they are all in the same shard, that the shard is served by that machine, and that all individual keys are still being managed by that server (just in case you sent your MULTI/EXEC group of commands in the middle of a shard migration). If all keys are ready and available, it executes all commands without stopping, sends the reply to the client, and replicates the writes to slaves. If even *one* key is not being handled by the master you are sending commands to, the entire group of commands is aborted, and you get errors. A similar thing happens when you use a Lua script with the keys that are passed.

 - Josiah

--
Pavel Shaydo

[Pavel Shaydo]

On Fri, 10 Apr 2015 14:46:54 -0700
Josiah Carlson <josiah....@gmail.com> wrote:

> Redis cluster absolutely has documentation on how MULTI/EXEC works.
> The server side of Redis cluster (and standard Redis for that matter)
> buffers all commands after the MULTI, then upon receiving EXEC, will
> check keys, make sure that they are all in the same shard, that the
> shard is served by that machine, and that all individual keys are
> still being managed by that server (just in case you sent your
> MULTI/EXEC group of commands in the middle of a shard migration). If
> all keys are ready and available, it executes all commands without
> stopping, sends the reply to the client, and replicates the writes to
> slaves. If even *one* key is not being handled by the master you are
> sending commands to, the entire group of commands is aborted, and you
> get errors. A similar thing happens when you use a Lua script with
> the keys that are passed.

That's not really what I see. Redis cluster checks if key is managed by
the node when it queues command, not upon receiving EXEC. I submitted
an issue https://github.com/antirez/redis/issues/2515 which contains
details.

Anyways, my question was more about client side, how redis cluster
client should implement MULTI/EXEC support. I see that
antirez/redis-rb-cluster doesn't support multi, so are there any
clients that do support it?

--
Pavel Shaydo

[Josiah Carlson]

I'm sure that the Python library does, though to what extent it works as you expect/experience, I can't say (I read the Redis 3.0 RC4 MULTI/EXEC source a couple weeks back, out of curiosity).

 - Josiah

--
Pavel Shaydo