ASK redirections & LUA scripts

[Carlos Abalde]

Hi,

I'm wondering what's the behaviour of a Redis instance X running in cluster mode when,

    (1) it receives an EVAL command hitting two keys K1 and K2 mapping to the same slot S;

    (2) the slot S is being migrated to a different instance Y;

    (3) key K1 has already been migrated to instance Y;

    (4) key K2 is still in instance X.

Will the LUA script running in X receive an ASK redirection when trying to read / write key K1? Are those redirections internally handled by the LUA engine? Any recommended best practices to handle these situations?

Thanks,

--

Carlos Abalde

[Josiah Carlson]

If you call EVAL or EVALSHA, and one of the keys that you pass as part of the KEYS arguments to the Lua script has been migrated, you will be informed of that fact through a -MOVED or -ASK redirection. Go here: http://redis.io/topics/cluster-spec and search for "migrating". When you get the -ASK redirection, you then hit the new machine with an ASKING call, see the section on "ASK redirection".

That is the documentation, I haven't had the chance to touch Redis Cluster yet, so your actual experience may vary.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Carlos Abalde]

> On 26 May 2015, at 01:28, Josiah Carlson <josiah....@gmail.com> wrote:
>
> If you call EVAL or EVALSHA, and one of the keys that you pass as part of the KEYS arguments to the Lua script has been migrated, you will be informed of that fact through a -MOVED or -ASK redirection. Go here: http://redis.io/topics/cluster-spec and search for "migrating". When you get the -ASK redirection, you then hit the new machine with an ASKING call, see the section on "ASK redirection".
>
> That is the documentation, I haven't had the chance to touch Redis Cluster yet, so your actual experience may vary.

Hi Josiah,

I'm aware of that part of the documentation in the spec. Let's assume everything works as you described in my original example:

1. Some client sends a EVAL command hitting K1 and K2 to instance X.
2. Instance X receives the command and notices that K1 and K2 map to the same slot S. S is owned by instance X, however S is being migrated to instance Y and one of the keys (K1) has already being migrated. The other key (K2) is still in instance X.
3. X replies with an -ASK redirection to the client.
4. The client sends again the EVAL command, now to instance Y, and using a pipeline in order to include the ASK command.
5. Instance Y receives the command but... ¿what's going to happen when the script tries to read / write key K2? That key has not been migrated yet to Y.

I've not been able to find anything in the documentation explaining the expected behaviour in this case:

- Maybe everything is 'magical' and LUA commands accessing to K2 will be forwarded to X but I don't expect that. That will break atomic execution of LUA scripts.
- Maybe LUA commands accessing to K2 will get a -MOVED redirection which will break the script execution. If so I guess Redis Cluster will not support this scenario and the script will be broken during the migration of the slot.

Thanks,

--
Carlos Abalde

[Jan-Erik Rediger]

If you want to use EVAL in Cluster mode, you want to pass all used keys
in the first arguments (which are later available in the KEYS variable
in the Lua script).
This way Redis can check what happens to the keys and you would get a
CROSSSLOT error if they are currently not all on the same node.

If you use other keys in your Lua script and the key is not available on
the node it is executed, your Lua script will fail with an error message
like:

(error) ERR Error running script (call to f_400bb3bd4925f5aa946852913b7a0d288bf62850): @user_script:1: @user_script: 1: Lua script attempted to access a non local key in a cluster node


I hope this clears things up, this should definitely be documented somewhere.

[Carlos Abalde]

> On 26 May 2015, at 10:38, Jan-Erik Rediger <jan...@fnordig.de> wrote:
>
> If you want to use EVAL in Cluster mode, you want to pass all used keys
> in the first arguments (which are later available in the KEYS variable
> in the Lua script).
> This way Redis can check what happens to the keys and you would get a
> CROSSSLOT error if they are currently not all on the same node.

Hi,

Yes. This definitely clears things up. I was assuming here that when using the EVAL command and all keys provided in the first argument mapped to the same slot a CROSSSLOT error was not possible. Now it's clear that during a slot migration CROSSSLOT errors are also possible even when all keys map to the same slot.

Thanks a lot,

--
Carlos Abalde

[Carlos Abalde]

> On 26 May 2015, at 10:38, Jan-Erik Rediger <jan...@fnordig.de> wrote:
>

> If you want to use EVAL in Cluster mode, you want to pass all used keys
> in the first arguments (which are later available in the KEYS variable
> in the Lua script).
> This way Redis can check what happens to the keys and you would get a
> CROSSSLOT error if they are currently not all on the same node.
>
> If you use other keys in your Lua script and the key is not available on
> the node it is executed, your Lua script will fail with an error message
> like:
>
> (error) ERR Error running script (call to f_400bb3bd4925f5aa946852913b7a0d288bf62850): @user_script:1: @user_script: 1: Lua script attempted to access a non local key in a cluster node
>
>
> I hope this clears things up, this should definitely be documented somewhere.

Btw, I've just realised that in fact this is documented in the cluster spec (http://redis.io/topics/cluster-spec#multiple-keys-operations):

"However multi-key operations may become unavailable when a resharding of the hash slot the keys belong to is in progress. More specifically, even during a resharding, the multi-key operations targeting keys that all exist and are still all in the same node (either the source or destination node) are still available. Operations about keys that don't exist or are, during the resharding, split between the source and destination nodes, will generate a -TRYAGAIN error. The client can try the operation after some time, or report back the error. As soon as the migration of the specified hash slot has terminated, all the multi key operations are available again for this hash slot."

[fj]

Hi Jan Erik,

I have reported a similar question/issue for MULTI/EXEC and migrating slots, but have not received any feedback yet.
Would you be able to have a quick look and comment on the issue posted here: https://github.com/antirez/redis/issues/2515 (MULTI/EXEC on cluster for migrated key)
For now I'm just looking to know whether it is a bug or intended behavior that we need to work around.

Best regards,

Flemming