Odd behavior during key migration...can it be explained?

[spencer...@gmail.com]

Hi.  I've got some code that will setup a little mini redis cluster on my local machine.  I then setup a bunch of keys and now I've been playing with migrating them.  I've successfully migrated keys from one node to another, but I don't understand the redirect behavior that I'm seeing.  Here is an example...

E:\redis>redis-cli -p 7000 migrate 127.0.0.1 7004 strawberries 0 0

OK

E:\redis>redis-cli -p 7000 get strawberries

(error) ASK 2250 127.0.0.1:7004

E:\redis>redis-cli -p 7004 get strawberries

(error) MOVED 2250 127.0.0.1:7000

Before the first command shown above, I had already set the slot in question (2250) as importing on the destination node and migrating on the source node.

What I don't understand is why, after migrating the key "strawberries", that the node on port 7004 would still give me a "moved" redirect.  Though the migration of all keys for slot 2250 might not be complete, shouldn't I still at this point be able to fetch the key from the destination node?

Once I issue the set-slot command on the destination node, the migration completes, and then I can fetch the key from 7004.

Do I just not understand the behavior here?  It doesn't make sense to me that both nodes would redirect toward one another.  The client would just bounce back and forth between the source and destination nodes looking for the key "strawberries" until the migration was complete.

I know that ASK and MOVED are two different kinds of redirects; the the former is meant to solve some sort of race condition.  I'll have to read up on that some more.  Until then, I really don't understand what's going on here.

This is Redis for windows, by the way.  Maybe it has a bug?

[Itamar Haber]

Hello Spencer,

Short explainer: `MIGRATE` moves keys between databases, whereas a cluster is a single logical database. Put differently, in the cluster keys are hashed to slot that, in turn, reside on shards - you can't migrate keys, only move slots around.

Cheers,

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/414e32af-d6df-454f-b942-4ff4ec1578c8o%40googlegroups.com.

--

Itamar Haber Technicalist Evangely Phone: +972.54.567.9692

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

[Spencer Parkin]

Itamar,

Thank you for your reply.

If the MIGRATE command can only move keys between databases, then I think the documentation at the end of the following page is wrong.

https://redis.io/commands/cluster-setslot

Specifically, step #3 in "Redis Cluster live resharding explained."  It says to use the MIGRATE command to move keys from one node of the cluster to another.

I was aware that, conceptually, there is just one database, and that in Redis cluster, this database's storage is spread across the nodes of the cluster.  That's why, when I looked at the documentation for the MIGRATE command, I was quite confused as to how this would be the means of moving keys between nodes of the cluster.

I will continue my research on how to migrate keys within a cluster.  The info has to be somewhere in the documentation.

On Tuesday, July 21, 2020 at 7:52:40 AM UTC-6, Itamar Haber wrote:

Hello Spencer,

Short explainer: `MIGRATE` moves keys between databases, whereas a cluster is a single logical database. Put differently, in the cluster keys are hashed to slot that, in turn, reside on shards - you can't migrate keys, only move slots around.

Cheers,

On Tue, Jul 21, 2020 at 8:57 AM <spencer...@gmail.com> wrote:

Hi.  I've got some code that will setup a little mini redis cluster on my local machine.  I then setup a bunch of keys and now I've been playing with migrating them.  I've successfully migrated keys from one node to another, but I don't understand the redirect behavior that I'm seeing.  Here is an example...

E:\redis>redis-cli -p 7000 migrate 127.0.0.1 7004 strawberries 0 0

OK

E:\redis>redis-cli -p 7000 get strawberries

(error) ASK 2250 127.0.0.1:7004

E:\redis>redis-cli -p 7004 get strawberries

(error) MOVED 2250 127.0.0.1:7000

Before the first command shown above, I had already set the slot in question (2250) as importing on the destination node and migrating on the source node.

What I don't understand is why, after migrating the key "strawberries", that the node on port 7004 would still give me a "moved" redirect.  Though the migration of all keys for slot 2250 might not be complete, shouldn't I still at this point be able to fetch the key from the destination node?

Once I issue the set-slot command on the destination node, the migration completes, and then I can fetch the key from 7004.

Do I just not understand the behavior here?  It doesn't make sense to me that both nodes would redirect toward one another.  The client would just bounce back and forth between the source and destination nodes looking for the key "strawberries" until the migration was complete.

I know that ASK and MOVED are two different kinds of redirects; the the former is meant to solve some sort of race condition.  I'll have to read up on that some more.  Until then, I really don't understand what's going on here.

This is Redis for windows, by the way.  Maybe it has a bug?

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/414e32af-d6df-454f-b942-4ff4ec1578c8o%40googlegroups.com.

[Spencer Parkin]

Okay, after carefully reading...

https://redis.io/topics/cluster-spec

...I have figure this out.  Look at the following snap-shot.

In this example, there is a slot (7162) that has a key (oranges) in the node at port 7002 and we want to move it to the node at port 7000.  We set the slots to importing and migrating where appropriate, then we pretend to be a client looking for the key, and get the expected behavior.  Then we migrate the key.  Then we again pretend to be a client looking for the key.  In order for us to be able to find the moved key, we have to both issue the ASKING command _and_ stay connected to the node, then issue the query for the key while again staying connected to the node.  For keys of the slot that have not yet migrated, we should still be able to find them in the source node, but to find migrated keys, we have to use the ASKING command.  This ASKING command solves a race condition that could occur where the client wants to set a value in the destination node, but then that new value gets stomped by a migration.  Maybe.  I don't know.  I'm still a bit fuzzy on that.  But in any case, I at least see now what a client needs to do to still be able to find keys during mid-migration.

And by the way, the MIGRATE command is used both inside and outside the Redis cluster protocol.

[Benjamin Sergeant]

I recommend reading the redis-cli.c source code, the cluster section shows what needs to happen to migrate keys/slots between cluster nodes.

I have a python cli+library that does something equivalent which you can look at too if you are python savy.

There's a function that really closely follow the 4 steps highlighted in the cluster 'migrated' doc

https://github.com/machinezone/rcc/blob/master/rcc/cluster/reshard.py#L37

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/b00234e8-e39d-4f76-a965-a775aaa42c23o%40googlegroups.com.