'[WARNING] Node has slots in importing state' while upgrading all nodes of redis cluster

346 views
Skip to first unread message

MANISH PATEL

unread,
Jul 15, 2020, 3:33:08 AM7/15/20
to Redis DB
Hi Team,

I am getting following error while upgrading all nodes of redis cluster(3 masters and 6 replicas).

>>> Check for open slots...
[WARNING] Node <IP_ADDRESS>:<PORT> has slots in importing state 5625,5820,5907,5932,6122,6462,6769,6918,6922,6981,6995,7056,7479,7688,7690,7709,7763,7809,7826,7882,8040,8176,8177,8241,8373,8471,8609,8812,8833,8842,8892,9021,9137,9423,9436,9549,9627,9817,10025,10124,10211,10594,10717,10758.
[WARNING] The following slots are open: 7479,9021,5625,8373,6769,8471,6922,8892,5932,8812,5820,8176,7809,7882,6981,7709,9549,8040,8177,8842,6995,7826,5907,8609,10211,7688,6918,10717,9436,10124,9423,7056,8833,6122,7690,8241,9627,6462,10594,7763,9817,10758,10025,9137.
>>> Check slots coverage...
[OK] All 16384 slots covered.

Is there any config issue or anything else ?

your help will be appreciated.

Thanks and Regards,
Manish

Benjamin Sergeant

unread,
Jul 15, 2020, 12:07:16 PM7/15/20
to redi...@googlegroups.com
This means that this slots are in 'migration' mode. You can reset that state if you are not resharding.

Can you run redis-cli in cluster fix mode ? Maybe it will reset the state of those slots.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/269d480b-bc24-432a-a1e2-c33e21800129n%40googlegroups.com.

Benjamin Sergeant

unread,
Jul 15, 2020, 12:18:22 PM7/15/20
to redi...@googlegroups.com
https://machinezone.github.io/rcc/

I wrote a resharding / redis-cluster helper tool which has a migrate command. You can use it to move slots from one machine to another explicitly.

Here I have a redis cluster running. I list its nodes.

(venv) rcc$ rcc cluster-nodes
27114fa175e7742b0944d4e2b7aca8b0893cfe92 127.0.0.1:30001 master 0-5460
adca3acc2875ef9380ab9b8fb8e9b62b86371f2e 127.0.0.1:30002 master 5461-10921
9676df67c7e0e13d1b2f5784b6c61ed061ee8c85 127.0.0.1:30003 master 10922-16383
23ec980b7732cfbf9d70ab9eb82c323b78efe12a 127.0.0.1:30004 slave 
c3c5f1a82b8c5a31e2a1fa8379c8dc38840e7699 127.0.0.1:30005 slave 
c66ba2f3468e42bee6779a227b4755145058dddc 127.0.0.1:30006 slave 


Now I move the slot 0 from the first master to the third master. I'm passing -v twice to see info and debug level messages printed. The input address are redis 'url' (I can improve things and make the redis::// part optional btw).

(venv) rcc$ rcc -v -v migrate --src-addr redis://127.0.0.1:30001 --dst-addr redis://127.0.0.1:30003 0
2020-07-15 09:13:55 DEBUG Using selector: KqueueSelector
2020-07-15 09:13:55 DEBUG key NODES -> slot 15728 -> connection 127.0.0.1:30001
2020-07-15 09:13:55 DEBUG Connecting to 127.0.0.1:30001...
2020-07-15 09:13:55 DEBUG [127.0.0.1:30001:ca25c95a] CLUSTER ('NODES',)
2020-07-15 09:13:55 INFO migrate from 27114fa175e7742b0944d4e2b7aca8b0893cfe92 to 9676df67c7e0e13d1b2f5784b6c61ed061ee8c85 slot [0]
2020-07-15 09:13:55 DEBUG key SETSLOT -> slot 1688 -> connection 127.0.0.1:30003
2020-07-15 09:13:55 DEBUG Connecting to 127.0.0.1:30003...
2020-07-15 09:13:55 DEBUG [127.0.0.1:30003:b38f5d2f] CLUSTER ('SETSLOT', '0', 'IMPORTING', '27114fa175e7742b0944d4e2b7aca8b0893cfe92')
2020-07-15 09:13:55 DEBUG key SETSLOT -> slot 1688 -> connection 127.0.0.1:30001
2020-07-15 09:13:55 DEBUG Connecting to 127.0.0.1:30001...
2020-07-15 09:13:55 DEBUG [127.0.0.1:30001:ca4447d7] CLUSTER ('SETSLOT', '0', 'MIGRATING', '9676df67c7e0e13d1b2f5784b6c61ed061ee8c85')
2020-07-15 09:13:55 DEBUG key GETKEYSINSLOT -> slot 16383 -> connection 127.0.0.1:30001
2020-07-15 09:13:55 DEBUG [127.0.0.1:30001:ca4447d7] CLUSTER ('GETKEYSINSLOT', '0', 100)
2020-07-15 09:13:55 DEBUG key SETSLOT -> slot 1688 -> connection 127.0.0.1:30001
2020-07-15 09:13:55 DEBUG Connecting to 127.0.0.1:30001...
2020-07-15 09:13:55 DEBUG [127.0.0.1:30001:d57f23bb] CLUSTER ('SETSLOT', '0', 'NODE', '9676df67c7e0e13d1b2f5784b6c61ed061ee8c85')
2020-07-15 09:13:55 DEBUG key SETSLOT -> slot 1688 -> connection 127.0.0.1:30002
2020-07-15 09:13:55 DEBUG Connecting to 127.0.0.1:30002...
2020-07-15 09:13:55 DEBUG [127.0.0.1:30002:9850c93a] CLUSTER ('SETSLOT', '0', 'NODE', '9676df67c7e0e13d1b2f5784b6c61ed061ee8c85')
2020-07-15 09:13:55 DEBUG key SETSLOT -> slot 1688 -> connection 127.0.0.1:30003
2020-07-15 09:13:55 DEBUG Connecting to 127.0.0.1:30003...
2020-07-15 09:13:55 DEBUG [127.0.0.1:30003:2fb4ed96] CLUSTER ('SETSLOT', '0', 'NODE', '9676df67c7e0e13d1b2f5784b6c61ed061ee8c85')


Now if I run cluster nodes again I can see that instance 3 has the slot 0 assigned.

(venv) rcc$ rcc cluster-nodes                                                                  
27114fa175e7742b0944d4e2b7aca8b0893cfe92 127.0.0.1:30001 master 1-5460
adca3acc2875ef9380ab9b8fb8e9b62b86371f2e 127.0.0.1:30002 master 5461-10921
9676df67c7e0e13d1b2f5784b6c61ed061ee8c85 127.0.0.1:30003 master 0 10922-16383
23ec980b7732cfbf9d70ab9eb82c323b78efe12a 127.0.0.1:30004 slave 
c3c5f1a82b8c5a31e2a1fa8379c8dc38840e7699 127.0.0.1:30005 slave 
c66ba2f3468e42bee6779a227b4755145058dddc 127.0.0.1:30006 slave 

MANISH PATEL

unread,
Jul 15, 2020, 2:08:07 PM7/15/20
to redi...@googlegroups.com
Thank you Benjamin.

I didn't not wanted to migrate slots in my case but it's showing importing state during upgrade of master and slaves node togather.

Is there any way to avoid this case ?

In this situation if I perform cluster fix then I'm loosing cached data. Is it expected ?


To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/49A5C608-E5C7-41FD-9681-47631F31F5CC%40gmail.com.

Benjamin Sergeant

unread,
Jul 15, 2020, 2:40:52 PM7/15/20
to redi...@googlegroups.com
Can you describe how you did your upgrade ?

Benjamin Sergeant

unread,
Jul 15, 2020, 2:42:40 PM7/15/20
to redi...@googlegroups.com
Cluster tutorial suggests:

It looks like slaves should be upgraded first, and then master. Is it what you did ?
===

Upgrading nodes in a Redis Cluster

Upgrading slave nodes is easy since you just need to stop the node and restart it with an updated version of Redis. If there are clients scaling reads using slave nodes, they should be able to reconnect to a different slave if a given one is not available.

Upgrading masters is a bit more complex, and the suggested procedure is:

  1. Use CLUSTER FAILOVER to trigger a manual failover of the master to one of its slaves (see the "Manual failover" section of this documentation).
  2. Wait for the master to turn into a slave.
  3. Finally upgrade the node as you do for slaves.
  4. If you want the master to be the node you just upgraded, trigger a new manual failover in order to turn back the upgraded node into a master.

Following this procedure you should upgrade one node after the other until all the nodes are upgraded.

====



MANISH PATEL

unread,
Jul 15, 2020, 2:47:24 PM7/15/20
to redi...@googlegroups.com
Hi,

I have upgraded using helm charts on kubernates (openshift) environment.

It will normally bring down all pods(nodes) almost at same time and bring new pods with updated redis image.

If I upgrade masters and slaves nodes seperately then this issue is not coming.


MANISH PATEL

unread,
Jul 15, 2020, 2:50:57 PM7/15/20
to redi...@googlegroups.com
Hi,

If I follow steps suggested by you and redis doc then issue won't occurs.

So means we shouldn't do all masters and slaves upgrade at same time.


Benjamin Sergeant

unread,
Jul 15, 2020, 2:55:23 PM7/15/20
to redi...@googlegroups.com
Yes that sounds right.

BTW I'm also using openshift (3.11), with custom resource files, but it's a bit hacky how I run it (but convenient and simple :). I want to try to use a replica-set like the redis-cluster-operator does it.

Reply all
Reply to author
Forward
0 new messages