Redis cluster key distribution

[Dipak Bhimani]

Hi All,

I am using the Redis cluster functionality but I have use case where I want to setup my own custom key distribution. I think ( I might be wrong here) as of now there is no way to customize the key distribution in Redis Cluster so I am wondering if is it possible to disable the key distribution ? I can write intelligent client which takes care of inserting/reading data on correct cluster node and I will still be able to use other functionality like fail-over,replication,slave promotion to master etc. 

Let me try to explain the need of customizing key distribution.

I have detailed information about entity abc which has several attributes ( One or more attributes can be updated independently) 

hmset c1:1 id 1 attr1 val1 attr2 val2

hmset c1:2 id 2 attr1 val1 attr2 val2

hmset c1:3 id 3 attr1 val1 attr2 val2

hmset c2:1 id 1 attr1 val1 attr2 val2

hmset c2:2 id 2 attr1 val1 attr2 val2

hmset c2:3 id 3 attr1 val1 attr2 val2

Now I have sorting/filtering use case (user should be able to sort on any of the above attribute ), so have started keeping one set which contains only key information. I am calling this key as "idset". 

sadd c1:idset c1:1 c1:2 c1:3 ( keeps key for all c1 entity)

sadd c2:idset c2:1 c2:2 c2:1 ( keeps key for all c2 entity)

With this I will be able to sort of any parameter and get all details about entity with following CLIs ( UI will be displayed this information in tabular format)

sort c1:idset asc  by *->id GET *->id  GET *->attr1  GET *->attr2  ( Sort by id)  [ sort by id for c1] 

sort c2:idset asc  by *->attr1 alpha GET *->id  GET *->attr1  GET *->attr2  ( sort by attr1 for c2) 

and so forth....

Now I have business requirement which allow to access only "c1" or "c2" at a time. So in my client I can precisely construct the above command. That's the reason I have kept one "idset" set per "c*". 

With this I would like to keep one "c*" at one node. 

node1 can have following - 

hmset c1:1 id 1 attr1 val1 attr2 val2

hmset c1:2 id 2 attr1 val1 attr2 val2

hmset c1:3 id 3 attr1 val1 attr2 val2

sadd c1:idset c1:1 c1:2 c1::3 

Node2 can have following - 

hmset c2:1 id 1 attr1 val1 attr2 val2

hmset c2:2 id 2 attr1 val1 attr2 val2

hmset c2:3 id 3 attr1 val1 attr2 val2

sadd c2:idset c2:1 c2:2 c2:3

So that I can make sure that whenever I run the sorting for any "c" ( c1/c2/cn), all the relevant data in one node. With Cluster behavior I might have key distributed across nodes for same c1 as it works on whole key hashing. So this will not work.

I hope I have explained it cleanly. I am open for changing the existing implementation.

Looking forward to hear more ideas/thoughts.

Thanks,

Dipak

[Josiah Carlson]

There was supposed to be a mechanism where you could tell Redis how to shard your keys. IIRC, if you have a key like "user:132:attribute" you should at some point be able to say "user:{132}:attribute or similar and Redis is supposed to understand that you want to shard on "132". This particular mechanism was chosen because several client libraries already support similar behavior in non-cluster scenarios (for client-only sharding).

I don't know the progress on that piece, whether it is done, or even if I have the exact syntax correct (it was decided over 2 years ago, IIRC). But there was supposed to be a semantic to specifying the portion of the key to use for sharding.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

[Dipak Bhimani]

Hi Salvatore,

Request you to throw some light on this topic. It would be great to have this kind of sharding mechanism in Redis Cluster.

Thanks,
Dipak

[Yiftach Shoolman]

This is simply called sharding by RegEx

--

Yiftach Shoolman
+972-54-7634621

[Dipak Bhimani]

Yes, this is what I precisely looking for sharding with RegEx. 

As Josiah mentioned that this is supported by some of the client library and I am aware of same support in Jedis. My real question was Does Redis Cluster supports the sharding of key with RegEx itself ? At looks like it doesn't at the moment as I have not seen any documentation around this. I have tried using "{}" in key but it looks like it is not working at the moment (simillar

One more interesting point here - Even If I try to insert the data in one of the Redis Cluster node directly, still Redis inserts the data into other nodes based on hashing of keys. This implies before Redis puts the data into memory, it calculate the hash of key and identify the right node who is responsible for handling this key.

I am looking for similar solution http://oldblog.antirez.com/post/redis-presharding.html ( refer the hashtag section) in Redis Cluster environment. 

Please share the information if anyone has face the similar experience/situation.

Thanks,

Dipak

[Manuel Pereira]

Hi all!

Any lights on this topic?

We at Appgree have a Redis production environment composed by 20-Master + 20-Slaves, that currently uses a client sharding mechanism... really looking forward for the Cluster :)

As Dipak Bhimani, we would like to have the feature on redis cluster to limit the part of the key that is used to calculate the Cluster Slot (what Yiftach calls "sharding by Regex", and Salvatore calls "Has tags" on this old post: http://oldblog.antirez.com/post/redis-presharding.html)

We need to use some multi-key commands as intersect and sort, and with this feature we would assure that all the keys are in the same Slot.

Is this feature planned to be implemented in Redis Cluster?

Thanks in advance!

[Salvatore Sanfilippo]

Hello,

in Redis Cluster there will be no way to run multi-key commands nor to
do regex-sharding.
The reason is pretty simple: one of the main features of Redis Cluster
is that it reshards online. I expect that in large clusters there will
be often keys moving from nodes to other nodes to rebalance, HW
upgrades, and so forth, so basically the idea that two keys are in the
same node is against a database that supports proper distribution.

There is the following path about this in the future: to use MIGRATE
COPY in order to move keys in space non-cluster instances and perform
inter-key operations. Maybe in the future we'll make this smarter.
Currently the barrier about this is that MIGRATE is blocking. For
resharding to block is ok since we reshard with a single client one
key after the other and are able to select the resharding speed,
however for MIGRATE COPY and high loads, this is not great... So the
idea is that MIGRATE in the next versions of Redis Cluster will block
only the key it is moving. This is definitely not very complex and
will make:

1) Resharding faster.
2) MIGRATE COPY useful to move data to other nodes to perform complex
computations offline.

Basically users should be able to mount setups where there are
computational nodes you can use to perform things like set
intersections between keys or similar things that have copies of the
keys running in the cluster. Those copies can have expires so that the
application logic can recopy updated versions of the keys when needed.
This can't accomodate all the use cases for a cluster where it is
still possible to perform multi-keys operations but should be able to
cover some use case definitely.

Without to mention that of course it is possible to design things that
move keys from nodes to other nodes to allow multi-keys ops: I'll
never do this because it is obvious that can't perform with the
latency users expect from Redis. Moving keys to computational nodes is
different because:

1) It is *evident* in the application logic you are moving stuff
between nodes: take your tradeoffs.
2) It is pretty clear that this approach encourages to cache stuff on
the computational nodes so that there is no need to move stuff at
every call.
3) It is always possible to improve over time creating more
interesting patterns / features when there are the conditions to make
things working well not just at API level.

Salvatore

Salvatore 'antirez' Sanfilippo
open source developer - GoPivotal
http://invece.org

To "attack a straw man" is to create the illusion of having refuted a
proposition by replacing it with a superficially similar yet
unequivalent proposition (the "straw man"), and to refute it
-- Wikipedia (Straw man page)

[Manuel Pereira]

Thanks Salvatore!

I read in twitter that after talking to @Yiftachsh you are considering to implement hash slots on redis cluster.

Would that allow us to use multi-key operations on cluster assuming only performing these operations with keys with the same hash-slot?

i.e:
SADD key1{shard1}
SADD key2{shard1}
SUNION key1{shard1} key2{shard1}

Thanks in advance.

[Manuel Pereira]

Hello, I forgot to mention that by making multi-key operations we would assume under our responsability than during re-sharding process some of those operations could fail (because maybe one key has been moved and the other one doesn't).

We would add or remove nodes/machines to our redis cluster in a very controlled way, preventing those kind of operations to happend during this process.

Waiting for conclusions on thread "change to Redis Cluster hashing algorithm".

Thanks!

[Salvatore Sanfilippo]

On Sat, Feb 8, 2014 at 10:06 AM, Manuel Pereira <man...@gmail.com> wrote:
> Hello, I forgot to mention that by making multi-key operations we would
> assume under our responsability than during re-sharding process some of
> those operations could fail (because maybe one key has been moved and the
> other one doesn't).

Hello Manuel,

that's the point. Under this assumption the implementation of
multi-key ops is 5 lines of code. The reason I'll not add it soon, at
least not in the first beta, is just to think more about it.
Basically the design space is minimal so there is no much to think
about, but, with the change in the hashing algorithm we open the door.

Now what happens is that if I'm sure about a thing is that I don't
want values to move around.
In turn this means that multi-keys op can only happen for keys in the
same hash slot that are in the same node.
Now keys in the same hash slot as you know are in the same node,
indeed, when the cluster is "stalbe" (on resharding in progress).

This means that during a resharding multi-key ops for keys into the
hash slot we are resharding in a given moment (we reshard one hash
slot after the other) have two options:

1) Become unavailabile.
2) Become inconsistent for some time.

Now I believe that the best thing is to default to "2", the operation
will be performed but if a key is not there, the result of the
operation will be inconsistent.
However I believe that we need to give the power to the user to
explicitly allow for "1".

There are multiple ways to do this:

1) Lua scripts that check the existence of all the keys, and a data
layout that ensures that keys are there anyway. If not all the keys
are found, the operation returns an error.
2) The addition of a MULTI/EXEC feature that allows to fail the
transaction if all the keys are not there, like that:

REQUIRE foo
REQUIRE bar
MULTI
SUNION foo bar
EXEC

This is semantically very similar to WATCH.

3) Some other different API to actually reach the same feature as of "2".

Because "2" or "3" don't need to exist ASAP, it is likely that beta2
will already have the ability to accept multi op commands.

Salvatore