Sharding question

45 views

Skip to first unread message

Jon Watte

unread,

Dec 20, 2009, 9:06:38 PM12/20/09

to Redis DB

The project summary says that Redis can shard using the client
library.

I feel that this is a mis-representation, but perhaps I'm missing
something. The reason is that you can't generally re-shard safely
while maintaining uptime when the sharding happens in the client
library, if you don't also add support for atomic sharding to the
server.

At a minimum, a server node needs to know what key range it is
responsible for, and reject requests for keys outside this range with
a unique error message, so that clients can re-try requests. If the
server doesn't do this, then there is a race condition where the
client doesn't know about a pending key space re-partition, and is
talking to the wrong server.

Also, moving data between nodes when re-sharding is something that you
really want to be supported on the server side.

Memcached doesn't have this problem, because it is not the
authoritative source of truth. Thus, if someone is talking to the
wrong server for a little bit, it doesn't matter, because either
they'll get a cache miss, and will go to the source of truth, or they
will do a cache put to the wrong node, which just means that whoever
comes next will get a cache miss.

Thus, I think that it's misleading to claim that Redis can support
sharding through a client-side library, the way that memcached does.
When you're dealing with a distributed source of authority/truth, the
game is signficantly changed.

Does Redis have any facility to deal with these problems, and I just
need to find the right place to read about it?