what client libraries support consistent hashing?

[Michael Jiang]

In addition to Ruby, what other client libs already support consistent hashing?

Also, in addition to supporting distributed Redis servers from client sides, is there any plan for a distributed cluster version and what is the road map?

Thanks!

[Jonathan Leibiusky]

Jedis (Redis java client) supports consistent hashing.
http://github.com/xetorthio/jedis

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Arun Vijayan C]

Rediska (PHP)

http://rediska.geometria-lab.net/documentation/configuration/key-distributor/

-ArunV

[Michael Jiang]

What is the class for doing consistent hashing in Jedis? I read ShardedJedisTest.java & ShardedJedisPoolTest.java but didn't find 'consistent hashing'. Does that mean in current version of Jedis, consistent hashing is the only sharding method?

Anyway, I did want to see how "consistent hashing" works in Jedis. So, I started 2 redis servers at local host, one at 6379 and the other at 6380. Then, I started 2 redis clients, connecting to these two servers respectively and issued "monitor" to watch for requests on both servers. I expected that key-value pairs be evenly distributed among two servers. But it showed that the server at 6379 got much more data than server at 6380. This may not indicate a problem since the keys used in test might be biased in terms of the hashing function used. But want to check if this is the case and more evenly distributed key-value pairs among servers will be seen when data size grows.

Thanks!

[Matt]

Predis
https://github.com/nrk/predis/ says it does, too. I plan to try it
out.

On Apr 17, 4:34 pm, Arun Vijayan C <arunvija...@gmail.com> wrote:
> Rediska (PHP)http://rediska.geometria-lab.net/documentation/configuration/key-dist...
>
> -ArunV

>
> On Mon, Apr 18, 2011 at 1:49 AM, Jonathan Leibiusky <ionat...@gmail.com>wrote:
>
>
>
> > Jedis (Redis java client) supports consistent hashing.
> >http://github.com/xetorthio/jedis
>

[Jonathan Leibiusky]

On Mon, Apr 18, 2011 at 6:59 PM, Michael Jiang <it.mj...@gmail.com> wrote:

What is the class for doing consistent hashing in Jedis? I read ShardedJedisTest.java & ShardedJedisPoolTest.java but didn't find 'consistent hashing'. Does that mean in current version of Jedis, consistent hashing is the only sharding method?

You can find it here:
https://github.com/xetorthio/jedis/blob/master/src/main/java/redis/clients/util/Sharded.java#L53

And the two different hashing algorithms supported are MD5 (https://github.com/xetorthio/jedis/blob/master/src/main/java/redis/clients/util/Hashing.java#L9) and MurmurHash (https://github.com/xetorthio/jedis/blob/master/src/main/java/redis/clients/util/MurmurHash.java).

I found that Murmur distribute better and faster.
 

Anyway, I did want to see how "consistent hashing" works in Jedis. So, I started 2 redis servers at local host, one at 6379 and the other at 6380. Then, I started 2 redis clients, connecting to these two servers respectively and issued "monitor" to watch for requests on both servers. I expected that key-value pairs be evenly distributed among two servers. But it showed that the server at 6379 got much more data than server at 6380. This may not indicate a problem since the keys used in test might be biased in terms of the hashing function used. But want to check if this is the case and more evenly distributed key-value pairs among servers will be seen when data size grows.

It is hard to tell. I have tested this in production and it is distributed evenly. Of course it won't be exact. For example 80GB of data with million of keys, distributed across 8 nodes, result in nodes with 9-11 GB each. And this is with MD5. I think Murmur would be even better.

Please let me know if you find something wrong or you have more questions!

 

[Josiah Carlson]

On Mon, Apr 18, 2011 at 7:20 PM, Jonathan Leibiusky <iona...@gmail.com> wrote:
>
>
> On Mon, Apr 18, 2011 at 6:59 PM, Michael Jiang <it.mj...@gmail.com> wrote:
>>
>> What is the class for doing consistent hashing in Jedis? I read
>> ShardedJedisTest.java & ShardedJedisPoolTest.java but didn't find
>> 'consistent hashing'. Does that mean in current version of Jedis, consistent
>> hashing is the only sharding method?
>
> You can find it here:
> https://github.com/xetorthio/jedis/blob/master/src/main/java/redis/clients/util/Sharded.java#L53
>
> And the two different hashing algorithms supported are MD5
> (https://github.com/xetorthio/jedis/blob/master/src/main/java/redis/clients/util/Hashing.java#L9)
> and MurmurHash
> (https://github.com/xetorthio/jedis/blob/master/src/main/java/redis/clients/util/MurmurHash.java).
>
> I found that Murmur distribute better and faster.

You just got lucky (or unlucky) with your data set. Assuming your
hashing algorithm has features consistent with H(key) -> all outputs
equally likely, then it should place N items into K buckets at a
distribution more or less equivalent to inserting those same items
into K random buckets. Given that, "hot buckets" should be at most 2x
the size of the "average" N/K size buckets with high probability (1 -
1/N if I remember the class I took some 7-8 years ago), with the odds
of even one bucket exceeding that having probability 1/N.

MD5 has sufficient qualities to be roughly equivalent to a random
distribution for keys that weren't crafted to attack it, so should
have had (within a factor of 2) the same distribution of any other
"good" function.

Regards,
- Josiah

[Gleicon Moraes]

This one for twisted has consistent hashing which respects a hint for a given server.

it's embedded on cyclone as I said in another thread: https://github.com/fiorix/cyclone

More cowbell, please !

[Michael Jiang]

It seems Jedis just support sharding (w/ MD5 or MURMUR hashing algorithms) but not consistent hashing from what I saw (removing/adding nodes). Anyone clarify this?

On Sun, Apr 17, 2011 at 1:19 PM, Jonathan Leibiusky <iona...@gmail.com> wrote:

[yuanjun Li]

According to this post,  https://github.com/xetorthio/jedis/wiki/AdvancedUsage#the-downside

shards cannot be added or removed from a running ShardedJedis

在 2011年5月4日星期三 UTC+8上午9:05:29，mj写道：