Redis update #7 python code and question

[adamjamesdrew]

import hashlib,redis
sha1 = hashlib.sha1

class RedisWrapper(redis.Redis):

def get_hash_name(self,key):
return sha1(key).hexdigest()[:4]

def exists(self,key):
return self.hexists(self.get_hash_name(key))

def get(self,key):
return self.hget(self.get_hash_name(key),key)

def set(self,key,value):
self.hset(self.get_hash_name(key),key,value)

def incrby(self,key,incr):
self.hincrby(self.get_hash_name(key),key,incr)

My questions is:
The MGET command is now broken. I use that command for performance
reasons. Can anyone think of an easy way to implement the MGET with
comparable performance?

[Josiah Carlson]

You need to know the name of all of your hash members in order to
perform a pipelined hget. But since you are scattering your hash
members over a bunch of different redis hashes (and even worse, you
are taking standard key/value entries and stuffing them into redis
hashes), that's not possible. That's also why you can't use hmgetall.

If you are trying to use your get_hash_name() as a method of sharding,
you are making a mistake with regards to it's use with hashes (and
really, your conflation of standard set() with hset() is part of
that).

Really, your set() method should be renamed and look something like...

def hset(self, key, field, value):
redis.Redis.hset(self, self.get_hash_name(key), field, value)

Other methods you list should be changed appropriately and similarly.

Is the code you have provided a mistake? What are you trying to do?

Regards,
- Josiah

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

[adamjamesdrew]

Those are all valid points. The reason I'm doing that is memory
savings. Please refer to this post by antirez.
http://antirez.com/post/redis-weekly-update-7.html

On Sep 6, 1:53 am, Josiah Carlson <josiah.carl...@gmail.com> wrote:
> You need to know the name of all of your hash members in order to
> perform a pipelined hget.  But since you are scattering your hash
> members over a bunch of different redis hashes (and even worse, you
> are taking standard key/value entries and stuffing them into redis
> hashes), that's not possible.  That's also why you can't use hmgetall.
>
> If you are trying to use your get_hash_name() as a method of sharding,
> you are making a mistake with regards to it's use with hashes (and
> really, your conflation of standard set() with hset() is part of
> that).
>
> Really, your set() method should be renamed and look something like...
>
> def hset(self, key, field, value):
>     redis.Redis.hset(self, self.get_hash_name(key), field, value)
>
> Other methods you list should be changed appropriately and similarly.
>
> Is the code you have provided a mistake?  What are you trying to do?
>
> Regards,
>  - Josiah
>

[Josiah Carlson]

I missed that article. My apologies.

... so you are really looking for a version of MGET to get the values
of a hash. Just use a sequence of HGET calls over a pipeline. The
redis-py library supports pipelines, and is pretty fast (I've found
non-transactional pipelines to be within 20-40% of the speed of
writing a custom command to do some equivalent operations on the
server).

- Josiah

[adamjamesdrew]

Thanks for the tip it's working great.

class RedisWrapper(redis.Redis):

def get_hash_name(self,key):
return sha1(key).hexdigest()[:4]

def exists(self,key):
return self.hexists(self.get_hash_name(key))

def get(self,key):
return self.hget(self.get_hash_name(key),key)

def set(self,key,value):
self.hset(self.get_hash_name(key),key,value)

def incrby(self,key,incr):
self.hincrby(self.get_hash_name(key),key,incr)

def incr(self,key,amount=1):
self.hincrby(self.get_hash_name(key),key,amount)

def mget(self, keys):
"""
Returns a list of values ordered identically to ``keys``
"""
pipe = self.pipeline(False)
for key in keys:
pipe.hget(self.get_hash_name(key),key)
result = pipe.execute()
return result

Here is my updated code.

On Sep 7, 11:55 am, Josiah Carlson <josiah.carl...@gmail.com> wrote:
> I missed that article.  My apologies.
>
> ... so you are really looking for a version of MGET to get the values
> of a hash.  Just use a sequence of HGET calls over a pipeline.  The
> redis-py library supports pipelines, and is pretty fast (I've found
> non-transactional pipelines to be within 20-40% of the speed of
> writing a custom command to do some equivalent operations on the
> server).
>
>  - Josiah
>