Would like HMDEL - Am I doing it wrong?

[AndySk]

Hello Redis list,

I have a use case where a process reads all the items out of a hash in
redis (using hgetall), then the process needs to delete all of the
items the process read after work has been done. The issue is that
many field-value pairs could have been added while the work was being
done, so the process doesn't want to delete the entire hash.

The issue is that in my process I can read all items by reading all
hashes (in my case around 2000 calls, this is fast) but would have to
call delete on each field which is many more calls, in my case around
2 million. It takes a long time over a network, though it is fine
locally. It would be much more efficient if there was an HMDEL
command - that would work perfectly for me as I could only make 2000
calls.

Am I doing something wrong, or would an HMDEL command make sense? We
do have HMGET and HMSET. Thanks in advance.

Andy

[Damian Janowski]

Can you try pipelining? (If you don't know how to do it, let us know
which client you're using)

[AndySk]

Thank you Damian, that's all I needed. The time for the process I'm
testing went from approx 13 minutes to 30 seconds with pipelining.
For reference, I'm using Andy McCurdy's redis-py which seems to do an
excellent job of supporting pipelining.

Much Appreciated,

Andy

On Jul 21, 7:57 am, Damian Janowski <djanow...@dimaion.com> wrote:

[Andy McCurdy]

Hey Andy,

I don't really have a preference on HMDEL. What I can offer is another possible solution that might work better for you.

When your worker finds a hash that it wants to process, RENAME the hash to a temporary key name. Do the HGETALL like you're currently doing on the temporary key. When finished, simply DELETE the temporary key. This way you don't need to worry about new key/values coming in. If that happens, a new hash will get created and your worker can pick it up at a later time. This should be much faster.

Hope this helps.

-andy

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[AndySk]

Hi Andy,

Thanks so much for your work on redis-py, and for following up on this
question as well!

Your approach sounds like an excellent idea, and I see that the RENAME
is atomic. I will have to try that, it sounds ideal for my
application.

Much appreciated,

Andy Skalet

> > redis-db+u...@googlegroups.com<redis-db%2Bunsubscribe@googlegroups.c om>
> > .

[Salvatore Sanfilippo]

Hey,

Just read the thread, another approach as suggested by Pieter IIRC is
to turn HDEL into a var-arg like DEL itself. This makes sense I guess.

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.

> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

--
Salvatore 'antirez' Sanfilippo
http://invece.org

"Once you have something that grows faster than education grows,
you’re always going to get a pop culture.", Alan Kay