Get number of keys, matching a pattern
Alexander Gladysh
I have a zillion keys in my redis DB.
Sometimes, very rarely, I need to count all of them, matching a
certain pattern. I'm OK (more or less) with the fact that the such
query would be really slow.
However, `KEYS pattern:*` is too much overhead — my client even does
not cope with creating data for the return value.
Note that I can't put my keys to a hash, since the values on these
keys are hashes themselves. I also can't keep a separate counter for
the number of keys. (Well, actually I can and do, but, very rarely, I
must correct this counter to compensate for software errors — this is
the use-case for this question.)
So, is there a way to simply count the keys without getting them all?
Or maybe it can be implemented in future versions?
Thanks,
Alexander.
Javier Guerra Giraldez
> Sometimes, very rarely, I need to count all of them, matching a
> certain pattern. I'm OK (more or less) with the fact that the such
> query would be really slow.
>
> However, `KEYS pattern:*` is too much overhead — my client even does
> not cope with creating data for the return value.
i've been thinking that there are some uses for simple specialized
slaves to a Redis server.
as i understand it, if you set a Redis slave, you would get a copy of
every data-modifying command sent to the master... but only those that
actually do some changes, right?
if that's true, it should be easy (famous last words) to write a
process that registers as a slave but doesn't keep a full replica of
all data, just something like this:
master sends "SET key val" counting-slave does "SADD partof(key) key"
master sends "DEL key" counting-slave does "SREM partof(key) key"
where partof(x) is the interesting part of the key (typically a
prefix). now, counting the number of keys with that prefix is just
"SCARD prefix"
and, of course, different pseudo-slaves could do lots of different nice things:
- automatically keeping inverse indexes
- specific or sophisticated statistics
- keep a subset of data, maybe in preparation for re-sharding.
- aggregate subsets of data from several masters
maybe a new 'connect as slave' function to client libraries?
--
Javier
Alexander Gladysh
> On Fri, Apr 8, 2011 at 5:01 PM, Alexander Gladysh <agla...@gmail.com> wrote:
>> Sometimes, very rarely, I need to count all of them, matching a
>> certain pattern. I'm OK (more or less) with the fact that the such
>> query would be really slow.
>> However, `KEYS pattern:*` is too much overhead — my client even does
>> not cope with creating data for the return value.
> master sends "SET key val" counting-slave does "SADD partof(key) key"
> master sends "DEL key" counting-slave does "SREM partof(key) key"
Would not work for me. I need to count the actual number of keys.
(Most of the time I can live with the simple counter, but from time to
time I *must* validate that the counter — or any other indirect data —
is not off.)
Besides, your way would heavily increase memory requirements — for my
zillion of keys even set of partof(key) amounts to lots of data.
Alexander.
Javier Guerra Giraldez
> Besides, your way would heavily increase memory requirements — for my
> zillion of keys even set of partof(key) amounts to lots of data.
of course. but it might be an improvement over the current approach
of keeping a full replicating slave just to do the counting without
overwhelming the master.
--
Javier
Josiah Carlson
It's not that hard to implement something that just returns the number
of keys that match a particular pattern. It would still need to scan
all of your keys, but at least it wouldn't need to build up a huge
buffer and send it to you.
- Josiah
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>
Alexander Gladysh
> It sounds to me like you need to write your own custom command.
> It's not that hard to implement something that just returns the number
> of keys that match a particular pattern. It would still need to scan
> all of your keys, but at least it wouldn't need to build up a huge
> buffer and send it to you.
This is not an option for me. I can't use patched Redis.
Alexander.
Josiah Carlson
Redis, only paying attention to the number of results coming over the
line?
If not, I guess you are out of luck unless someone implements it and
pushes it into mainline Redis (feel free to post the branch here :) ).
- Josiah
Alexander Gladysh
> Could you hack your own client to discard the actual results from
> Redis, only paying attention to the number of results coming over the
> line?
I could, but I really want to avoid this.
> If not, I guess you are out of luck unless someone implements it and
> pushes it into mainline Redis (feel free to post the branch here :) ).
This is more or less what I asked in the first place. :-)
Alexander.
catwell
https://github.com/catwell/redis/commit/7e9aa29deafa6ec2ebe84ed2156695f02a70d2e8
I haven't written it but I often rebase it against antirez's master. I
would also like it to be merged into mainline if possible.
Alexander Gladysh
Looks simple enough — and useful.
Is there a known motivation for not including it into mainline?
Alexander.