Subscribing to keys and notifications
Marcus
be notified of various actions. For example, a client could subscribe
to a key and be notified when the key is updated. This would allow
clients to monitor many keys in a non-blocking manner.
Salvatore Sanfilippo
Hello Marcus,
thanks to BLPOP now this is possible without the need to add more
primitives to Redis.
Basically the notified client BLPOP a key and the notifier LPUSH
something inside this key.
It is possible to use the pushed value to distinguish among different
notifications.
BLPOP is not intended just as blocking POP operation, but also as a
primitive to build other blocking application-level operations.
Cheers,
Salvatore
--
Salvatore 'antirez' Sanfilippo
http://invece.org
"Once you have something that grows faster than education grows,
you’re always going to get a pop culture.", Alan Kay
Steve Farrell
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
Salvatore Sanfilippo
> Salvatore,
> Is it possible to block on multiple keys? I'm thinking of beanstalk-type
> functionality, where a worker can listen to multiple tubes, each with jobs
> of varying priorities. Can you do (more or less) the same with BLPOP?
The only way to listen for multiple keys is that the worker listen for
a single key with BLPOP notify.me, and the notifier LPUSHES notify.me
<name of the key modified>.
I wonder if this can be used in order to address your problem.
Marcus
chat_777. Two clients can subscribe to chat_777 to be notified when
any changes take place. This way, both clients can see changes to
chat_777. Now, lets say that both of those redis clients also
subscribe to chat_778.
With the BLPOP implementation there would be two problems, right?
1) A client could not monitor chat_777 and chat_778 at the same time.
Instead, a third key would have to be created called chat_meta and
based on what was pushed to chat_meta, then the client could look at
any number of the keys it is really interested in. Or, can clients
monitor more than one key with this method? (assuming the client code
was using non-blocking sockets and if that is the case does redis
report the key in the response so we know which key changed).
2) Using the BLPOP functionality only one client gets the value at a
time. There is no way to say "When a value is pushed broadcast it to
these clients" So, if we had X clients the app code would have to
write to X chat_meta_N keys whenever there was an update to a key the
clients might be interested in.
On Dec 30, 12:28 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> On Wed, Dec 30, 2009 at 8:25 PM, Steve Farrell <st...@farrell.org> wrote:
> > Salvatore,
> > Is it possible to block on multiple keys? I'm thinking of beanstalk-type
> > functionality, where a worker can listen to multiple tubes, each with jobs
> > of varying priorities. Can you do (more or less) the same with BLPOP?
>
> The only way to listen for multiple keys is that the worker listen for
> a single key with BLPOP notify.me, and the notifier LPUSHES notify.me
> <name of the key modified>.
>
> I wonder if this can be used in order to address your problem.
>
> Cheers,
> Salvatore
>
> --
> Salvatore 'antirez' Sanfilippohttp://invece.org
Steve Farrell
Steve Farrell
Salvatore Sanfilippo
> I think for the most part this would look a little bit like the twitter
> sample implementation in the redis documentation: the sender delivers the
> message to all recipients.
> Clients could connect to a socket, and the socket server can do BLPOP, and
> then write the data to all listening clients.
> (I'm assuming BLPOP has a way for the client to configure a timeout, right?
> That'd be necessary to switch back and forth between the socket and the
> redis socket... well, unless you could get a handle to the socket in the
> client, and select on it and the listen socket... would that work?)
Steve, Marcus,
indeed BLPOP is not a truly general notification primitive. In theory
we could extend it allowing to listen for multiple keys, and I've to
confess you that this is a *trivial* extension to the current
implementation, it's a few lines of code. But there are a few things
preventing me from doing this:
1) Redis is a database. BLPOP made sense as Redis is a data structure
server, and when used as messaging system it makes a lot more sense to
block waiting for more data into a list. But should we really provide
more than this at this stage? After all BLPOP is not perfect but can
already perform a lot of notification tasks.
2) BLPOP was requested months ago. After some time it became obvious
it was really needed. Every new primitive should follow this iter, so
a vararg BLPOP requires to be proven useful and *generally* useful.
BLPOP is, there are tons of people that can benefit from blocking for
more list data, but I'm not sure currently a general notification
system is so useful. We'll see if in the next months we'll get more
requests about this with the problem explained, and actually it will
be hard/impossible to find a good solution with what we already have,
*and if this usage* is ok for Redis or outside our field of action.
3) Did you noticed that Redis list command names resembe Tcl's? This
is why in the past I was interested in Tcl as it is an interesting
language and one very interesting to implement. Tcl implements a
general notification system for variables, you can have a function
called if somebody alter the value of a var. This prevented a number
of speedups to the bytecode compiler, and if you ask me, it is not
worth it. I suspect that a similar thing could happen with Redis. A
notification system like the one suggested by Marcus is truly complex
to handle as it is not easy to implement in an orthogonal way. I'm not
convinced this is a good thing for Redis.
So for now it's almost impossible that Redis will get a general
notification system, and unlikely but not impossible that it will get
a BLPOP able to listen to more keys at once in the short period. It is
instead absolutely ok to implement multi-keys BLPOP once in the next
months its usefulness will be proved.
Salvatore Sanfilippo
> I see... so the the sender of the message would need to know to send both
> kinds of messages to the same key, instead of the recipient knowing to
> listen for two different keys. That seems workable in many cases.
Just an idea, don't know how viable it is, but it's not impossible
that when a client receives a message it's not interested in using
BLPOP it can PUSH it back into the list so that the next client will
be notified.
It's not a fully reliable way to do it as clients can crash, but I
guess in many real-world scenarios this may not be a problem, as the
same problem there is with BLPOP itself indeed (btw I'll implement
BRPOPLPUSH in the next days).
It's just an idea but I bet we have still a lot of things to discover
about BLPOP usages, as the primitive is trivial but the patterns we
can build using BLPOP and other primitives are still not fully
explored.
Steve Farrell
Marcus
isn't returning much.
On Dec 30, 5:17 pm, Steve Farrell <st...@farrell.org> wrote:
> That would be interesting, particularly if you could guarantee that each
> client only got notified once. It reminds me of a feature that's missing in
> beanstalk, at least for my purposes: there's no way to deliver a reliable
> message to a unknown collection of workers (you have to make a tube for each
> worker if you want that behavior).
>
> I definitely understand and appreciate your point about not wanting to make
> redis into a general notification system. In general, I'm personally happy
> using beanstalk and redis together. (I'm thrilled with redis, in
> particular.) I was just curious about the BLPOP function and was thinking
> through the distinction.
>
> Thanks!
>
> > redis-db+u...@googlegroups.com<redis-db%2Bunsu...@googlegroups.com>
Marcus
I see what you mean. This sort of notification is really outside of
the scope of Redis and I can see where it would add overhead for a
feature that probably a few percent of people would benefit from.
There are many existing messaging systems, which would probably be
better suited to handle notifications than hacking something into
Redis.
> Salvatore 'antirez' Sanfilippohttp://invece.org
Videla Alvaro
There are many messaging systems out there, like RabbitMQ for example, why you need that into Redis?
I really liked that Redis had a very specific scope before, but sometimes it looks like it's getting to many features.
My two cents,
Alvaro
Steve Farrell
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
Ezra Zygmuntowicz
On Dec 30, 2009, at 11:28 AM, Salvatore Sanfilippo wrote:
> On Wed, Dec 30, 2009 at 8:25 PM, Steve Farrell <st...@farrell.org>
> wrote:
>> Salvatore,
>> Is it possible to block on multiple keys? I'm thinking of
>> beanstalk-type
>> functionality, where a worker can listen to multiple tubes, each
>> with jobs
>> of varying priorities. Can you do (more or less) the same with
>> BLPOP?
>
> The only way to listen for multiple keys is that the worker listen for
> a single key with BLPOP notify.me, and the notifier LPUSHES notify.me
> <name of the key modified>.
>
> I wonder if this can be used in order to address your problem.
I am writing a small agent/actor library on top of the new blpop
right now and the way I am listening to multiple lists is to run a
thread for each key I want to blpop from with its own redis connection
object. We will see if this ends up working out well or if another way
of doing this would be better.
I could like to see a blpop that takes multiple keys to block against
but I also will wait and use current blpop for a while before I ask
for this as i'd like redis to stay as primitive as possible.
Now there is one other way to do this type of multiple listen thing.
You would have one list that workers blpop against and when you push a
message into there you add a key prefix describing the job and a data
key that contains the work data:
producer:
set 0xdeadbeef {'some': 'json', 'for': 'work'}
rpush worklist foo:0xdeadbeef
worker:
res = blpop worklist
type, key = res.split(':')
data = get key
work_on_#{type}(data)
delete key
Basically you have the producer push the json data or whatever else
is required for this work unit into a key '0xdeadbeef' then you rpush
a key with the type of work and the key to the data like "foo:
0xdeadbeef" into the worklist. The workers all use blpop to subscribe
to the worklist and when they get an item they split the string on :
to get a work type and a key into the data for the work, then dispatch
into the right method to work on this type of data and get the data
from the key and pass it into your work method.
Cheers-
Ezra Zygmuntowicz
e...@engineyard.com
Salvatore Sanfilippo
On Dec 31 2009, 10:28 pm, Ezra Zygmuntowicz <e...@engineyard.com>
wrote:
> I am writing a small agent/actor library on top of the new blpop
> right now and the way I am listening to multiple lists is to run a
> thread for each key I want to blpop from with its own redis connection
> object. We will see if this ends up working out well or if another way
> of doing this would be better.
>
> I could like to see a blpop that takes multiple keys to block against
> but I also will wait and use current blpop for a while before I ask
> for this as i'd like redis to stay as primitive as possible.
Hello Ezra!
if you are using multiple connections to write a small agent/actor
library I think this is a strong hint about the need of multi-keys
BLPOP...
at this point I'm tempted about implementing it.
Another reason to implement it now is that I realized there is to
change the API of BLPOP to support multiple values, as for example:
BLPOP key1 key2 key3 timeout
will need to return a two-elements multi-bulk-reply where the first
element is the key and the second the popped value.
So I wonder if you and the other guys interested in multi-keys BLPOP
like the interface of returning what is basically a two elements
array.
Cheers and thanks for sharing,
Salvatore
Salvatore Sanfilippo
On Dec 31 2009, 10:28 pm, Ezra Zygmuntowicz <e...@engineyard.com>
wrote:
> Now there is one other way to do this type of multiple listen thing.
> You would have one list that workers blpop against and when you push a
> message into there you add a key prefix describing the job and a data
> key that contains the work data:
About this, the limit here is that workers can't just listen to a few
specific queues right? Otherwise the "repush if you are not
interested" trick is needed.
Probably another evidence of multi-key BLPOP need.
Cheers,
Salvatore
Ezra Zygmuntowicz
Yeah if it is not too hard to add a multi key blpop that returns a
two element array would rock. I cannot figure out a way to subscribe
to multiple queues without having a thread and redis object for each
key, or by forcing all workers to listen to all messages on a single
list.
Before too many people write stuff with blpop I think we should
change it to always return a two element array of the key popped from
and the value popped. This way it can work the same whether you blpop
against one key or multiple. I think adding varargs to blpop would
make it much more useful as a primitive.
Cheers-
Ezra Zygmuntowicz
e...@engineyard.com