Curious about why BRPOPLPUSH doesn't currently support multiple source?
Ethan Collins
single destination and that should be fine with many requirements.
I faced this need while wrting up a redis based job queue. I have
multiple workers linked with a single redis list, N:1 relation (with N
ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track
0'. I now have the necessity to talk to any particular worker process,
especially for status, statistics etc purposes. For this, the best way
is to add 1 admin list for each of the worker processes (1:1). This
makes the command look like 'BRPOPLPUSH worker_N_admin job_list
job_track 0', expecting it to handle the source lists in the same
manner as BRPOP does.
Is this something that can be supported easily? Any other better way
of doing this?
Regards,
Ethan.
Pedro Melo
I have the exact same requirement. For now, I'm working around it with
a helper process that does routing between the global task list and
each worker own work list, based on availability information that each
worker puts on Redis.
It works but its not very pretty.
I havent looked at the possibility of implementing this routing logic
inside Redis with Lua in the scripting branch. Its something to
consider.
Salvatore: I know from past threads about scripting that the idea is
to send the entire Lua script every time I need to execute it. Are
there any plans to add an alternative method by which I could add new
Redis commands that would be implemented in Lua?
I don't quite remember why storing the script into a key to be reused
was a bad idea though. Something like:
SET my_new_command_script LUA_CODE
ROUTE NEW_COMMAND my_new_command
From that point on, the Redis server would accept NEW_COMMAND as a
valid command and forward it to the scripting engine with the
"my_new_command" script as the Lua code to execute.
Thanks,
--
Pedro Melo
@pedromelo
http://www.simplicidade.org/
http://about.me/melo
xmpp:me...@simplicidade.org
mailto:me...@simplicidade.org
Salvatore Sanfilippo
> Salvatore: I know from past threads about scripting that the idea is
> to send the entire Lua script every time I need to execute it. Are
> there any plans to add an alternative method by which I could add new
> Redis commands that would be implemented in Lua?
Hello Pedro,
we implemented EVALSHA for this (bandwidth) issue. Basically you send
EVALSHA <sha1 of your script>. If the script was already seen in that
server it gets executed, otherwise you get an error back and you'll
instead use EVAL for this time. The next time the EVALSHA will be
accepted.
Since EVALSHA will usually work you just send EVALSHA the first time,
so there is no additional round trip cost.
This is as good as defining commands from the point of view of
performances, but with the huge benefit that you don't need to make
sure your program is calling a Redis instance with the right version
of a command you implemented server-side and so forth. Also if you
have a cluster of Redis instances it is very handy to consider every
instance as every other regardless of defined script versions and so
forth.
All this is already into unstable.
Salvatore
--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele
Ethan Collins
logic using redis primitives. In this case, I think Lua script is a
workaround, as Pedro mentions.
From an API standpoint as well, I think the signature of command
BRPOPLPUSH should be 'symmetrical' with other similar commands, like
BRPOP/BLPOP. That is also the reason I feel this BRPOPLPUSH command
needs the update to support multiple source lists.
@Salvatore: What is your opinion about this update? If you agree and
if it's simple, is it possible to add it quickly?
Regards,
Ethan.
On Sep 24, 8:47 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
Salvatore Sanfilippo
On Sun, Sep 25, 2011 at 11:48 AM, Ethan Collins
<collins...@gmail.com> wrote:
> Well, adding Lua scripts is a method to add a layer of intelligent
> logic using redis primitives. In this case, I think Lua script is a
> workaround, as Pedro mentions.
Yes I was replying to Pedro about the EVAL/EVALSHA thing. Sure,
scripting is a way to model things without having to implement them
and is a workaround, but in that case it is not a good workaround
since blocking operations are not callable
within scripts.
But even if it scripting could solve very well that problem, I think
this does not really answer your question about why BRPOPLPUSH can
wait for just one list, while BLPOP/BRPOP is able to wait for multiple
lists.
There are mainly two reasons:
1) There is no obvious semantic for BRPOPLPUSH listening to more than
a single list. I bet that if you implement it as "wait for N lists,
push against one" other users will complain that we need a "wait for N
lists, push against N lists", such as:
BLPOPRPUSH queue1 queue2 track1 track2
But even if we implement the above, is not a generalization of your
use case pushing against a single list.
So basically we are going to make the command either still too simple
to cover all the cases, or bloated.
2) In the "push against single list" information, how to store the
information about the originating list in the target list?
And if you can't have that, why it is different than putting that info
inside the list element pushed itself? And this brings us to the point
three, that is, you can do that already without making Redis more
complex.
3) Your use case is perfectly addressable just adding the information
about the message type in the data itself.
You want to do:
BRPOPLPUSH job_queue admin_queue tracking_list
So whatever the push is against the queue or the admin list you are
going to pollute the tracking list with admin messages.
So you can do that already as simple as:
BRPOPLPUSH job_and_admin_list tracking_list
And when you have to push simply use a single byte as a message prefix
that can be "A" for admin messages, or "J" for jobs.
Thanks for the discussion as it is interesting to review old design
decisions from time to time, but I'm still pretty sure that the
current simple approach is the best for us.
Salvatore
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
Ethan Collins
Thanks for your quick response and explanation. I had thought about
your solution before posting the request, but that didn't solve my
problem. My problem doesn't stem from decoding the message type, my
messages are serialized python dicts with sufficient information held
inside. Probably, I didn't explain my need clearly or haven't
understood something properly. I will try to explain again.
My usecase is a single producer (redis list) multiple consumer model:
3 worker processes, P0/P1/P2 feeding on a single list 'job_queue' (all
processes can run any kind of job that job_queue provides). Each
process performs: 'BRPOPLPUSH job_queue job_track 0'.
In this case, the problem I face is when I want to explicitly push a
command to a particular process (this command is meant for gathering
statistics and other information from the context of that particular
process). Let's look at an example: I want to gather info regarding
the process P1, so I lpush the respective command. What can happen is
that P1 is actually busy on a lengthy job. So, either P0 or P2
responds to it. I backoff for a random time and repush the same
command again, expecting to hit P1. But in the meantime, the
application on the client side had pushed other commands, and P1 is
again busy with it's execution.
To, solve this, I thought of changing it as: 'BRPOPLPUSH admin_queue_1
job_queue 0'. NOTE that admin_queue is ahead of job_queue, as the keys
are checked in the order provided. And there are 3 admin_queues here,
for 3 processes. Now I can lpush the command directly to P1's
admin_queue_1 and it replies back to me with the first lpush of the
command.
Plese feel free to let me know if I went wrong somewhere or missing
something. As I understand your solution, it will not be able to
address this case.
Regards,
Ethan.
On Sep 25, 4:03 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> Hello Ethan
>
> On Sun, Sep 25, 2011 at 11:48 AM, Ethan Collins
>
David Brewer
the idea of the change to BRPOPLPUSH that Ethan is suggesting. Like
Ethan, I think that supporting multiple source keys would make the
command conceptually simpler because it would work in same way as
BRPOP, which is after all part of the name of the command. Based on
the name I would expect the command to work like a BRPOP which then
feeds into an LPUSH, but that is not currently the case.
I can offer my own use case if that's helpful. I'm implementing some
task queue functionality. A simple use case is that we have four
lists. Three of them represent different priorities, and one of them
is to store details of tasks which are currently in-progress. Let's
call our lists "high-priority", "medium-priority", "low-priority", and
"running".
The contents of the lists are tasks, and the information stored about
the task says nothing about its priority. That information is
represented only in the location of the task in the lists. When a
worker grabs a task to start working on it, I want to select the
oldest task from the high-priority list, the oldest task from the
medium-priority list, or the oldest task from the low-priority list,
whichever is first available. And, I want to simultaneously move this
task into the running queue in an atomic fashion.
In other words, I would like to do:
BRPOPLPUSH high-priority medium-priority low-priority running 30
One nice point about changing the command to work this way is that in
addition to making it consistent with BRPOP, it would be backwards
compatible with current uses of the command.
Is there a way to accomplish this use case using the current Redis
primitives that I'm missing? I don't think that putting information
about priority into the tasks themselves (collapsing the priority
lists into a single list) helps, because there is no efficient way to
use that information to sort the items in the queue so that high
priority items are selected first.
Thanks,
David Brewer
Suresh
multiple destinations, like
BRPOPLPUSH src1 dst1 src2 dst2 ...
My use case is as follows:
Suppose I use Redis as a Q server with Lists for Q's, each Q has a
name and is basically a Redis List storead against the name as Key.
For the Q client I would like to use the safe queues pattern mentioned
in the documentation for the command RPOPLPUSH.
Now, if the client is to "listen" to multiple Q's on the same Redis
server, I would require a socket connection for each Q because
BRPOPLPUSH does not support multiple Keys.
If it were to support multiple Keys I could have a dedicated socket
connection and start a BRPOPLPUSH command with all the Q's. Once I
Dequeue an element. I would cyclically rotate the list of keys to
ensure fairness. Of course the target key of the dequeued item would
be cleared once the message is processed successfully.
Suresh
On Oct 7, 4:22 am, David Brewer <david.bre...@gmail.com> wrote:
> I would like to chime in on this conversation to say that I support
> command conceptually simpler because it would work in same way as
> BRPOP, which is after all part of the name of the command. Based on
> the name I would expect the command to work like a BRPOP which then
> feeds into an LPUSH, but that is not currently the case.
>
> I can offer my own use case if that's helpful. I'm implementing some
> task queue functionality. A simple use case is that we have four
> lists. Three of them represent different priorities, and one of them
> is to store details of tasks which are currently in-progress. Let's
> call our lists "high-priority", "medium-priority", "low-priority", and
> "running".
>
> The contents of the lists are tasks, and the information stored about
> the task says nothing about its priority. That information is
> represented only in the location of the task in the lists. When a
> worker grabs a task to start working on it, I want to select the
> oldest task from the high-priority list, the oldest task from the
> medium-priority list, or the oldest task from the low-priority list,
> whichever is first available. And, I want to simultaneously move this
> task into the running queue in an atomic fashion.
>
> In other words, I would like to do:
>
> One nice point about changing the command to work this way is that in
> addition to making it consistent with BRPOP, it would be backwards
> compatible with current uses of the command.
>
> Is there a way to accomplish this use case using the current Redis
> primitives that I'm missing? I don't think that putting information
> about priority into the tasks themselves (collapsing the priority
> lists into a single list) helps, because there is no efficient way to
> use that information to sort the items in the queue so that high
> priority items are selected first.
>
> Thanks,
>
> David Brewer
>
> On Sep 25, 8:23 am, Ethan Collins <collins.eth...@gmail.com> wrote:
>
>
>
>
>
>
>
> > Hi Salvatore,
>
> > Thanks for your quick response and explanation. I had thought about
> > your solution before posting the request, but that didn't solve my
> > problem. My problem doesn't stem from decoding the message type, my
> > messages are serialized python dicts with sufficient information held
> > inside. Probably, I didn't explain my need clearly or haven't
> > understood something properly. I will try to explain again.
>
> > My usecase is a single producer (redis list) multiple consumer model:
> > 3 worker processes, P0/P1/P2 feeding on a single list 'job_queue' (all
> > processes can run any kind of job that job_queue provides). Each
> > > push against one" other users will complain that we need a "wait for N
> > > lists, push against N lists", such as:
>
> > > BLPOPRPUSH queue1 queue2 track1 track2
>
> > > But even if we implement the above, is not a generalization of your
> > > use case pushing against a single list.
> > > So basically we are going to make the command either still too simple
> > > to cover all the cases, or bloated.
>
> > > 2) In the "push against single list" information, how to store the
> > > information about the originating list in the target list?
> > > And if you can't have that, why it is different than putting that info
> > > inside the list element pushed itself? And this brings us to the point
> > > three, that is, you can do that already without making Redis more
> > > complex.
>
> > > 3) Your use case is perfectly addressable just adding the information
> > > about the message type in the data itself.
>
> > > You want to do:
>
> > > So whatever the push is against the queue or the admin list you are
> > > going to pollute the tracking list with admin messages.
> > > So you can do that already as simple as:
>
> > > And when you have to push simply use a single byte as a message prefix
> > > that can be "A" for admin messages, or "J" for jobs.
>
> > > Thanks for the discussion as it is interesting to review old design
> > > decisions from time to time, but I'm still pretty sure that the
> > > current simple approach is the best for us.
>
> > > Salvatore
>
> > > > From an API standpoint as well, I think the signature of command
Pedro Melo
On Mon, Oct 10, 2011 at 9:25 AM, Suresh <suresh.m...@gmail.com> wrote:
> I too would like to see BRPOPLPUSH support multiple sources, but with
> multiple destinations, like
>
> BRPOPLPUSH src1 dst1 src2 dst2 ...
>
> My use case is as follows:
>
> Suppose I use Redis as a Q server with Lists for Q's, each Q has a
> name and is basically a Redis List storead against the name as Key.
>
> For the Q client I would like to use the safe queues pattern mentioned
> in the documentation for the command RPOPLPUSH.
>
> Now, if the client is to "listen" to multiple Q's on the same Redis
> server, I would require a socket connection for each Q because
> BRPOPLPUSH does not support multiple Keys.
Exactly. This solution so far solves all the requested use cases. Is
it that bad to open multiple connections? Honest question, really.
Most of those connections will be mostly idle, the server uses epolll
so the cost is pretty negligeble (I've personally seen a *Perl*-based
XMPP server using epoll with a bit over 150k connections work just
fine, used for a singalling/notification service)...
Isn't it simpler to just use multiple connections instead of making
the command semantics harder?
Bye,
Suresh
Why is it that BRPOP takes multiple keys. Taking the consistency point
of view BRPOP need not accept multiple keys either.
The SUBSCRIBE command provided by Redis also allows listening for
multiple channels on a socket.
If we need to open a socket connection for each Q (List) in my
previous example we need one thread per List in Java which would mean
100 threads for 100 Qs. Of course there are ways to work around that
using Non Blocking IO or polling. But my point is having BRPOPLPUSH
support multiple sources and targets would make things easier for the
client.
Regards,
Suresh
> mailto:m...@simplicidade.org
Pedro Melo
On Mon, Oct 10, 2011 at 12:03 PM, Suresh <suresh.m...@gmail.com> wrote:
> Why is it that BRPOP takes multiple keys. Taking the consistency point
> of view BRPOP need not accept multiple keys either.
I disagree with the consistency argument here. BRPOP is single
direction: you remove stuff from lists, so the semantics is clear an
obvious.
BRPOPLPUSH is a two step command: remove from one, push into the
other. So I can think at least several semantics with multiple lists,
and I would not be able to say which one would be the correct, obvious
one:
* multiple sources, single destination;
* single source, multiple destinations;
* multiple sources, multiple destinations:
* map one-to-one: like having the current BRPOPLPUSH in parallel;
* any RPOPed element will be LPUSHed to all destinations.
Which one is the correct one?
So, no, there is no consistency argument to he had IMHO.
I would like to have a solution for multiple blocking actions in a
single TCP connection, but for now I think the multiple TCP
connections is the correct answer.
> The SUBSCRIBE command provided by Redis also allows listening for
> multiple channels on a socket.
True, but still very different use case, not a consistency issue.
> If we need to open a socket connection for each Q (List) in my
> previous example we need one thread per List in Java which would mean
> 100 threads for 100 Qs.
err... I'm sure you don't. Use one of the event loops for Java where
you can have a single thread serving all of your connection, I'm sure
that is possible.
> Of course there are ways to work around that
> using Non Blocking IO or polling.
Exactly.
> But my point is having BRPOPLPUSH
> support multiple sources and targets would make things easier for the
> client.
I don't disagree that a solution for blocking on several lists and
acting on the output would be easier for many, including myself who
use Redis as a task queue and had to work around this very same issue.
I'm just pointing that there are several semantics for such solution,
and personally I would not dare to say which one is the most obvious,
or most powerful, or most useful.
Best regards,
Suresh
> > Why is it that BRPOP takes multiple keys. Taking the consistency point
> > of view BRPOP need not accept multiple keys either.
>
> I disagree with the consistency argument here. BRPOP is single
> direction: you remove stuff from lists, so the semantics is clear an
> obvious.
>
> BRPOPLPUSH is a two step command: remove from one, push into the
> other. So I can think at least several semantics with multiple lists,
> and I would not be able to say which one would be the correct, obvious
> one:
multiple source multiple destination command unlike BRPOP.
However to me BRPOPLPUSH src1 dst1 src2 dst2 ... looks like a natural
and useful extension to BRPOPLPUSH src dst. But opinions and use cases
do differ.
> * multiple sources, single destination;
> * single source, multiple destinations;
single source command.
> * multiple sources, multiple destinations:
> * map one-to-one: like having the current BRPOPLPUSH in parallel;
about.
> * any RPOPed element will be LPUSHed to all destinations.
command to me.
...
> I'm just pointing that there are several semantics for such solution,
> and personally I would not dare to say which one is the most obvious,
> or most powerful, or most useful.
sources opens up multiple possibilities.
I am just pointing out the usefulness of this way of extending the
command which seems to be an obvious one to me.
It looks like this is not prioritized or not considered because of
multiple possibilities. At least in this thread I have seen only two
requirements:
1. What I mentioned
2. Multiple sources and a single destination which can be achieved by
repeating the destination key.
> Best regards,
> --
> Pedro Melo
> xmpp:m...@simplicidade.org
> mailto:m...@simplicidade.org
Suresh
David Brewer
> Hi,
>
> On Mon, Oct 10, 2011 at 12:03 PM, Suresh <suresh.m...@gmail.com> wrote:
>> Why is it that BRPOP takes multiple keys. Taking the consistency point
>> of view BRPOP need not accept multiple keys either.
>
> I disagree with the consistency argument here. BRPOP is single
> direction: you remove stuff from lists, so the semantics is clear an
> obvious.
>
> BRPOPLPUSH is a two step command: remove from one, push into the
> other. So I can think at least several semantics with multiple lists,
> and I would not be able to say which one would be the correct, obvious
> one:
>
> * multiple sources, single destination;
> * single source, multiple destinations;
> * multiple sources, multiple destinations:
> * map one-to-one: like having the current BRPOPLPUSH in parallel;
> * any RPOPed element will be LPUSHed to all destinations.
>
> Which one is the correct one?
I agree that there are multiple options here, all of which could be
useful for different use cases. But, I disagree that it's so
difficult to choose which one is the obvious case for an extension to
BRPOPLPUSH.
To me it seems obvious that BRPOPLPUSH is semantically meant to be an
atomic combination of BRPOP + LPUSH. So, it should behave like
performing a BRPOP and then an LPUSH, except it should do it
atomically. BRPOP supports popping an element from the first
non-empty list argument, and therefore ideally BRPOPLPUSH would
support that as well. LPUSH only supports pushing an element into one
list, so BRPOPLPUSH should support only one destination. I suppose you
could also parse the command as B(locking) RPOP + LPUSH -- which would
describe the current behavior of the command. Any other behavior for
this command would be very confusing.
The other use cases are useful, but it seems to me that they would be
candidates for a new, different commands with different names.
> So, no, there is no consistency argument to he had IMHO.
>
> I would like to have a solution for multiple blocking actions in a
> single TCP connection, but for now I think the multiple TCP
> connections is the correct answer.
I don't think multiple TCP connections addresses my use case of trying
to pick the first available task out of a series of lists (with each
list representing a priority), and then placing it in a "running"
list, in an atomic fashion. Currently I am choosing to use a call to
BRPOP to pick the task, followed by a call to LPUSH to put in the
running queue, and accepting the risk that if my process crashes at
the wrong time it may not end up in the running queue. I don't have
the option to use multiple calls to BRPOPLPUSH with its current
behavior, because the command would block on the "high priority" list
and I wouldn't see incoming "regular priority" or "low priority"
tasks.
Thanks,
David Brewer
Felix Gallo
F.
Pedro Melo
On Mon, Oct 10, 2011 at 7:13 PM, Felix Gallo <felix...@gmail.com> wrote:
> I originally thought scripting via lua was a bad design decision, but I've
> come around to it, and this is a perfect example of what it's for. You have
> a complex algorithm you'd like to execute atomically, and the existing
> command set doesn't seem suitable. You might give the scripting branch a
> try.
Not yet, if at all... Last I heard from Salvatore, Lua scripts cannot
block or call blocking Redis commands.
Bye,
Josiah Carlson
> Hi,
>
> On Mon, Oct 10, 2011 at 7:13 PM, Felix Gallo <felix...@gmail.com> wrote:
>> I originally thought scripting via lua was a bad design decision, but I've
>> come around to it, and this is a perfect example of what it's for. You have
>> a complex algorithm you'd like to execute atomically, and the existing
>> command set doesn't seem suitable. You might give the scripting branch a
>> try.
>
> Not yet, if at all... Last I heard from Salvatore, Lua scripts cannot
> block or call blocking Redis commands.
... well, they block, but they block *everything* until they are done,
which wouldn't help in this situation.
Regards,
- Josiah
skomski
Dave Peticolas
I would like to add another vote for extending BRPOPLPUSH to support
multiple lists. I think it is consistent with the existing behavior of BRPOP
and would make implementing prioritized job queues a snap, unlike the
use of multiple connections.
thanks,
dave
Pieter Noordhuis
in the pull request:
Where the regular blocking pop is unidirectional, BRPOPLPUSH is
bidirectional: it pops from a list and pushed onto another list. If
you look at the return value of the blocking pop, you'll see that it
both returns the list the value was popped from together with the
value itself. There is no way to preserve this information for
BRPOPLPUSH. A value is popped from one list and pushed to another. If
you think about a BRPOPLPUSH that pops from multiple lists to finally
push the value to another list, that information is lost. So, when
that information is lost anyway, why not push those values to a single
list to begin with?
Cheers,
Pieter
Dave Peticolas
This is not going to be supports. To echo what Salvatore already said
in the pull request:
Where the regular blocking pop is unidirectional, BRPOPLPUSH is
bidirectional: it pops from a list and pushed onto another list. If
you look at the return value of the blocking pop, you'll see that it
both returns the list the value was popped from together with the
value itself. There is no way to preserve this information for
BRPOPLPUSH. A value is popped from one list and pushed to another. If
you think about a BRPOPLPUSH that pops from multiple lists to finally
push the value to another list, that information is lost. So, when
that information is lost anyway, why not push those values to a single
list to begin with?
Well, for a collection of job queues with multiple priorities, the simple reason is
that you need the information before you do the pop (the information is the order
of the lists you give to the command, i.e., the relative queue priorities) and you
don't need it afterwards, at least you can live without it.
But since BRPOPLPUSH does not currently support this, surely if the command
is extended to support multiple keys, the reply value could be extended for the
multiple key case to also return the list from which the value was popped just like
BRPOP? Hence, no information loss at all and backwards compatibility with the
single key case.
cheers,
dave
Dave Peticolas
2012/3/27 Pieter Noordhuis <pcnoo...@gmail.com>This is not going to be supports. To echo what Salvatore already said
in the pull request:
Where the regular blocking pop is unidirectional, BRPOPLPUSH is
bidirectional: it pops from a list and pushed onto another list. If
you look at the return value of the blocking pop, you'll see that it
both returns the list the value was popped from together with the
value itself. There is no way to preserve this information for
BRPOPLPUSH. A value is popped from one list and pushed to another. If
you think about a BRPOPLPUSH that pops from multiple lists to finally
push the value to another list, that information is lost. So, when
that information is lost anyway, why not push those values to a single
list to begin with?
Well, for a collection of job queues with multiple priorities, the simple reason is
that you need the information before you do the pop (the information is the order
of the lists you give to the command, i.e., the relative queue priorities) and you
don't need it afterwards, at least you can live without it.
But since BRPOPLPUSH does not currently support this, surely if the command
is extended to support multiple keys, the reply value could be extended for the
multiple key case to also return the list from which the value was popped just like
BRPOP? Hence, no information loss at all and backwards compatibility with the
single key case.
And just to reply to the objection that the information is still lost in redis itself, you
could instead have the multikey case push two elements onto the list, the name
of the original list and the element itself.