When multiple source lists are given, BRPOPLPUSH can push them to a
single destination and that should be fine with many requirements.
I faced this need while wrting up a redis based job queue. I have
multiple workers linked with a single redis list, N:1 relation (with N
ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track
0'. I now have the necessity to talk to any particular worker process,
especially for status, statistics etc purposes. For this, the best way
is to add 1 admin list for each of the worker processes (1:1). This
makes the command look like 'BRPOPLPUSH worker_N_admin job_list
job_track 0', expecting it to handle the source lists in the same
manner as BRPOP does.
Is this something that can be supported easily? Any other better way
of doing this?
<collins.eth...@gmail.com> wrote: > When multiple source lists are given, BRPOPLPUSH can push them to a > single destination and that should be fine with many requirements.
> I faced this need while wrting up a redis based job queue. I have > multiple workers linked with a single redis list, N:1 relation (with N > ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > 0'. I now have the necessity to talk to any particular worker process, > especially for status, statistics etc purposes. For this, the best way > is to add 1 admin list for each of the worker processes (1:1). This > makes the command look like 'BRPOPLPUSH worker_N_admin job_list > job_track 0', expecting it to handle the source lists in the same > manner as BRPOP does.
> Is this something that can be supported easily? Any other better way > of doing this?
I have the exact same requirement. For now, I'm working around it with a helper process that does routing between the global task list and each worker own work list, based on availability information that each worker puts on Redis.
It works but its not very pretty.
I havent looked at the possibility of implementing this routing logic inside Redis with Lua in the scripting branch. Its something to consider.
Salvatore: I know from past threads about scripting that the idea is to send the entire Lua script every time I need to execute it. Are there any plans to add an alternative method by which I could add new Redis commands that would be implemented in Lua?
I don't quite remember why storing the script into a key to be reused was a bad idea though. Something like:
SET my_new_command_script LUA_CODE ROUTE NEW_COMMAND my_new_command
From that point on, the Redis server would accept NEW_COMMAND as a valid command and forward it to the scripting engine with the "my_new_command" script as the Lua code to execute.
On Sat, Sep 24, 2011 at 12:56 PM, Pedro Melo <m...@simplicidade.org> wrote: > Salvatore: I know from past threads about scripting that the idea is > to send the entire Lua script every time I need to execute it. Are > there any plans to add an alternative method by which I could add new > Redis commands that would be implemented in Lua?
Hello Pedro,
we implemented EVALSHA for this (bandwidth) issue. Basically you send EVALSHA <sha1 of your script>. If the script was already seen in that server it gets executed, otherwise you get an error back and you'll instead use EVAL for this time. The next time the EVALSHA will be accepted.
Since EVALSHA will usually work you just send EVALSHA the first time, so there is no additional round trip cost. This is as good as defining commands from the point of view of performances, but with the huge benefit that you don't need to make sure your program is calling a Redis instance with the right version of a command you implemented server-side and so forth. Also if you have a cluster of Redis instances it is very handy to consider every instance as every other regardless of defined script versions and so forth.
All this is already into unstable.
Salvatore
-- Salvatore 'antirez' Sanfilippo open source developer - VMware
http://invece.org "We are what we repeatedly do. Excellence, therefore, is not an act, but a habit." -- Aristotele
Well, adding Lua scripts is a method to add a layer of intelligent
logic using redis primitives. In this case, I think Lua script is a
workaround, as Pedro mentions.
From an API standpoint as well, I think the signature of command
BRPOPLPUSH should be 'symmetrical' with other similar commands, like
BRPOP/BLPOP. That is also the reason I feel this BRPOPLPUSH command
needs the update to support multiple source lists.
@Salvatore: What is your opinion about this update? If you agree and
if it's simple, is it possible to add it quickly?
Regards,
Ethan.
On Sep 24, 8:47 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> On Sat, Sep 24, 2011 at 12:56 PM, Pedro Melo <m...@simplicidade.org> wrote:
> > Salvatore: I know from past threads about scripting that the idea is
> > to send the entire Lua script every time I need to execute it. Are
> > there any plans to add an alternative method by which I could add new
> > Redis commands that would be implemented in Lua?
> Hello Pedro,
> we implemented EVALSHA for this (bandwidth) issue. Basically you send
> EVALSHA <sha1 of your script>. If the script was already seen in that
> server it gets executed, otherwise you get an error back and you'll
> instead use EVAL for this time. The next time the EVALSHA will be
> accepted.
> Since EVALSHA will usually work you just send EVALSHA the first time,
> so there is no additional round trip cost.
> This is as good as defining commands from the point of view of
> performances, but with the huge benefit that you don't need to make
> sure your program is calling a Redis instance with the right version
> of a command you implemented server-side and so forth. Also if you
> have a cluster of Redis instances it is very handy to consider every
> instance as every other regardless of defined script versions and so
> forth.
<collins.eth...@gmail.com> wrote: > Well, adding Lua scripts is a method to add a layer of intelligent > logic using redis primitives. In this case, I think Lua script is a > workaround, as Pedro mentions.
Yes I was replying to Pedro about the EVAL/EVALSHA thing. Sure, scripting is a way to model things without having to implement them and is a workaround, but in that case it is not a good workaround since blocking operations are not callable within scripts.
But even if it scripting could solve very well that problem, I think this does not really answer your question about why BRPOPLPUSH can wait for just one list, while BLPOP/BRPOP is able to wait for multiple lists.
There are mainly two reasons:
1) There is no obvious semantic for BRPOPLPUSH listening to more than a single list. I bet that if you implement it as "wait for N lists, push against one" other users will complain that we need a "wait for N lists, push against N lists", such as:
BLPOPRPUSH queue1 queue2 track1 track2
But even if we implement the above, is not a generalization of your use case pushing against a single list. So basically we are going to make the command either still too simple to cover all the cases, or bloated.
2) In the "push against single list" information, how to store the information about the originating list in the target list? And if you can't have that, why it is different than putting that info inside the list element pushed itself? And this brings us to the point three, that is, you can do that already without making Redis more complex.
3) Your use case is perfectly addressable just adding the information about the message type in the data itself.
You want to do:
BRPOPLPUSH job_queue admin_queue tracking_list
So whatever the push is against the queue or the admin list you are going to pollute the tracking list with admin messages. So you can do that already as simple as:
BRPOPLPUSH job_and_admin_list tracking_list
And when you have to push simply use a single byte as a message prefix that can be "A" for admin messages, or "J" for jobs.
Thanks for the discussion as it is interesting to review old design decisions from time to time, but I'm still pretty sure that the current simple approach is the best for us.
> From an API standpoint as well, I think the signature of command > BRPOPLPUSH should be 'symmetrical' with other similar commands, like > BRPOP/BLPOP. That is also the reason I feel this BRPOPLPUSH command > needs the update to support multiple source lists.
> @Salvatore: What is your opinion about this update? If you agree and > if it's simple, is it possible to add it quickly?
> Regards, > Ethan.
> On Sep 24, 8:47 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote: >> On Sat, Sep 24, 2011 at 12:56 PM, Pedro Melo <m...@simplicidade.org> wrote: >> > Salvatore: I know from past threads about scripting that the idea is >> > to send the entire Lua script every time I need to execute it. Are >> > there any plans to add an alternative method by which I could add new >> > Redis commands that would be implemented in Lua?
>> Hello Pedro,
>> we implemented EVALSHA for this (bandwidth) issue. Basically you send >> EVALSHA <sha1 of your script>. If the script was already seen in that >> server it gets executed, otherwise you get an error back and you'll >> instead use EVAL for this time. The next time the EVALSHA will be >> accepted.
>> Since EVALSHA will usually work you just send EVALSHA the first time, >> so there is no additional round trip cost. >> This is as good as defining commands from the point of view of >> performances, but with the huge benefit that you don't need to make >> sure your program is calling a Redis instance with the right version >> of a command you implemented server-side and so forth. Also if you >> have a cluster of Redis instances it is very handy to consider every >> instance as every other regardless of defined script versions and so >> forth.
>> http://invece.org >> "We are what we repeatedly do. Excellence, therefore, is not an act, >> but a habit." -- Aristotele
> -- > You received this message because you are subscribed to the Google Groups "Redis DB" group. > To post to this group, send email to redis-db@googlegroups.com. > To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
-- Salvatore 'antirez' Sanfilippo open source developer - VMware
http://invece.org "We are what we repeatedly do. Excellence, therefore, is not an act, but a habit." -- Aristotele
Thanks for your quick response and explanation. I had thought about
your solution before posting the request, but that didn't solve my
problem. My problem doesn't stem from decoding the message type, my
messages are serialized python dicts with sufficient information held
inside. Probably, I didn't explain my need clearly or haven't
understood something properly. I will try to explain again.
My usecase is a single producer (redis list) multiple consumer model:
3 worker processes, P0/P1/P2 feeding on a single list 'job_queue' (all
processes can run any kind of job that job_queue provides). Each
process performs: 'BRPOPLPUSH job_queue job_track 0'.
In this case, the problem I face is when I want to explicitly push a
command to a particular process (this command is meant for gathering
statistics and other information from the context of that particular
process). Let's look at an example: I want to gather info regarding
the process P1, so I lpush the respective command. What can happen is
that P1 is actually busy on a lengthy job. So, either P0 or P2
responds to it. I backoff for a random time and repush the same
command again, expecting to hit P1. But in the meantime, the
application on the client side had pushed other commands, and P1 is
again busy with it's execution.
To, solve this, I thought of changing it as: 'BRPOPLPUSH admin_queue_1
job_queue 0'. NOTE that admin_queue is ahead of job_queue, as the keys
are checked in the order provided. And there are 3 admin_queues here,
for 3 processes. Now I can lpush the command directly to P1's
admin_queue_1 and it replies back to me with the first lpush of the
command.
Plese feel free to let me know if I went wrong somewhere or missing
something. As I understand your solution, it will not be able to
address this case.
Regards,
Ethan.
On Sep 25, 4:03 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> <collins.eth...@gmail.com> wrote:
> > Well, adding Lua scripts is a method to add a layer of intelligent
> > logic using redis primitives. In this case, I think Lua script is a
> > workaround, as Pedro mentions.
> Yes I was replying to Pedro about the EVAL/EVALSHA thing. Sure,
> scripting is a way to model things without having to implement them
> and is a workaround, but in that case it is not a good workaround
> since blocking operations are not callable
> within scripts.
> But even if it scripting could solve very well that problem, I think
> this does not really answer your question about why BRPOPLPUSH can
> wait for just one list, while BLPOP/BRPOP is able to wait for multiple
> lists.
> There are mainly two reasons:
> 1) There is no obvious semantic for BRPOPLPUSH listening to more than
> a single list. I bet that if you implement it as "wait for N lists,
> push against one" other users will complain that we need a "wait for N
> lists, push against N lists", such as:
> BLPOPRPUSH queue1 queue2 track1 track2
> But even if we implement the above, is not a generalization of your
> use case pushing against a single list.
> So basically we are going to make the command either still too simple
> to cover all the cases, or bloated.
> 2) In the "push against single list" information, how to store the
> information about the originating list in the target list?
> And if you can't have that, why it is different than putting that info
> inside the list element pushed itself? And this brings us to the point
> three, that is, you can do that already without making Redis more
> complex.
> 3) Your use case is perfectly addressable just adding the information
> about the message type in the data itself.
> You want to do:
> BRPOPLPUSH job_queue admin_queue tracking_list
> So whatever the push is against the queue or the admin list you are
> going to pollute the tracking list with admin messages.
> So you can do that already as simple as:
> BRPOPLPUSH job_and_admin_list tracking_list
> And when you have to push simply use a single byte as a message prefix
> that can be "A" for admin messages, or "J" for jobs.
> Thanks for the discussion as it is interesting to review old design
> decisions from time to time, but I'm still pretty sure that the
> current simple approach is the best for us.
> Salvatore
> > From an API standpoint as well, I think the signature of command
> > BRPOPLPUSH should be 'symmetrical' with other similar commands, like
> > BRPOP/BLPOP. That is also the reason I feel this BRPOPLPUSH command
> > needs the update to support multiple source lists.
> > @Salvatore: What is your opinion about this update? If you agree and
> > if it's simple, is it possible to add it quickly?
> > Regards,
> > Ethan.
> > On Sep 24, 8:47 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> >> On Sat, Sep 24, 2011 at 12:56 PM, Pedro Melo <m...@simplicidade.org> wrote:
> >> > Salvatore: I know from past threads about scripting that the idea is
> >> > to send the entire Lua script every time I need to execute it. Are
> >> > there any plans to add an alternative method by which I could add new
> >> > Redis commands that would be implemented in Lua?
> >> Hello Pedro,
> >> we implemented EVALSHA for this (bandwidth) issue. Basically you send
> >> EVALSHA <sha1 of your script>. If the script was already seen in that
> >> server it gets executed, otherwise you get an error back and you'll
> >> instead use EVAL for this time. The next time the EVALSHA will be
> >> accepted.
> >> Since EVALSHA will usually work you just send EVALSHA the first time,
> >> so there is no additional round trip cost.
> >> This is as good as defining commands from the point of view of
> >> performances, but with the huge benefit that you don't need to make
> >> sure your program is calling a Redis instance with the right version
> >> of a command you implemented server-side and so forth. Also if you
> >> have a cluster of Redis instances it is very handy to consider every
> >> instance as every other regardless of defined script versions and so
> >> forth.
> >>http://invece.org > >> "We are what we repeatedly do. Excellence, therefore, is not an act,
> >> but a habit." -- Aristotele
> > --
> > You received this message because you are subscribed to the Google Groups "Redis DB" group.
> > To post to this group, send email to redis-db@googlegroups.com.
> > To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com.
> > For more options, visit this group athttp://groups.google.com/group/redis-db?hl=en.
I would like to chime in on this conversation to say that I support
the idea of the change to BRPOPLPUSH that Ethan is suggesting. Like
Ethan, I think that supporting multiple source keys would make the
command conceptually simpler because it would work in same way as
BRPOP, which is after all part of the name of the command. Based on
the name I would expect the command to work like a BRPOP which then
feeds into an LPUSH, but that is not currently the case.
I can offer my own use case if that's helpful. I'm implementing some
task queue functionality. A simple use case is that we have four
lists. Three of them represent different priorities, and one of them
is to store details of tasks which are currently in-progress. Let's
call our lists "high-priority", "medium-priority", "low-priority", and
"running".
The contents of the lists are tasks, and the information stored about
the task says nothing about its priority. That information is
represented only in the location of the task in the lists. When a
worker grabs a task to start working on it, I want to select the
oldest task from the high-priority list, the oldest task from the
medium-priority list, or the oldest task from the low-priority list,
whichever is first available. And, I want to simultaneously move this
task into the running queue in an atomic fashion.
One nice point about changing the command to work this way is that in
addition to making it consistent with BRPOP, it would be backwards
compatible with current uses of the command.
Is there a way to accomplish this use case using the current Redis
primitives that I'm missing? I don't think that putting information
about priority into the tasks themselves (collapsing the priority
lists into a single list) helps, because there is no efficient way to
use that information to sort the items in the queue so that high
priority items are selected first.
Thanks,
David Brewer
On Sep 25, 8:23 am, Ethan Collins <collins.eth...@gmail.com> wrote:
> Thanks for your quick response and explanation. I had thought about
> your solution before posting the request, but that didn't solve my
> problem. My problem doesn't stem from decoding the message type, my
> messages are serialized python dicts with sufficient information held
> inside. Probably, I didn't explain my need clearly or haven't
> understood something properly. I will try to explain again.
> My usecase is a single producer (redis list) multiple consumer model:
> 3 worker processes, P0/P1/P2 feeding on a single list 'job_queue' (all
> processes can run any kind of job that job_queue provides). Each
> process performs: 'BRPOPLPUSH job_queue job_track 0'.
> In this case, the problem I face is when I want to explicitly push a
> command to a particular process (this command is meant for gathering
> statistics and other information from the context of that particular
> process). Let's look at an example: I want to gather info regarding
> the process P1, so I lpush the respective command. What can happen is
> that P1 is actually busy on a lengthy job. So, either P0 or P2
> responds to it. I backoff for a random time and repush the same
> command again, expecting to hit P1. But in the meantime, the
> application on the client side had pushed other commands, and P1 is
> again busy with it's execution.
> To, solve this, I thought of changing it as: 'BRPOPLPUSH admin_queue_1
> job_queue 0'. NOTE that admin_queue is ahead of job_queue, as the keys
> are checked in the order provided. And there are 3 admin_queues here,
> for 3 processes. Now I can lpush the command directly to P1's
> admin_queue_1 and it replies back to me with the first lpush of the
> command.
> Plese feel free to let me know if I went wrong somewhere or missing
> something. As I understand your solution, it will not be able to
> address this case.
> Regards,
> Ethan.
> On Sep 25, 4:03 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> > Hello Ethan
> > On Sun, Sep 25, 2011 at 11:48 AM, Ethan Collins
> > <collins.eth...@gmail.com> wrote:
> > > Well, adding Lua scripts is a method to add a layer of intelligent
> > > logic using redis primitives. In this case, I think Lua script is a
> > > workaround, as Pedro mentions.
> > Yes I was replying to Pedro about the EVAL/EVALSHA thing. Sure,
> > scripting is a way to model things without having to implement them
> > and is a workaround, but in that case it is not a good workaround
> > since blocking operations are not callable
> > within scripts.
> > But even if it scripting could solve very well that problem, I think
> > this does not really answer your question about why BRPOPLPUSH can
> > wait for just one list, while BLPOP/BRPOP is able to wait for multiple
> > lists.
> > There are mainly two reasons:
> > 1) There is no obvious semantic for BRPOPLPUSH listening to more than
> > a single list. I bet that if you implement it as "wait for N lists,
> > push against one" other users will complain that we need a "wait for N
> > lists, push against N lists", such as:
> > BLPOPRPUSH queue1 queue2 track1 track2
> > But even if we implement the above, is not a generalization of your
> > use case pushing against a single list.
> > So basically we are going to make the command either still too simple
> > to cover all the cases, or bloated.
> > 2) In the "push against single list" information, how to store the
> > information about the originating list in the target list?
> > And if you can't have that, why it is different than putting that info
> > inside the list element pushed itself? And this brings us to the point
> > three, that is, you can do that already without making Redis more
> > complex.
> > 3) Your use case is perfectly addressable just adding the information
> > about the message type in the data itself.
> > So whatever the push is against the queue or the admin list you are
> > going to pollute the tracking list with admin messages.
> > So you can do that already as simple as:
> > BRPOPLPUSH job_and_admin_list tracking_list
> > And when you have to push simply use a single byte as a message prefix
> > that can be "A" for admin messages, or "J" for jobs.
> > Thanks for the discussion as it is interesting to review old design
> > decisions from time to time, but I'm still pretty sure that the
> > current simple approach is the best for us.
> > Salvatore
> > > From an API standpoint as well, I think the signature of command
> > > BRPOPLPUSH should be 'symmetrical' with other similar commands, like
> > > BRPOP/BLPOP. That is also the reason I feel this BRPOPLPUSH command
> > > needs the update to support multiple source lists.
> > > @Salvatore: What is your opinion about this update? If you agree and
> > > if it's simple, is it possible to add it quickly?
> > > Regards,
> > > Ethan.
> > > On Sep 24, 8:47 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> > >> On Sat, Sep 24, 2011 at 12:56 PM, Pedro Melo <m...@simplicidade.org> wrote:
> > >> > Salvatore: I know from past threads about scripting that the idea is
> > >> > to send the entire Lua script every time I need to execute it. Are
> > >> > there any plans to add an alternative method by which I could add new
> > >> > Redis commands that would be implemented in Lua?
> > >> Hello Pedro,
> > >> we implemented EVALSHA for this (bandwidth) issue. Basically you send
> > >> EVALSHA <sha1 of your script>. If the script was already seen in that
> > >> server it gets executed, otherwise you get an error back and you'll
> > >> instead use EVAL for this time. The next time the EVALSHA will be
> > >> accepted.
> > >> Since EVALSHA will usually work you just send EVALSHA the first time,
> > >> so there is no additional round trip cost.
> > >> This is as good as defining commands from the point of view of
> > >> performances, but with the huge benefit that you don't need to make
> > >> sure your program is calling a Redis instance with the right version
> > >> of a command you implemented server-side and so forth. Also if you
> > >> have a cluster of Redis instances it is very handy to consider every
> > >> instance as every other regardless of defined script versions and so
> > >> forth.
> > >>http://invece.org > > >> "We are what we repeatedly do. Excellence, therefore, is not an act,
> > >> but a habit." -- Aristotele
> > > --
> > > You received this message because you are subscribed to the Google Groups "Redis DB" group.
> > > To post to this group, send email to redis-db@googlegroups.com.
> > > To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com.
> > > For more options, visit this group athttp://groups.google.com/group/redis-db?hl=en.
I too would like to see BRPOPLPUSH support multiple sources, but with
multiple destinations, like
BRPOPLPUSH src1 dst1 src2 dst2 ...
My use case is as follows:
Suppose I use Redis as a Q server with Lists for Q's, each Q has a
name and is basically a Redis List storead against the name as Key.
For the Q client I would like to use the safe queues pattern mentioned
in the documentation for the command RPOPLPUSH.
Now, if the client is to "listen" to multiple Q's on the same Redis
server, I would require a socket connection for each Q because
BRPOPLPUSH does not support multiple Keys.
If it were to support multiple Keys I could have a dedicated socket
connection and start a BRPOPLPUSH command with all the Q's. Once I
Dequeue an element. I would cyclically rotate the list of keys to
ensure fairness. Of course the target key of the dequeued item would
be cleared once the message is processed successfully.
Suresh
On Oct 7, 4:22 am, David Brewer <david.bre...@gmail.com> wrote:
> I would like to chime in on this conversation to say that I support
> the idea of the change toBRPOPLPUSHthat Ethan is suggesting. Like
> Ethan, I think that supporting multiple source keys would make the
> command conceptually simpler because it would work in same way as
> BRPOP, which is after all part of the name of the command. Based on
> the name I would expect the command to work like a BRPOP which then
> feeds into an LPUSH, but that is not currently the case.
> I can offer my own use case if that's helpful. I'm implementing some
> task queue functionality. A simple use case is that we have four
> lists. Three of them represent different priorities, and one of them
> is to store details of tasks which are currently in-progress. Let's
> call our lists "high-priority", "medium-priority", "low-priority", and
> "running".
> The contents of the lists are tasks, and the information stored about
> the task says nothing about its priority. That information is
> represented only in the location of the task in the lists. When a
> worker grabs a task to start working on it, I want to select the
> oldest task from the high-priority list, the oldest task from the
> medium-priority list, or the oldest task from the low-priority list,
> whichever is first available. And, I want to simultaneously move this
> task into the running queue in an atomic fashion.
> One nice point about changing the command to work this way is that in
> addition to making it consistent with BRPOP, it would be backwards
> compatible with current uses of the command.
> Is there a way to accomplish this use case using the current Redis
> primitives that I'm missing? I don't think that putting information
> about priority into the tasks themselves (collapsing the priority
> lists into a single list) helps, because there is no efficient way to
> use that information to sort the items in the queue so that high
> priority items are selected first.
> Thanks,
> David Brewer
> On Sep 25, 8:23 am, Ethan Collins <collins.eth...@gmail.com> wrote:
> > Hi Salvatore,
> > Thanks for your quick response and explanation. I had thought about
> > your solution before posting the request, but that didn't solve my
> > problem. My problem doesn't stem from decoding the message type, my
> > messages are serialized python dicts with sufficient information held
> > inside. Probably, I didn't explain my need clearly or haven't
> > understood something properly. I will try to explain again.
> > My usecase is a single producer (redis list) multiple consumer model:
> > 3 worker processes, P0/P1/P2 feeding on a single list 'job_queue' (all
> > processes can run any kind of job that job_queue provides). Each
> > process performs: 'BRPOPLPUSHjob_queue job_track 0'.
> > In this case, the problem I face is when I want to explicitly push a
> > command to a particular process (this command is meant for gathering
> > statistics and other information from the context of that particular
> > process). Let's look at an example: I want to gather info regarding
> > the process P1, so I lpush the respective command. What can happen is
> > that P1 is actually busy on a lengthy job. So, either P0 or P2
> > responds to it. I backoff for a random time and repush the same
> > command again, expecting to hit P1. But in the meantime, the
> > application on the client side had pushed other commands, and P1 is
> > again busy with it's execution.
> > To, solve this, I thought of changing it as: 'BRPOPLPUSHadmin_queue_1
> > job_queue 0'. NOTE that admin_queue is ahead of job_queue, as the keys
> > are checked in the order provided. And there are 3 admin_queues here,
> > for 3 processes. Now I can lpush the command directly to P1's
> > admin_queue_1 and it replies back to me with the first lpush of the
> > command.
> > Plese feel free to let me know if I went wrong somewhere or missing
> > something. As I understand your solution, it will not be able to
> > address this case.
> > > On Sun, Sep 25, 2011 at 11:48 AM, Ethan Collins
> > > <collins.eth...@gmail.com> wrote:
> > > > Well, adding Lua scripts is a method to add a layer of intelligent
> > > > logic using redis primitives. In this case, I think Lua script is a
> > > > workaround, as Pedro mentions.
> > > Yes I was replying to Pedro about the EVAL/EVALSHA thing. Sure,
> > > scripting is a way to model things without having to implement them
> > > and is a workaround, but in that case it is not a good workaround
> > > since blocking operations are not callable
> > > within scripts.
> > > But even if it scripting could solve very well that problem, I think
> > > this does not really answer your question about whyBRPOPLPUSHcan
> > > wait for just one list, while BLPOP/BRPOP is able to wait for multiple
> > > lists.
> > > There are mainly two reasons:
> > > 1) There is no obvious semantic forBRPOPLPUSHlistening to more than
> > > a single list. I bet that if you implement it as "wait for N lists,
> > > push against one" other users will complain that we need a "wait for N
> > > lists, push against N lists", such as:
> > > BLPOPRPUSH queue1 queue2 track1 track2
> > > But even if we implement the above, is not a generalization of your
> > > use case pushing against a single list.
> > > So basically we are going to make the command either still too simple
> > > to cover all the cases, or bloated.
> > > 2) In the "push against single list" information, how to store the
> > > information about the originating list in the target list?
> > > And if you can't have that, why it is different than putting that info
> > > inside the list element pushed itself? And this brings us to the point
> > > three, that is, you can do that already without making Redis more
> > > complex.
> > > 3) Your use case is perfectly addressable just adding the information
> > > about the message type in the data itself.
> > > So whatever the push is against the queue or the admin list you are
> > > going to pollute the tracking list with admin messages.
> > > So you can do that already as simple as:
> > >BRPOPLPUSHjob_and_admin_list tracking_list
> > > And when you have to push simply use a single byte as a message prefix
> > > that can be "A" for admin messages, or "J" for jobs.
> > > Thanks for the discussion as it is interesting to review old design
> > > decisions from time to time, but I'm still pretty sure that the
> > > current simple approach is the best for us.
> > > Salvatore
> > > > From an API standpoint as well, I think the signature of command
> > > >BRPOPLPUSHshould be 'symmetrical' with other similar commands, like
> > > > BRPOP/BLPOP. That is also the reason I feel thisBRPOPLPUSHcommand
> > > > needs the update to support multiple source lists.
> > > > @Salvatore: What is your opinion about this update? If you agree and
> > > > if it's simple, is it possible to add it quickly?
> > > > Regards,
> > > > Ethan.
> > > > On Sep 24, 8:47 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> > > >> On Sat, Sep 24, 2011 at 12:56 PM, Pedro Melo <m...@simplicidade.org> wrote:
> > > >> > Salvatore: I know from past threads about scripting that the idea is
> > > >> > to send the entire Lua script every time I need to execute it. Are
> > > >> > there any plans to add an alternative method by which I could add new
> > > >> > Redis commands that would be implemented in Lua?
> > > >> Hello Pedro,
> > > >> we implemented EVALSHA for this (bandwidth) issue. Basically you send
> > > >> EVALSHA <sha1 of your script>. If the script was already seen in that
> > > >> server it gets executed, otherwise you get an error back and you'll
> > > >> instead use EVAL for this time. The next time the EVALSHA will be
> > > >> accepted.
> > > >> Since EVALSHA will usually work you just send EVALSHA the first time,
> > > >> so there is no additional round trip cost.
> > > >> This is as good as defining commands from the point of view of
> > > >> performances, but with the huge benefit that you don't need to make
> > > >> sure your program is calling a Redis instance with the right version
> > > >> of a command you implemented server-side and so forth. Also if you
> > > >> have a cluster of Redis instances it is very handy to consider every
> > > >> instance as every other regardless of defined script versions and so
> > > >> forth.
On Mon, Oct 10, 2011 at 9:25 AM, Suresh <suresh.mahalin...@gmail.com> wrote: > I too would like to see BRPOPLPUSH support multiple sources, but with > multiple destinations, like
> BRPOPLPUSH src1 dst1 src2 dst2 ...
> My use case is as follows:
> Suppose I use Redis as a Q server with Lists for Q's, each Q has a > name and is basically a Redis List storead against the name as Key.
> For the Q client I would like to use the safe queues pattern mentioned > in the documentation for the command RPOPLPUSH.
> Now, if the client is to "listen" to multiple Q's on the same Redis > server, I would require a socket connection for each Q because > BRPOPLPUSH does not support multiple Keys.
Exactly. This solution so far solves all the requested use cases. Is it that bad to open multiple connections? Honest question, really.
Most of those connections will be mostly idle, the server uses epolll so the cost is pretty negligeble (I've personally seen a *Perl*-based XMPP server using epoll with a bit over 150k connections work just fine, used for a singalling/notification service)...
Isn't it simpler to just use multiple connections instead of making the command semantics harder?
Why is it that BRPOP takes multiple keys. Taking the consistency point
of view BRPOP need not accept multiple keys either.
The SUBSCRIBE command provided by Redis also allows listening for
multiple channels on a socket.
If we need to open a socket connection for each Q (List) in my
previous example we need one thread per List in Java which would mean
100 threads for 100 Qs. Of course there are ways to work around that
using Non Blocking IO or polling. But my point is having BRPOPLPUSH
support multiple sources and targets would make things easier for the
client.
Regards,
Suresh
On Oct 10, 1:40 pm, Pedro Melo <m...@simplicidade.org> wrote:
> On Mon, Oct 10, 2011 at 9:25 AM, Suresh <suresh.mahalin...@gmail.com> wrote:
> > I too would like to see BRPOPLPUSH support multiple sources, but with
> > multiple destinations, like
> > BRPOPLPUSH src1 dst1 src2 dst2 ...
> > My use case is as follows:
> > Suppose I use Redis as a Q server with Lists for Q's, each Q has a
> > name and is basically a Redis List storead against the name as Key.
> > For the Q client I would like to use the safe queues pattern mentioned
> > in the documentation for the command RPOPLPUSH.
> > Now, if the client is to "listen" to multiple Q's on the same Redis
> > server, I would require a socket connection for each Q because
> > BRPOPLPUSH does not support multiple Keys.
> Exactly. This solution so far solves all the requested use cases. Is
> it that bad to open multiple connections? Honest question, really.
> Most of those connections will be mostly idle, the server uses epolll
> so the cost is pretty negligeble (I've personally seen a *Perl*-based
> XMPP server using epoll with a bit over 150k connections work just
> fine, used for a singalling/notification service)...
> Isn't it simpler to just use multiple connections instead of making
> the command semantics harder?
On Mon, Oct 10, 2011 at 12:03 PM, Suresh <suresh.mahalin...@gmail.com> wrote: > Why is it that BRPOP takes multiple keys. Taking the consistency point > of view BRPOP need not accept multiple keys either.
I disagree with the consistency argument here. BRPOP is single direction: you remove stuff from lists, so the semantics is clear an obvious.
BRPOPLPUSH is a two step command: remove from one, push into the other. So I can think at least several semantics with multiple lists, and I would not be able to say which one would be the correct, obvious one:
* multiple sources, single destination; * single source, multiple destinations; * multiple sources, multiple destinations: * map one-to-one: like having the current BRPOPLPUSH in parallel; * any RPOPed element will be LPUSHed to all destinations.
Which one is the correct one?
So, no, there is no consistency argument to he had IMHO.
I would like to have a solution for multiple blocking actions in a single TCP connection, but for now I think the multiple TCP connections is the correct answer.
> The SUBSCRIBE command provided by Redis also allows listening for > multiple channels on a socket.
True, but still very different use case, not a consistency issue.
> If we need to open a socket connection for each Q (List) in my > previous example we need one thread per List in Java which would mean > 100 threads for 100 Qs.
err... I'm sure you don't. Use one of the event loops for Java where you can have a single thread serving all of your connection, I'm sure that is possible.
> Of course there are ways to work around that > using Non Blocking IO or polling.
Exactly.
> But my point is having BRPOPLPUSH > support multiple sources and targets would make things easier for the > client.
I don't disagree that a solution for blocking on several lists and acting on the output would be easier for many, including myself who use Redis as a task queue and had to work around this very same issue. I'm just pointing that there are several semantics for such solution, and personally I would not dare to say which one is the most obvious, or most powerful, or most useful.
On Oct 10, 4:21 pm, Pedro Melo <m...@simplicidade.org> wrote:
> Hi,
> On Mon, Oct 10, 2011 at 12:03 PM, Suresh <suresh.mahalin...@gmail.com> wrote:
> > Why is it that BRPOP takes multiple keys. Taking the consistency point
> > of view BRPOP need not accept multiple keys either.
> I disagree with the consistency argument here. BRPOP is single
> direction: you remove stuff from lists, so the semantics is clear an
> obvious.
> BRPOPLPUSH is a two step command: remove from one, push into the
> other. So I can think at least several semantics with multiple lists,
> and I would not be able to say which one would be the correct, obvious
> one:
I agree that there are multiple options possible when thinking of a
multiple source multiple destination command unlike BRPOP.
However to me BRPOPLPUSH src1 dst1 src2 dst2 ... looks like a natural
and useful extension to BRPOPLPUSH src dst. But opinions and use cases
do differ.
> * multiple sources, single destination;
Possible By repeating the same destination key.
> * single source, multiple destinations;
As I said the above does not look like an obvious extension of the
single source command.
> * multiple sources, multiple destinations:
> * map one-to-one: like having the current BRPOPLPUSH in parallel;
The above is the one I and another person in this thread were talking
about.
> * any RPOPed element will be LPUSHed to all destinations.
Again the above does not look like an extension of the single source
command to me.
...
> I'm just pointing that there are several semantics for such solution,
> and personally I would not dare to say which one is the most obvious,
> or most powerful, or most useful.
I too do not disagree that trying to extend BRPOPLPUSH to multiple
sources opens up multiple possibilities.
I am just pointing out the usefulness of this way of extending the
command which seems to be an obvious one to me.
It looks like this is not prioritized or not considered because of
multiple possibilities. At least in this thread I have seen only two
requirements:
1. What I mentioned
2. Multiple sources and a single destination which can be achieved by
repeating the destination key.
On Mon, Oct 10, 2011 at 4:21 AM, Pedro Melo <m...@simplicidade.org> wrote: > Hi,
> On Mon, Oct 10, 2011 at 12:03 PM, Suresh <suresh.mahalin...@gmail.com> wrote: >> Why is it that BRPOP takes multiple keys. Taking the consistency point >> of view BRPOP need not accept multiple keys either.
> I disagree with the consistency argument here. BRPOP is single > direction: you remove stuff from lists, so the semantics is clear an > obvious.
> BRPOPLPUSH is a two step command: remove from one, push into the > other. So I can think at least several semantics with multiple lists, > and I would not be able to say which one would be the correct, obvious > one:
> * multiple sources, single destination; > * single source, multiple destinations; > * multiple sources, multiple destinations: > * map one-to-one: like having the current BRPOPLPUSH in parallel; > * any RPOPed element will be LPUSHed to all destinations.
> Which one is the correct one?
I agree that there are multiple options here, all of which could be useful for different use cases. But, I disagree that it's so difficult to choose which one is the obvious case for an extension to BRPOPLPUSH.
To me it seems obvious that BRPOPLPUSH is semantically meant to be an atomic combination of BRPOP + LPUSH. So, it should behave like performing a BRPOP and then an LPUSH, except it should do it atomically. BRPOP supports popping an element from the first non-empty list argument, and therefore ideally BRPOPLPUSH would support that as well. LPUSH only supports pushing an element into one list, so BRPOPLPUSH should support only one destination. I suppose you could also parse the command as B(locking) RPOP + LPUSH -- which would describe the current behavior of the command. Any other behavior for this command would be very confusing.
The other use cases are useful, but it seems to me that they would be candidates for a new, different commands with different names.
> So, no, there is no consistency argument to he had IMHO.
> I would like to have a solution for multiple blocking actions in a > single TCP connection, but for now I think the multiple TCP > connections is the correct answer.
I don't think multiple TCP connections addresses my use case of trying to pick the first available task out of a series of lists (with each list representing a priority), and then placing it in a "running" list, in an atomic fashion. Currently I am choosing to use a call to BRPOP to pick the task, followed by a call to LPUSH to put in the running queue, and accepting the risk that if my process crashes at the wrong time it may not end up in the running queue. I don't have the option to use multiple calls to BRPOPLPUSH with its current behavior, because the command would block on the "high priority" list and I wouldn't see incoming "regular priority" or "low priority" tasks.
I originally thought scripting via lua was a bad design decision, but I've come around to it, and this is a perfect example of what it's for. You have a complex algorithm you'd like to execute atomically, and the existing command set doesn't seem suitable. You might give the scripting branch a try.
F.
On Mon, Oct 10, 2011 at 11:01 AM, David Brewer <david.bre...@gmail.com>wrote:
> On Mon, Oct 10, 2011 at 4:21 AM, Pedro Melo <m...@simplicidade.org> wrote: > > Hi,
> > On Mon, Oct 10, 2011 at 12:03 PM, Suresh <suresh.mahalin...@gmail.com> > wrote: > >> Why is it that BRPOP takes multiple keys. Taking the consistency point > >> of view BRPOP need not accept multiple keys either.
> > I disagree with the consistency argument here. BRPOP is single > > direction: you remove stuff from lists, so the semantics is clear an > > obvious.
> > BRPOPLPUSH is a two step command: remove from one, push into the > > other. So I can think at least several semantics with multiple lists, > > and I would not be able to say which one would be the correct, obvious > > one:
> > * multiple sources, single destination; > > * single source, multiple destinations; > > * multiple sources, multiple destinations: > > * map one-to-one: like having the current BRPOPLPUSH in parallel; > > * any RPOPed element will be LPUSHed to all destinations.
> > Which one is the correct one?
> I agree that there are multiple options here, all of which could be > useful for different use cases. But, I disagree that it's so > difficult to choose which one is the obvious case for an extension to > BRPOPLPUSH.
> To me it seems obvious that BRPOPLPUSH is semantically meant to be an > atomic combination of BRPOP + LPUSH. So, it should behave like > performing a BRPOP and then an LPUSH, except it should do it > atomically. BRPOP supports popping an element from the first > non-empty list argument, and therefore ideally BRPOPLPUSH would > support that as well. LPUSH only supports pushing an element into one > list, so BRPOPLPUSH should support only one destination. I suppose you > could also parse the command as B(locking) RPOP + LPUSH -- which would > describe the current behavior of the command. Any other behavior for > this command would be very confusing.
> The other use cases are useful, but it seems to me that they would be > candidates for a new, different commands with different names.
> > So, no, there is no consistency argument to he had IMHO.
> > I would like to have a solution for multiple blocking actions in a > > single TCP connection, but for now I think the multiple TCP > > connections is the correct answer.
> I don't think multiple TCP connections addresses my use case of trying > to pick the first available task out of a series of lists (with each > list representing a priority), and then placing it in a "running" > list, in an atomic fashion. Currently I am choosing to use a call to > BRPOP to pick the task, followed by a call to LPUSH to put in the > running queue, and accepting the risk that if my process crashes at > the wrong time it may not end up in the running queue. I don't have > the option to use multiple calls to BRPOPLPUSH with its current > behavior, because the command would block on the "high priority" list > and I wouldn't see incoming "regular priority" or "low priority" > tasks.
> Thanks,
> David Brewer
> -- > You received this message because you are subscribed to the Google Groups > "Redis DB" group. > To post to this group, send email to redis-db@googlegroups.com. > To unsubscribe from this group, send email to > redis-db+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/redis-db?hl=en.
On Mon, Oct 10, 2011 at 7:13 PM, Felix Gallo <felixga...@gmail.com> wrote: > I originally thought scripting via lua was a bad design decision, but I've > come around to it, and this is a perfect example of what it's for. You have > a complex algorithm you'd like to execute atomically, and the existing > command set doesn't seem suitable. You might give the scripting branch a > try.
Not yet, if at all... Last I heard from Salvatore, Lua scripts cannot block or call blocking Redis commands.
On Mon, Oct 10, 2011 at 12:35 PM, Pedro Melo <m...@simplicidade.org> wrote: > Hi,
> On Mon, Oct 10, 2011 at 7:13 PM, Felix Gallo <felixga...@gmail.com> wrote: >> I originally thought scripting via lua was a bad design decision, but I've >> come around to it, and this is a perfect example of what it's for. You have >> a complex algorithm you'd like to execute atomically, and the existing >> command set doesn't seem suitable. You might give the scripting branch a >> try.
> Not yet, if at all... Last I heard from Salvatore, Lua scripts cannot > block or call blocking Redis commands.
... well, they block, but they block *everything* until they are done, which wouldn't help in this situation.
On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
> When multiple source lists are given, BRPOPLPUSH can push them to a > single destination and that should be fine with many requirements.
> I faced this need while wrting up a redis based job queue. I have > multiple workers linked with a single redis list, N:1 relation (with N > ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > 0'. I now have the necessity to talk to any particular worker process, > especially for status, statistics etc purposes. For this, the best way > is to add 1 admin list for each of the worker processes (1:1). This > makes the command look like 'BRPOPLPUSH worker_N_admin job_list > job_track 0', expecting it to handle the source lists in the same > manner as BRPOP does.
> Is this something that can be supported easily? Any other better way > of doing this?
> Regards, > Ethan. On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
> When multiple source lists are given, BRPOPLPUSH can push them to a > single destination and that should be fine with many requirements.
> I faced this need while wrting up a redis based job queue. I have > multiple workers linked with a single redis list, N:1 relation (with N > ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > 0'. I now have the necessity to talk to any particular worker process, > especially for status, statistics etc purposes. For this, the best way > is to add 1 admin list for each of the worker processes (1:1). This > makes the command look like 'BRPOPLPUSH worker_N_admin job_list > job_track 0', expecting it to handle the source lists in the same > manner as BRPOP does.
> Is this something that can be supported easily? Any other better way > of doing this?
> Regards, > Ethan. On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
> When multiple source lists are given, BRPOPLPUSH can push them to a > single destination and that should be fine with many requirements.
> I faced this need while wrting up a redis based job queue. I have > multiple workers linked with a single redis list, N:1 relation (with N > ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > 0'. I now have the necessity to talk to any particular worker process, > especially for status, statistics etc purposes. For this, the best way > is to add 1 admin list for each of the worker processes (1:1). This > makes the command look like 'BRPOPLPUSH worker_N_admin job_list > job_track 0', expecting it to handle the source lists in the same > manner as BRPOP does.
> Is this something that can be supported easily? Any other better way > of doing this?
> Regards, > Ethan. On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
> When multiple source lists are given, BRPOPLPUSH can push them to a > single destination and that should be fine with many requirements.
> I faced this need while wrting up a redis based job queue. I have > multiple workers linked with a single redis list, N:1 relation (with N > ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > 0'. I now have the necessity to talk to any particular worker process, > especially for status, statistics etc purposes. For this, the best way > is to add 1 admin list for each of the worker processes (1:1). This > makes the command look like 'BRPOPLPUSH worker_N_admin job_list > job_track 0', expecting it to handle the source lists in the same > manner as BRPOP does.
> Is this something that can be supported easily? Any other better way > of doing this?
> Regards, > Ethan. On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
> When multiple source lists are given, BRPOPLPUSH can push them to a > single destination and that should be fine with many requirements.
> I faced this need while wrting up a redis based job queue. I have > multiple workers linked with a single redis list, N:1 relation (with N > ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > 0'. I now have the necessity to talk to any particular worker process, > especially for status, statistics etc purposes. For this, the best way > is to add 1 admin list for each of the worker processes (1:1). This > makes the command look like 'BRPOPLPUSH worker_N_admin job_list > job_track 0', expecting it to handle the source lists in the same > manner as BRPOP does.
> Is this something that can be supported easily? Any other better way > of doing this?
> Regards, > Ethan. On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
> When multiple source lists are given, BRPOPLPUSH can push them to a > single destination and that should be fine with many requirements.
> I faced this need while wrting up a redis based job queue. I have > multiple workers linked with a single redis list, N:1 relation (with N > ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > 0'. I now have the necessity to talk to any particular worker process, > especially for status, statistics etc purposes. For this, the best way > is to add 1 admin list for each of the worker processes (1:1). This > makes the command look like 'BRPOPLPUSH worker_N_admin job_list > job_track 0', expecting it to handle the source lists in the same > manner as BRPOP does.
> Is this something that can be supported easily? Any other better way > of doing this?
> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>> When multiple source lists are given, BRPOPLPUSH can push them to a >> single destination and that should be fine with many requirements.
>> I faced this need while wrting up a redis based job queue. I have >> multiple workers linked with a single redis list, N:1 relation (with N >> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >> 0'. I now have the necessity to talk to any particular worker process, >> especially for status, statistics etc purposes. For this, the best way >> is to add 1 admin list for each of the worker processes (1:1). This >> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >> job_track 0', expecting it to handle the source lists in the same >> manner as BRPOP does.
>> Is this something that can be supported easily? Any other better way >> of doing this?
>> Regards, >> Ethan.
> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>> When multiple source lists are given, BRPOPLPUSH can push them to a >> single destination and that should be fine with many requirements.
>> I faced this need while wrting up a redis based job queue. I have >> multiple workers linked with a single redis list, N:1 relation (with N >> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >> 0'. I now have the necessity to talk to any particular worker process, >> especially for status, statistics etc purposes. For this, the best way >> is to add 1 admin list for each of the worker processes (1:1). This >> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >> job_track 0', expecting it to handle the source lists in the same >> manner as BRPOP does.
>> Is this something that can be supported easily? Any other better way >> of doing this?
>> Regards, >> Ethan.
> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>> When multiple source lists are given, BRPOPLPUSH can push them to a >> single destination and that should be fine with many requirements.
>> I faced this need while wrting up a redis based job queue. I have >> multiple workers linked with a single redis list, N:1 relation (with N >> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >> 0'. I now have the necessity to talk to any particular worker process, >> especially for status, statistics etc purposes. For this, the best way >> is to add 1 admin list for each of the worker processes (1:1). This >> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >> job_track 0', expecting it to handle the source lists in the same >> manner as BRPOP does.
>> Is this something that can be supported easily? Any other better way >> of doing this?
>> Regards, >> Ethan.
> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>> When multiple source lists are given, BRPOPLPUSH can push them to a >> single destination and that should be fine with many requirements.
>> I faced this need while wrting up a redis based job queue. I have >> multiple workers linked with a single redis list, N:1 relation (with N >> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >> 0'. I now have the necessity to talk to any particular worker process, >> especially for status, statistics etc purposes. For this, the best way >> is to add 1 admin list for each of the worker processes (1:1). This >> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >> job_track 0', expecting it to handle the source lists in the same >> manner as BRPOP does.
>> Is this something that can be supported easily? Any other better way >> of doing this?
>> Regards, >> Ethan.
> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>> When multiple source lists are given, BRPOPLPUSH can push them to a >> single destination and that should be fine with many requirements.
>> I faced this need while wrting up a redis based job queue. I have >> multiple workers linked with a single redis list, N:1 relation (with N >> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >> 0'. I now have the necessity to talk to any particular worker process, >> especially for status, statistics etc purposes. For this, the best way >> is to add 1 admin list for each of the worker processes (1:1). This >> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >> job_track 0', expecting it to handle the source lists in the same >> manner as BRPOP does.
>> Is this something that can be supported easily? Any other better way >> of doing this?
>> Regards, >> Ethan.
> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>> When multiple source lists are given, BRPOPLPUSH can push them to a >> single destination and that should be fine with many requirements.
>> I faced this need while wrting up a redis based job queue. I have >> multiple workers linked with a single redis list, N:1 relation (with N >> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >> 0'. I now have the necessity to talk to any particular worker process, >> especially for status, statistics etc purposes. For this, the best way >> is to add 1 admin list for each of the worker processes (1:1). This >> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >> job_track 0', expecting it to handle the source lists in the same >> manner as BRPOP does.
>> Is this something that can be supported easily? Any other better way >> of doing this?
>> Regards, >> Ethan.
I would like to add another vote for extending BRPOPLPUSH to support multiple lists. I think it is consistent with the existing behavior of BRPOP and would make implementing prioritized job queues a snap, unlike the use of multiple connections.
This is not going to be supports. To echo what Salvatore already said in the pull request:
Where the regular blocking pop is unidirectional, BRPOPLPUSH is bidirectional: it pops from a list and pushed onto another list. If you look at the return value of the blocking pop, you'll see that it both returns the list the value was popped from together with the value itself. There is no way to preserve this information for BRPOPLPUSH. A value is popped from one list and pushed to another. If you think about a BRPOPLPUSH that pops from multiple lists to finally push the value to another list, that information is lost. So, when that information is lost anyway, why not push those values to a single list to begin with?
>> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>>> When multiple source lists are given, BRPOPLPUSH can push them to a >>> single destination and that should be fine with many requirements.
>>> I faced this need while wrting up a redis based job queue. I have >>> multiple workers linked with a single redis list, N:1 relation (with N >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >>> 0'. I now have the necessity to talk to any particular worker process, >>> especially for status, statistics etc purposes. For this, the best way >>> is to add 1 admin list for each of the worker processes (1:1). This >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >>> job_track 0', expecting it to handle the source lists in the same >>> manner as BRPOP does.
>>> Is this something that can be supported easily? Any other better way >>> of doing this?
>>> Regards, >>> Ethan.
>> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>>> When multiple source lists are given, BRPOPLPUSH can push them to a >>> single destination and that should be fine with many requirements.
>>> I faced this need while wrting up a redis based job queue. I have >>> multiple workers linked with a single redis list, N:1 relation (with N >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >>> 0'. I now have the necessity to talk to any particular worker process, >>> especially for status, statistics etc purposes. For this, the best way >>> is to add 1 admin list for each of the worker processes (1:1). This >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >>> job_track 0', expecting it to handle the source lists in the same >>> manner as BRPOP does.
>>> Is this something that can be supported easily? Any other better way >>> of doing this?
>>> Regards, >>> Ethan.
>> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>>> When multiple source lists are given, BRPOPLPUSH can push them to a >>> single destination and that should be fine with many requirements.
>>> I faced this need while wrting up a redis based job queue. I have >>> multiple workers linked with a single redis list, N:1 relation (with N >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >>> 0'. I now have the necessity to talk to any particular worker process, >>> especially for status, statistics etc purposes. For this, the best way >>> is to add 1 admin list for each of the worker processes (1:1). This >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >>> job_track 0', expecting it to handle the source lists in the same >>> manner as BRPOP does.
>>> Is this something that can be supported easily? Any other better way >>> of doing this?
>>> Regards, >>> Ethan.
>> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>>> When multiple source lists are given, BRPOPLPUSH can push them to a >>> single destination and that should be fine with many requirements.
>>> I faced this need while wrting up a redis based job queue. I have >>> multiple workers linked with a single redis list, N:1 relation (with N >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >>> 0'. I now have the necessity to talk to any particular worker process, >>> especially for status, statistics etc purposes. For this, the best way >>> is to add 1 admin list for each of the worker processes (1:1). This >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >>> job_track 0', expecting it to handle the source lists in the same >>> manner as BRPOP does.
>>> Is this something that can be supported easily? Any other better way >>> of doing this?
>>> Regards, >>> Ethan.
>> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>>> When multiple source lists are given, BRPOPLPUSH can push them to a >>> single destination and that should be fine with many requirements.
>>> I faced this need while wrting up a redis based job queue. I have >>> multiple workers linked with a single redis list, N:1 relation (with N >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >>> 0'. I now have the necessity to talk to any particular worker process, >>> especially for status, statistics etc purposes. For this, the best way >>> is to add 1 admin list for each of the worker processes (1:1). This >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >>> job_track 0', expecting it to handle the source lists in the same >>> manner as BRPOP does.
>>> Is this something that can be supported easily? Any other better way >>> of doing this?
>>> Regards, >>> Ethan.
>> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>>> When multiple source lists are given, BRPOPLPUSH can push them to a >>> single destination and that should be fine with many requirements.
>>> I faced this need while wrting up a redis based job queue. I have >>> multiple workers linked with a single redis list, N:1 relation (with N >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >>> 0'. I now have the necessity to talk to any particular worker process, >>> especially for status, statistics etc purposes. For this, the best way >>> is to add 1 admin list for each of the worker processes (1:1). This >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >>> job_track 0', expecting it to handle the source lists in the same >>> manner as BRPOP does.
>>> Is this something that can be supported easily? Any other better way >>> of doing this?
>>> Regards, >>> Ethan.
> I would like to add another vote for extending BRPOPLPUSH to support > multiple lists. I think it is consistent with the existing behavior of BRPOP > and would make implementing prioritized job queues a snap, unlike the > use of multiple connections.
> thanks, > dave
> -- > You received this message because you are subscribed to the Google Groups > "Redis DB" group. > To post to this group, send email to redis-db@googlegroups.com. > To unsubscribe from this group, send email to > redis-db+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/redis-db?hl=en.
> This is not going to be supports. To echo what Salvatore already said > in the pull request:
> Where the regular blocking pop is unidirectional, BRPOPLPUSH is > bidirectional: it pops from a list and pushed onto another list. If > you look at the return value of the blocking pop, you'll see that it > both returns the list the value was popped from together with the > value itself. There is no way to preserve this information for > BRPOPLPUSH. A value is popped from one list and pushed to another. If > you think about a BRPOPLPUSH that pops from multiple lists to finally > push the value to another list, that information is lost. So, when > that information is lost anyway, why not push those values to a single > list to begin with?
Well, for a collection of job queues with multiple priorities, the simple reason is that you need the information before you do the pop (the information is the order of the lists you give to the command, i.e., the relative queue priorities) and you don't need it afterwards, at least you can live without it.
But since BRPOPLPUSH does not currently support this, surely if the command is extended to support multiple keys, the reply value could be extended for the multiple key case to also return the list from which the value was popped just like BRPOP? Hence, no information loss at all and backwards compatibility with the single key case.
> >> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
> >>> When multiple source lists are given, BRPOPLPUSH can push them to a > >>> single destination and that should be fine with many requirements.
> >>> I faced this need while wrting up a redis based job queue. I have > >>> multiple workers linked with a single redis list, N:1 relation (with N > >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > >>> 0'. I now have the necessity to talk to any particular worker process, > >>> especially for status, statistics etc purposes. For this, the best way > >>> is to add 1 admin list for each of the worker processes (1:1). This > >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list > >>> job_track 0', expecting it to handle the source lists in the same > >>> manner as BRPOP does.
> >>> Is this something that can be supported easily? Any other better way > >>> of doing this?
> >>> Regards, > >>> Ethan.
> >> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
> >>> When multiple source lists are given, BRPOPLPUSH can push them to a > >>> single destination and that should be fine with many requirements.
> >>> I faced this need while wrting up a redis based job queue. I have > >>> multiple workers linked with a single redis list, N:1 relation (with N > >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > >>> 0'. I now have the necessity to talk to any particular worker process, > >>> especially for status, statistics etc purposes. For this, the best way > >>> is to add 1 admin list for each of the worker processes (1:1). This > >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list > >>> job_track 0', expecting it to handle the source lists in the same > >>> manner as BRPOP does.
> >>> Is this something that can be supported easily? Any other better way > >>> of doing this?
> >>> Regards, > >>> Ethan.
> >> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
> >>> When multiple source lists are given, BRPOPLPUSH can push them to a > >>> single destination and that should be fine with many requirements.
> >>> I faced this need while wrting up a redis based job queue. I have > >>> multiple workers linked with a single redis list, N:1 relation (with N > >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > >>> 0'. I now have the necessity to talk to any particular worker process, > >>> especially for status, statistics etc purposes. For this, the best way > >>> is to add 1 admin list for each of the worker processes (1:1). This > >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list > >>> job_track 0', expecting it to handle the source lists in the same > >>> manner as BRPOP does.
> >>> Is this something that can be supported easily? Any other better way > >>> of doing this?
> >>> Regards, > >>> Ethan.
> >> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
> >>> When multiple source lists are given, BRPOPLPUSH can push them to a > >>> single destination and that should be fine with many requirements.
> >>> I faced this need while wrting up a redis based job queue. I have > >>> multiple workers linked with a single redis list, N:1 relation (with N > >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > >>> 0'. I now have the necessity to talk to any particular worker process, > >>> especially for status, statistics etc purposes. For this, the best way > >>> is to add 1 admin list for each of the worker processes (1:1). This > >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list > >>> job_track 0', expecting it to handle the source lists in the same > >>> manner as BRPOP does.
> >>> Is this something that can be supported easily? Any other better way > >>> of doing this?
> >>> Regards, > >>> Ethan.
> >> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
> >>> When multiple source lists are given, BRPOPLPUSH can push them to a > >>> single destination and that should be fine with many requirements.
> >>> I faced this need while wrting up a redis based job queue. I have > >>> multiple workers linked with a single redis list, N:1 relation (with N > >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > >>> 0'. I now have the necessity to talk to any particular worker process, > >>> especially for status, statistics etc purposes. For this, the best way > >>> is to add 1 admin list for each of the worker processes (1:1). This > >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list > >>> job_track 0', expecting it to handle the source lists in the same > >>> manner as BRPOP does.
> >>> Is this something that can be supported easily? Any other better way > >>> of doing this?
> >>> Regards, > >>> Ethan.
> >> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
> >>> When multiple source lists are given, BRPOPLPUSH can push them to a > >>> single destination and that should be fine with many requirements.
> >>> I faced this need while wrting up a redis based job queue. I have > >>> multiple workers linked with a single redis list, N:1 relation (with N > >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > >>> 0'. I now have the necessity to talk to any particular worker process, > >>> especially for status, statistics etc purposes. For this, the best way > >>> is to add 1 admin list for each of the worker processes (1:1). This > >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list > >>> job_track 0', expecting it to handle the source lists in the same > >>> manner as BRPOP does.
> >>> Is this something that can be supported easily? Any other better way > >>> of doing this?
> >>> Regards, > >>> Ethan.
> > I would like to add another vote for extending BRPOPLPUSH to support > > multiple lists. I think it is consistent with the existing behavior of > BRPOP > > and would make implementing prioritized job queues a snap, unlike the > > use of multiple connections.
> > thanks, > > dave
> > -- > > You received this message because you are subscribed to the Google Groups > > "Redis DB" group. > > To post to this group, send email to redis-db@googlegroups.com. > > To unsubscribe from this group, send email to > > redis-db+unsubscribe@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/redis-db?hl=en.
> -- > You received this message because you are subscribed to the Google Groups > "Redis DB" group. > To post to this group, send email to redis-db@googlegroups.com. > To unsubscribe from this group, send email to > redis-db+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/redis-db?hl=en.
>> This is not going to be supports. To echo what Salvatore already said >> in the pull request:
>> Where the regular blocking pop is unidirectional, BRPOPLPUSH is >> bidirectional: it pops from a list and pushed onto another list. If >> you look at the return value of the blocking pop, you'll see that it >> both returns the list the value was popped from together with the >> value itself. There is no way to preserve this information for >> BRPOPLPUSH. A value is popped from one list and pushed to another. If >> you think about a BRPOPLPUSH that pops from multiple lists to finally >> push the value to another list, that information is lost. So, when >> that information is lost anyway, why not push those values to a single >> list to begin with?
> Well, for a collection of job queues with multiple priorities, the simple > reason is > that you need the information before you do the pop (the information is > the order > of the lists you give to the command, i.e., the relative queue priorities) > and you > don't need it afterwards, at least you can live without it.
> But since BRPOPLPUSH does not currently support this, surely if the command > is extended to support multiple keys, the reply value could be extended > for the > multiple key case to also return the list from which the value was popped > just like > BRPOP? Hence, no information loss at all and backwards compatibility with > the > single key case.
And just to reply to the objection that the information is still lost in redis itself, you could instead have the multikey case push two elements onto the list, the name of the original list and the element itself.
>> >> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>> >>> When multiple source lists are given, BRPOPLPUSH can push them to a >> >>> single destination and that should be fine with many requirements.
>> >>> I faced this need while wrting up a redis based job queue. I have >> >>> multiple workers linked with a single redis list, N:1 relation (with N >> >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >> >>> 0'. I now have the necessity to talk to any particular worker process, >> >>> especially for status, statistics etc purposes. For this, the best way >> >>> is to add 1 admin list for each of the worker processes (1:1). This >> >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >> >>> job_track 0', expecting it to handle the source lists in the same >> >>> manner as BRPOP does.
>> >>> Is this something that can be supported easily? Any other better way >> >>> of doing this?
>> >>> Regards, >> >>> Ethan.
>> >> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>> >>> When multiple source lists are given, BRPOPLPUSH can push them to a >> >>> single destination and that should be fine with many requirements.
>> >>> I faced this need while wrting up a redis based job queue. I have >> >>> multiple workers linked with a single redis list, N:1 relation (with N >> >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >> >>> 0'. I now have the necessity to talk to any particular worker process, >> >>> especially for status, statistics etc purposes. For this, the best way >> >>> is to add 1 admin list for each of the worker processes (1:1). This >> >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >> >>> job_track 0', expecting it to handle the source lists in the same >> >>> manner as BRPOP does.
>> >>> Is this something that can be supported easily? Any other better way >> >>> of doing this?
>> >>> Regards, >> >>> Ethan.
>> >> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>> >>> When multiple source lists are given, BRPOPLPUSH can push them to a >> >>> single destination and that should be fine with many requirements.
>> >>> I faced this need while wrting up a redis based job queue. I have >> >>> multiple workers linked with a single redis list, N:1 relation (with N >> >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >> >>> 0'. I now have the necessity to talk to any particular worker process, >> >>> especially for status, statistics etc purposes. For this, the best way >> >>> is to add 1 admin list for each of the worker processes (1:1). This >> >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >> >>> job_track 0', expecting it to handle the source lists in the same >> >>> manner as BRPOP does.
>> >>> Is this something that can be supported easily? Any other better way >> >>> of doing this?
>> >>> Regards, >> >>> Ethan.
>> >> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>> >>> When multiple source lists are given, BRPOPLPUSH can push them to a >> >>> single destination and that should be fine with many requirements.
>> >>> I faced this need while wrting up a redis based job queue. I have >> >>> multiple workers linked with a single redis list, N:1 relation (with N >> >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >> >>> 0'. I now have the necessity to talk to any particular worker process, >> >>> especially for status, statistics etc purposes. For this, the best way >> >>> is to add 1 admin list for each of the worker processes (1:1). This >> >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >> >>> job_track 0', expecting it to handle the source lists in the same >> >>> manner as BRPOP does.
>> >>> Is this something that can be supported easily? Any other better way >> >>> of doing this?
>> >>> Regards, >> >>> Ethan.
>> >> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>> >>> When multiple source lists are given, BRPOPLPUSH can push them to a >> >>> single destination and that should be fine with many requirements.
>> >>> I faced this need while wrting up a redis based job queue. I have >> >>> multiple workers linked with a single redis list, N:1 relation (with N >> >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >> >>> 0'. I now have the necessity to talk to any particular worker process, >> >>> especially for status, statistics etc purposes. For this, the best way >> >>> is to add 1 admin list for each of the worker processes (1:1). This >> >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >> >>> job_track 0', expecting it to handle the source lists in the same >> >>> manner as BRPOP does.
>> >>> Is this something that can be supported easily? Any other better way >> >>> of doing this?
>> >>> Regards, >> >>> Ethan.
>> >> On Saturday, 24 September 2011 00:02:42 UTC+2, Ethan Collins wrote:
>> >>> When multiple source lists are given, BRPOPLPUSH can push them to a >> >>> single destination and that should be fine with many requirements.
>> >>> I faced this need while wrting up a redis based job queue. I have >> >>> multiple workers linked with a single redis list, N:1 relation (with N >> >>> ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track >> >>> 0'. I now have the necessity to talk to any particular worker process, >> >>> especially for status, statistics etc purposes. For this, the best way >> >>> is to add 1 admin list for each of the worker processes (1:1). This >> >>> makes the command look like 'BRPOPLPUSH worker_N_admin job_list >> >>> job_track 0', expecting it to handle the source lists in the same >> >>> manner as BRPOP does.
>> >>> Is this something that can be supported easily? Any other better way >> >>> of doing this?
>> >>> Regards, >> >>> Ethan.
>> > I would like to add another vote for extending BRPOPLPUSH to support >> > multiple lists. I think it is consistent with the existing behavior of >> BRPOP >> > and would make implementing prioritized job queues a snap, unlike the >> > use of multiple connections.
>> > thanks, >> > dave
>> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "Redis DB" group. >> > To post to this group, send email to redis-db@googlegroups.com. >> > To unsubscribe from this group, send email to >> > redis-db+unsubscribe@googlegroups.com. >> > For more options, visit this group at >> > http://groups.google.com/group/redis-db?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups >> "Redis DB" group. >> To post to this group, send email to redis-db@googlegroups.com. >> To unsubscribe from this group, send email to >> redis-db+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/redis-db?hl=en.
Sorry to pull up an old discussion, but this seemed to have some healthy discussion about BRPOPLPUSH supporting multiple sources.
There was some debate on how to make the API clean, and if you can specify multiple sources then should we allow multiple destinations for example.
For example, in our use case;
* 8 workers need to pop jobs off multiple queues in an atomic/blocking fashion * Once popped, the jobs need to be put into another queue (in our case, the destination queue is always the same). * Specifying multiple sources to BRPOPLPUSH is not an option, so we use BRPOP but lose some atomicity. * Alternatively we could attempt to LPOPRPUSH over our list of queues in sequence, but this massively increases the number of commands each worker is executing, and increases overall latency, thus reducing performance.
Salvadore - can you please clarify if your position on this has changed at all?
On Friday, September 23, 2011 10:02:42 PM UTC, Ethan Collins wrote:
> When multiple source lists are given, BRPOPLPUSH can push them to a > single destination and that should be fine with many requirements.
> I faced this need while wrting up a redis based job queue. I have > multiple workers linked with a single redis list, N:1 relation (with N > ranging between 2-4). For this I use 'BRPOPLPUSH job_list job_track > 0'. I now have the necessity to talk to any particular worker process, > especially for status, statistics etc purposes. For this, the best way > is to add 1 admin list for each of the worker processes (1:1). This > makes the command look like 'BRPOPLPUSH worker_N_admin job_list > job_track 0', expecting it to handle the source lists in the same > manner as BRPOP does.
> Is this something that can be supported easily? Any other better way > of doing this?
