redis counter to limit jobs running
281 views
Skip to first unread message
Karl
Oct 17, 2010, 9:39:53 PM10/17/10
to Redis DB
We are trying to limit the number of backend ruby jobs running for a
particular user to 5 by keeping a count within redis. Each time a job
runs, it first checks the count to see if it is less than 5. If it is
less than 5, it increments the count, executes, and afterwords
decrements the count. If before running, the job checks the count and
it is greater than or equal to 5, the job re-queues itself to try and
run again later.
The obvious problem is that 2 or more jobs can check the count at the
same time and see that it is 4, causing the number of jobs running to
jump above our limit to 6. Is it possible, within a redis multi
transaction, to increment the counter only if the value is less than
5?
thx
-karl
particular user to 5 by keeping a count within redis. Each time a job
runs, it first checks the count to see if it is less than 5. If it is
less than 5, it increments the count, executes, and afterwords
decrements the count. If before running, the job checks the count and
it is greater than or equal to 5, the job re-queues itself to try and
run again later.
The obvious problem is that 2 or more jobs can check the count at the
same time and see that it is 4, causing the number of jobs running to
jump above our limit to 6. Is it possible, within a redis multi
transaction, to increment the counter only if the value is less than
5?
thx
-karl
Josiah Carlson
Oct 18, 2010, 11:17:50 AM10/18/10
to redi...@googlegroups.com
You don't need transactions. Because increment/decrement operations
are atomic, you can simply:
1. increment your counter, the new total is returned as a result of
the operation
2. if the value is greater than 5, immediately decrement and re-queue yourself
3. if the value is <= 5, run the operation, and decrement when done
are atomic, you can simply:
1. increment your counter, the new total is returned as a result of
the operation
2. if the value is greater than 5, immediately decrement and re-queue yourself
3. if the value is <= 5, run the operation, and decrement when done
This works because 2 racing operations will be ordered one after the
other in Redis, so one will actually win, and one will get the 5 as a
result, while the other gets a 6.
- Josiah
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>
Andy Lawman
Oct 18, 2010, 11:26:34 AM10/18/10
to redi...@googlegroups.com
One way to approach this is to issue an INCR when you wish to start a job without first testing the value. If the response is <= 5 then simply run the job as you describe. If it's > 5 issue a DECR, don't run the job and requeue it.
Andy.
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/redis-db?hl=en.You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at
IMPORTANT - CONFIDENTIALITY NOTICE - This e-mail is intended only for the use of the addressee/s above. It may contain information which is privileged, confidential or otherwise protected from disclosure under applicable laws. If the reader of this transmission is not the intended recipient, you are hereby notified that any dissemination, printing, distribution, copying, disclosure or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this transmission in error, please immediately notify us by reply e-mail or using the address below and delete the message and any attachments from your system.
Amadeus Services Ltd, World Business Centre 3, 1208 Newall Road, Hounslow, Middlesex, TW6 2TA, Registered number 4040059
Luke Antins
Oct 18, 2010, 11:47:43 AM10/18/10
to redi...@googlegroups.com
Does anyone have a suggestion for deal with workers failing to DECR
after it's finished processing?
after it's finished processing?
I was thinking perhaps a LIST of timeouts? but that would still have its
own issues, though it shouldn't seize up?
--
Luke Antins
http://lividpenguin.com
Andy Lawman wrote:
>
> One way to approach this is to issue an INCR when you wish to start a
> job without first testing the value. If the response is <= 5 then
> simply run the job as you describe. If it's > 5 issue a DECR, don't
> run the job and requeue it.
>
> Andy.
>
>
>
> From: Karl <karl...@gmail.com>
> To: Redis DB <redi...@googlegroups.com>
> Date: 18/10/2010 15:42
> Subject: redis counter to limit jobs running
> Sent by: redi...@googlegroups.com
> ------------------------------------------------------------------------
> Hounslow, Middlesex, TW6 2TA, Registered number 4040059 --
Josiah Carlson
Oct 18, 2010, 12:27:03 PM10/18/10
to redi...@googlegroups.com
As long as your number of workers is small, and you don't care where
it's running, a list is fine. If you have a lot of workers, they are
distributed across many machines, and you care about what's running,
then using a zset with timeouts as the score, and something like
<machine>:<pid> as the member.
it's running, a list is fine. If you have a lot of workers, they are
distributed across many machines, and you care about what's running,
then using a zset with timeouts as the score, and something like
<machine>:<pid> as the member.
- Josiah
Karl Baum
Oct 18, 2010, 3:28:36 PM10/18/10
to redi...@googlegroups.com
Great idea! Thanks to both Andy and Josiah :-).
Luke Antins
Oct 18, 2010, 5:51:49 PM10/18/10
to redi...@googlegroups.com
Thank you, I'll try this out tonight.
--
Luke Antins
http://lividpenguin.com
Luke Antins
Oct 18, 2010, 8:08:16 PM10/18/10
to Redis DB
Just been trying out a solution with zset, much nicer and easier then
using a list!
* Check if the limit has been hit using zcard.
* If the limit was hit, use zremrangebyscore to remove old lock(s).
* If we couldn't remove a old lock, we stop here and try again later.
* If all is still good, acquire a lock. Using <hostname>:<pid> for the
member as suggested and timeout for the score.
* Perform work.
* Remove lock using zdel.
I guess its possible for the number of workers to be grater then the
limit? but that occasionally happening really isn't an issue for me.
Thanks again!
---
Luke Antins
http://lividpenguin.com/
On Oct 18, 10:51 pm, Luke Antins <l...@lividpenguin.com> wrote:
> Thank you, I'll try this out tonight.
>
> --
> Luke Antinshttp://lividpenguin.com
using a list!
* Check if the limit has been hit using zcard.
* If the limit was hit, use zremrangebyscore to remove old lock(s).
* If we couldn't remove a old lock, we stop here and try again later.
* If all is still good, acquire a lock. Using <hostname>:<pid> for the
member as suggested and timeout for the score.
* Perform work.
* Remove lock using zdel.
I guess its possible for the number of workers to be grater then the
limit? but that occasionally happening really isn't an issue for me.
Thanks again!
---
Luke Antins
http://lividpenguin.com/
On Oct 18, 10:51 pm, Luke Antins <l...@lividpenguin.com> wrote:
> Thank you, I'll try this out tonight.
>
> --
>
>
>
> Josiah Carlson wrote:
> > As long as your number of workers is small, and you don't care where
> > it's running, a list is fine. If you have a lot of workers, they are
> > distributed across many machines, and you care about what's running,
> > then using a zset with timeouts as the score, and something like
> > <machine>:<pid> as the member.
>
> > - Josiah
>
>
>
> Josiah Carlson wrote:
> > As long as your number of workers is small, and you don't care where
> > it's running, a list is fine. If you have a lot of workers, they are
> > distributed across many machines, and you care about what's running,
> > then using a zset with timeouts as the score, and something like
> > <machine>:<pid> as the member.
>
> > - Josiah
>
> > On Mon, Oct 18, 2010 at 8:47 AM, Luke Antins<l...@lividpenguin.com> wrote:
>
> >> Does anyone have a suggestion for deal with workers failing to DECR after
> >> it's finished processing?
>
> >> I was thinking perhaps a LIST of timeouts? but that would still have its own
> >> issues, though it shouldn't seize up?
>
> >> --
> >> Luke Antins
> >>http://lividpenguin.com
>
> >> Andy Lawman wrote:
>
> >>> One way to approach this is to issue an INCR when you wish to start a job
> >>> without first testing the value. If the response is<= 5 then simply run the
> >>> job as you describe. If it's> 5 issue a DECR, don't run the job and requeue
> >>> it.
>
> >>> Andy.
>
>
> >> Does anyone have a suggestion for deal with workers failing to DECR after
> >> it's finished processing?
>
> >> I was thinking perhaps a LIST of timeouts? but that would still have its own
> >> issues, though it shouldn't seize up?
>
> >> --
> >> Luke Antins
> >>http://lividpenguin.com
>
> >> Andy Lawman wrote:
>
> >>> One way to approach this is to issue an INCR when you wish to start a job
> >>> without first testing the value. If the response is<= 5 then simply run the
> >>> job as you describe. If it's> 5 issue a DECR, don't run the job and requeue
> >>> it.
>
> >>> Andy.
>
Reply all
Reply to author
Forward
0 new messages