A Task Queue Implementation
Monnand
As you may already know, I have just uploaded a go package named
gotaskqueue to github.com. As the name suggests, it is an
implementation of task queue, with which you can add several tasks,
specify their running time, and wait them to be run in background at
specific time point.
github page: https://github.com/monnand/gotaskqueue
Document: http://uniqush.org/wiki/UniqushGoTaskQueue
Similar project (from go project dashboard):
- cron.go: https://github.com/rk/cron.go
I hope this package may help you in some way. Have fun!
Regards,
-Monnand
roger peppe
go func() {
<-time.After(eventTime - time.Nanoseconds())
doSomething()
}
?
Monnand
> how is this better than simply doing:
>
> go func() {
> <-time.After(eventTime - time.Nanoseconds())
> doSomething()
>
> }
>
> ?
simpler.
However, you may want to have a better structured way to delay the
works if things become complicated. And reuse the old code as much as
possible.
Here is an example:
There is a project, requiring an exponential backoff retry mechanism.
Moreover, inside that project, a cache is needed, which will
synchronize to the database every, say, 10 min.
If you are trying to implement these two different jobs in the way you
described before, you may find there are much code doing similar jobs.
For me, I would rather like a clean way to implement the real job.
I'm not sure if I expressed my idea clearly. It is more than welcome
to read the document here: http://uniqush.org/wiki/UniqushGoTaskQueue
It may be help to you in its way of decoupling the job details and
other stuff.
Another minor point: If we have a job which should be processed 10min
later, then your code would create a goroutine waiting for 10min and
then process the data; while my package put all tasks in a priority
queue and run the tasks when they are ready to run in a separate
goroutine. OK, I know goroutines are cheap. It should not be a big
advantage at this point.
Regards,
-Monnand
roger peppe
> On Sep 16, 10:05 am, roger peppe <rogpe...@gmail.com> wrote:
>> how is this better than simply doing:
>>
>> go func() {
>> <-time.After(eventTime - time.Nanoseconds())
>> doSomething()
>>
>> }
>>
>> ?
>
> Well... Yes, you are right, we can always do that which is much
> simpler.
>
> However, you may want to have a better structured way to delay the
> works if things become complicated. And reuse the old code as much as
> possible.
>
> Here is an example:
> There is a project, requiring an exponential backoff retry mechanism.
> Moreover, inside that project, a cache is needed, which will
> synchronize to the database every, say, 10 min.
>
> If you are trying to implement these two different jobs in the way you
> described before, you may find there are much code doing similar jobs.
i'm not convinced. the code to do the delay is one line.
> Another minor point: If we have a job which should be processed 10min
> later, then your code would create a goroutine waiting for 10min and
> then process the data; while my package put all tasks in a priority
> queue and run the tasks when they are ready to run in a separate
> goroutine. OK, I know goroutines are cheap. It should not be a big
> advantage at this point.
there's also time.AfterFunc which doesn't start a new goroutine
for each job.
can you give an example that can't implemented in less lines of code
with the existing primitives?
Monnand
<snip>
> for each job.
>
queue could have any number of tasks.
> can you give an example that can't implemented in less lines of code
> with the existing primitives?
be enough to understand my idea.
Suppose you want to print a hello world every 2 seconds after waiting
5 seconds. First, define you job:
type HelloWorld struct {
n int
}
func (j *HelloWorld) Run(s gotaskqueue.TimeSetter) {
if j.n >= 3 { s.Stop() }
fmt.Print("Hello World\n")
}
Then we create a task queue and our task:
ch := make(chan Task)
q := gotaskqueue.NewTaskQueue(ch)
t := gotaskqueue.NewPeriodicTask(q, ch)
t.SetPeriod(2)
t.After(5)
I think it may not to explain the meaning of each line. You can
understand it from the names.
Now we put the task into the queue:
ch <- t
Another advantage of using a library, I believe, is flexibility. You
can change little code when requirement changed.
Suppose we want to change the program to delay printing using
exponential backoff algorithm ( http://en.wikipedia.org/wiki/Exponential_backoff
), starting from 2 seconds. The only changes we need are:
- t := gotaskqueue.NewPeriodicTask(q, ch)
+ t := gotaskqueue.NewExpBackoffTask(q, ch, 2)
- t.SetPeriod(2)
Done. You may argue that the exponential backoff algorithm is built in
our library, which makes the code easy and it is not fair to compare
this with time.Timer. Well. Yes. That's why we need the library.
Besides exponential backoff algorithm, I think I will provide more
algorithms, which are common used to calculate delay time of works.
I hope this example is enough to show you why I write this package. It
is very glad to discuss with you.
Regards,
-Monnand
roger peppe
>
> <snip>
>> there's also time.AfterFunc which doesn't start a new goroutine
>> for each job.
>>
>
> One difference: The Timer type represents a single event; while a task
> queue could have any number of tasks.
>
>> can you give an example that can't implemented in less lines of code
>> with the existing primitives?
>
> Sure. I will only show you how to use my package to do the job. It may
> be enough to understand my idea.
>
> Suppose you want to print a hello world every 2 seconds after waiting
> 5 seconds. First, define you job:
>
> type HelloWorld struct {
> n int
> }
> func (j *HelloWorld) Run(s gotaskqueue.TimeSetter) {
> if j.n >= 3 { s.Stop() }
> fmt.Print("Hello World\n")
> }
>
> Then we create a task queue and our task:
> ch := make(chan Task)
> q := gotaskqueue.NewTaskQueue(ch)
> t := gotaskqueue.NewPeriodicTask(q, ch)
> t.SetPeriod(2)
> t.After(5)
isn't the following code simpler and easier to understand?
go func() {
<-time.After(5e9)
for i := 0; i < 5; i++ {
fmt.Printf("Hello, world\n")
<-time.After(2e9)
}
}
or for exponential backoff:
go func() {
backoff := 0.1e9
for backoff < 120e9 {
fmt.Printf("Hello, world\n")
<-time.After(backoff)
backoff *= 2
}
}
go has goroutines to deal with state, why not use them?
> Besides exponential backoff algorithm, I think I will provide more
> algorithms, which are common used to calculate delay time of works.
if there are other, less obvious, algorithms, then i agree
that a package to implement them might be useful.
but the time calculation can be orthogonal to the task
queuing mechanism, and it's potentially more useful that way.
i could imagine an interface like this:
type TimeSequence interface {
NextDelay() (int64, bool)
}
then either function above might be written as:
go func() {
for {
fmt.Printf("Hello, world\n")
delay, ok := seq.NextDelay()
if !ok {
break
}
<-time.After(delay) }
}
and exponential backoff might be factored out into code like this:
type exponentialBackoff struct {
delay int64
max int64
}
func (t *exponentialBackoff) NextDelay() (int64, bool) {
delay := t.delay
if delay >= t.max {
return 0, false
}
t.delay *= 2
return delay, true
}
func NewExponentialBackoff(initialNanoseconds, maxNanoseconds int64)
TimeSequence {
return &exponentialBackoff{initialNanoseconds, maxNanoseconds}
}
if there were sufficiently involved time calculation algorithms that
you wanted to plug and play with, then i could see that
this might be a win. TaskQueue itself makes things harder though, not
easier, i think, sorry.