Scheduling execution
Dustin
I noticed that the protocol lists both methods of scheduling
execution as effectively deprecated. Are there any thoughts on having
a supported mechanism for this?
Ideally, I'd want to just submit with a delay. The cron facility
looks particularly odd, and the epoch one is more subject to time skew
than necessary. Ideally, I just want to queue a job to run in five
seconds or so. If I want to run at an exact time, I can just
calculate that.
Brian Aker
On Jul 26, 2009, at 5:53 PM, Dustin wrote:
> Ideally, I'd want to just submit with a delay. The cron facility
> looks particularly odd, and the epoch one is more subject to time skew
This was a bit that Allan and I sketched out at one point. The open
question... should we be able to do "scheduled" jobs inside of the
server. I believe it is required longterm.
> than necessary. Ideally, I just want to queue a job to run in five
> seconds or so. If I want to run at an exact time, I can just
> calculate that.
Re-occurring tasks are the reason for this... we have not spent a lot
of time figuring out the ins and outs of this.
Cheers,
-Brian
Eric Day
use the generic SUBMIT_JOB with options method we discussed at the
Gearman hackathon in April. We didn't want to keep adding a command
(or set of commands) for each new option. I still need to add this
into the C server and library, but once it is in we will have arbitrary
options and bitfields to use.
-Eric
Dustin
On Jul 26, 6:31 pm, Brian Aker <br...@tangent.org> wrote:
> > than necessary. Ideally, I just want to queue a job to run in five
> > seconds or so. If I want to run at an exact time, I can just
> > calculate that.
>
> Re-occurring tasks are the reason for this... we have not spent a lot
> of time figuring out the ins and outs of this.
Is gearman-as-cron something that makes a lot of sense?
Does gearman-as-at exist (in roadmap form) as an independent piece
of functionality?
Vick Khera
> Is gearman-as-cron something that makes a lot of sense?
>
> Does gearman-as-at exist (in roadmap form) as an independent piece
> of functionality?
Both of these functionalities would solve a rather substantial class
of work we have. Right now we have our own external scheduler and
persistent queues, but moving more work to gearman would simplify our
architecture.
Thomas Koch
trivially get something like this already with todays gearman. Just use the
gearman shell client and put calls to it either in a crontab or in the at
command.
Thomas Koch, http://www.koch.ro
Brian Moon
> trivially get something like this already with todays gearman. Just use the
> gearman shell client and put calls to it either in a crontab or in the at
> command.
Cron, perhaps. But, AT would be useful from an application POV. I can
not easily schedule a job from a web application from another data
center using at on another server. It would also be very useful for
retries. In my worker, if a job was not able to be processed, I could
requeue that job for "1 minute from now" or use a timestamp equivalent.
Brian.
Dustin
On Jul 27, 1:35 pm, Brian Moon <br...@moonspot.net> wrote:
> Cron, perhaps. But, AT would be useful from an application POV. I can
> not easily schedule a job from a web application from another data
> center using at on another server. It would also be very useful for
> retries. In my worker, if a job was not able to be processed, I could
> requeue that job for "1 minute from now" or use a timestamp equivalent.
introduce a new sort of ``soft'' failure where the failure itself
indicates a delay. This way you can hand control of the job back from
the worker without having to pass the job info back and forth again.
However, an initial enqueue with delay is also pretty quickly
beneficial. I should like to queue a job to run one minute from now.
This kind of thing is useful for app timeouts, consumer-driven loops,
all kinds of stuff. I think there's a lot of opportunities for
improvement around the background job queueing. :)
Ian Rogers
Was there any progress on this idea in the last year?
Regards,
Ian
Ian Rogers
Ok, by the silence I guess there wasn't any progress :-)
Does anyone here have experience with resilient/distributed job
schedulers? I've found this article -
http://www.pcpro.co.uk/realworld/355924/the-perfect-open-source-task-scheduler
- but all the proposed tools are still lacking in resilience it seems.
Any pointers gratefully recieved.
Cheers,
Ian
Rhett Garber
your task scheduling needs:
It's a work in progress, of course, but the general idea is to replace
our dependence on cron with something more native to a distributed
environment.
Blog post:
http://engineeringblog.yelp.com/2010/09/tron.html
Comments appreciated.
Rhett
Vick Khera
> Does anyone here have experience with resilient/distributed job schedulers?
> I've found this article -
> http://www.pcpro.co.uk/realworld/355924/the-perfect-open-source-task-scheduler
> - but all the proposed tools are still lacking in resilience it seems.
>
Nice! GNUbatch looks promising to me for a project I have *just*
started. Thanks for sharing. How do you see this particular one
lacking in resilience?
Ian Rogers
From reading the GNUbatch user guide
http://cvs.savannah.gnu.org/viewvc/*checkout*/gnubatch/doc/pdf/UserG.pdf?root=gnubatch
it wasn't clear to me how the job schedule is made resilient.
Multiple clients can connect to the schedule/queue holder, and the
runner can spawn jobs on multiple servers, but if a schedule holder goes
down how does a backup take over? I may have just missed something in
the doc though.
Ian