Clojure job scheduler

[Trevor]

What's the best way to kick off Clojure code at scheduled times? I
have some that would run once a day. Some that might run 2 or 3 times
a day based upon a test being met.

1. I could write a function that sleeps an interval, check the time
differential to perform a time-box triggered function, but would that
consume too much memory?, cause long term problems?

2. I could use quartz, but that seems like overkill.

3. I could set a job-schedule using the OS to run a clojure script.
I'd rather not, I would like to do things like send emails / check
status via web app (making option 1 more appealing).

I'm looking for input/guidance. What are your experiences?

Thanks,

[gaz jones]

the work library has a function which it describes as 'cron for
clojure functions':

https://github.com/clj-sys/work.git

cant say i have used it, but i noticed it in there recently whilst
looking for other things.

here is the function:

(defn schedule-work
"schedules work. cron for clojure fns. Schedule a single fn with a
pool to run every n seconds,
where n is specified by the rate arg, or supply a vector of fn-rate
tuples to schedule a bunch of fns at once."
([f rate]
(let [pool (Executors/newSingleThreadScheduledExecutor)]
(.scheduleAtFixedRate
pool (with-log f) (long 0) (long rate) TimeUnit/SECONDS)
pool))
([jobs]
(let [pool (Executors/newSingleThreadScheduledExecutor)]
(doall (for [[f rate] jobs]
(schedule-work pool f rate)))
pool)))

hope that is useful,
cheers
gaz

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

[Michael Gardner]

On Jan 7, 2011, at 11:13 AM, Trevor wrote:

> 3. I could set a job-schedule using the OS to run a clojure script.
> I'd rather not, I would like to do things like send emails / check
> status via web app (making option 1 more appealing).

Could you elaborate on why a scheduled job to run a Clojure script would be inappropriate for those kinds of tasks? Anything you can do from a daemon, you should be able to do from a cron job.

[Ken Wesson]

Let's see. On the one hand, running a job scheduler from inside
Clojure results in cranking up a big, slow to start up, expensive JVM
process once a session, on startup, which then runs some tasks
periodically, and the scheduling itself can be done in a nice language
like, say, Clojure, with nice error reporting.

On the other hand, running a job scheduler from outside Clojure
results in cranking up a big, slow to start up, expensive JVM process
every single time a task needs to run, each of which runs one task
once, and the scheduling itself must be done in an icky language like
shell or cron's idiosyncratic "crontab" files with icky error
reporting (e.g., need to run a local mail *server* to receive error
notifications).

I think I can see why someone might prefer to have the scheduling done
inside the JVM. :)

[Michael Gardner]

On Jan 7, 2011, at 9:19 PM, Ken Wesson wrote:

> On the other hand, running a job scheduler from outside Clojure
> results in cranking up a big, slow to start up, expensive JVM process
> every single time a task needs to run, each of which runs one task
> once, and the scheduling itself must be done in an icky language like
> shell or cron's idiosyncratic "crontab" files with icky error
> reporting (e.g., need to run a local mail *server* to receive error
> notifications).

If you care about startup times, you can use nailgun. But that shouldn't matter unless you're running the job every minute or something.

As for scheduling, crontabs are really not hard to figure out. If you need more complex scheduling, you can do that from your Clojure script (essentially using cron to set the polling interval).

And what kinds of error reporting could you do from a persistent daemon that you couldn't also do from a cron job? Besides, most systems that have cron also come with postfix (though it's disabled by default on Mac OS X), so all you have to do is add your email address to /etc/aliases. Email-based error reporting for background tasks is really nice because you don't have to remember to check some log file or other task-specific status indicator periodically (which has burned me in the past).

But this is all somewhat beside the point. What Trevor said sounded as though the specific types of tasks he mentioned (sending emails and checking some kind of status via web app) were particularly unsuited to scheduled jobs; I was asking what it was about those tasks in particular that made him lean towards a daemon instead.

[Jeff Rose]

We use the Quartz library for job scheduling in our Clojure projects.
It's nice to have this done within the JVM so that we can easily
deploy to a new server without needing to configure cron (and the
differences with cron across platforms...).

http://www.quartz-scheduler.org/

If you want to call the run-import-process function every morning at
6am, it looks like this:

(ns foo.core
(:import [org.quartz.impl StdSchedulerFactory]
[org.quartz CronTrigger JobDetail]
[java.io File]))

(defn run-import-process
[]
(println "running import process..."))

(def scheduler* (atom nil))
(def CRON-STR-6AM "0 0 6 * * ?") ; every day at 6am
;(def CRON-STR-6AM "* * * * * ?") ; every second, for testing

(deftype ImportJob []
org.quartz.Job
(execute [this context] (run-import-process)))

(defn start-import-manager
[]
(let [scheduler (StdSchedulerFactory/getDefaultScheduler)
trigger (CronTrigger. "import-trigger" "import" CRON-STR-6AM)
job (JobDetail. "import-job" "import" ImportJob)]
(reset! scheduler* scheduler)
(.scheduleJob scheduler job trigger)
(.start scheduler)))

(defn stop-import-manager
[& [wait?]]
(if wait?
(.shutdown @scheduler* true)
(.shutdown @scheduler*)))

Cheers,
Jeff

[Ken Wesson]

On Sat, Jan 8, 2011 at 12:04 AM, Michael Gardner <gard...@gmail.com> wrote:
> On Jan 7, 2011, at 9:19 PM, Ken Wesson wrote:
>
>> On the other hand, running a job scheduler from outside Clojure
>> results in cranking up a big, slow to start up, expensive JVM process
>> every single time a task needs to run, each of which runs one task
>> once, and the scheduling itself must be done in an icky language like
>> shell or cron's idiosyncratic "crontab" files with icky error
>> reporting (e.g., need to run a local mail *server* to receive error
>> notifications).
>
> If you care about startup times, you can use nailgun. But that shouldn't matter unless you're running the job every minute or something.

Obviously, that requires knowing about, and learning how to use,
nailgun. Solutions with a higher cost in
novel-tools-you-have-to-figure-out-how-to-use are not, all other
things being equal, superior ones.

> As for scheduling, crontabs are really not hard to figure out. If you need more complex scheduling, you can do that from your Clojure script (essentially using cron to set the polling interval).

If you're going to do that anyway, you might as well do the whole
thing from inside Clojure.

> And what kinds of error reporting could you do from a persistent daemon that you couldn't also do from a cron job? Besides, most
> systems that have cron also come with postfix (though it's disabled by default on Mac OS X), so all you have to do is add your email
> address to /etc/aliases. Email-based error reporting for background tasks is really nice because you don't have to remember to check
> some log file or other task-specific status indicator periodically (which has burned me in the past).

Well, both Windows and MacOS have variations on the nifty concept of
"tray notification".

> But this is all somewhat beside the point. What Trevor said sounded as though the specific types of tasks he mentioned (sending
> emails and checking some kind of status via web app) were particularly unsuited to scheduled jobs; I was asking what it was about
> those tasks in particular that made him lean towards a daemon instead.

Maybe he needs timely responses to something, so something more akin
to a web server than a periodically-run job?

[Trevor]

Thanks, everyone for all you help.

I noticed a few questions I should answer.

re: email option: I really just planned on sending a gmail to indicate
the job succeeded or failed. Being somewhat new to programming the
straightest path, for me, would be using clojure code (I'm not a
network guru, so for me it's grab a library and use use it).

re: webapp status: The job I want to run is really 3 jobs bundled in
one (they need not all run, but they at least need to run sequentially
if they do). So when I see an email notifying the fail, I will use the
web app to determine if #1, #2 or #3 failed. If #1 failed, then I can
trigger 2 and 3. I want this to be a eyeball decision, not a
programmatic one.

Really, I just don't like cron jobs. I'd rather stay with in clojure
if I can where I'm comfortable that I'm not somehow pooching the
system, plus it just seems like something a language ought to be able
to do.

I noticed a point made about not having to deal with OS differences,
which while not an immediate problem for me, is still noteworthy. At
some point I'd like to distribute my code, and not leave that burden
to others.

I'm leaning towards just building my own, testing it out (learn more
this way).
I looked at the function gaz, provided, but it didn't seem like what I
would implement, but I may end up there. If that fails I will probably
use quartz-scheduler.

Once again - thank you for all the replies.



On Jan 8, 6:14 am, Ken Wesson <kwess...@gmail.com> wrote:

[Stuart Sierra]

In a long-running process, Java's ScheduledThreadPoolExecutor [1] will handle this. Since Clojure's functions implement both Java Callable and Runnable, they can be passed as arguments to the .submit and .schedule* methods.

-Stuart Sierra

clojure.com

[1] http://download.oracle.com/javase/6/docs/api/java/util/concurrent/ScheduledThreadPoolExecutor.html

[Patrik Fredriksson]

I have used cron4j in a small project, it's like a more lightweight
version of Quartz and fits nicely with Clojure: http://www.sauronsoftware.it/projects/cron4j/

Code example here: https://gist.github.com/388555

/Patrik

[Trevor]

That works & it's really easy to use - Thanks.

[Sun Ning]

By the way, overtone has made a simple library for scheduled task. You
may take a look:
https://github.com/overtone/at-at

On Mon 11 Jun 2012 04:31:41 PM CST, Joao_Salcedo wrote:
> hi Trevor could you share how did you solve the issue.
>
> I would like to learn more about it.
> If you can share your solution would be great