In-process scheduler as part of the worker

[dnagir]

Hi,

Seeking for some strong opinions and critique about this technique:

https://gist.github.com/4056879

(The only assumption it has is that every worker has access to ENV['PIDFILE'] variable in a predefined format (which is provisioned by Chef)).

What it does:

1. Can control what exactly workers do from Ruby and easily re-deploy changes.

2. Can control scheduling without external processes (e.g.: [whenever, cron] --start a process, load full Rails--> enqueue the job).

(a bonus - specs on how the workers are started)

Note that the option with whenever works perfectly fine. The only problem is that all the background jobs are run as part of the normal DelayedJob worker.

Thus need to be enqueued. So cron loads full Rails app to just insert a single record (enqueue the job) which is horribly inefficient.

My thinking: if we already have background workers (that are monitored with monit) why do we need another piece of infrastructure to schedule the jobs?

So I decided to try how the in-process scheduling would work which seems to be extremely lightweight and fast.

Ay constructive criticism on this? Potential gotchas, issues?

Cheers,

Dmytrii.