Re: [delayed_job] How quickly does DJ pick up new jobs?

[David Genord II]

Short answer: immediately.

Longer answer: it depends.

If there are no jobs in the queue and you add a job to the queue it can take up to 5 seconds for the job to run. This is configured by Delayed::Worker.sleep_delay and is set to 5 by default. How this works in practice, is that the worker tries checks the DB for something to do, if nothing is found the process sleeps for sleep_delay seconds, and then checks again.

If the worker is already working on a job, the new job will get run when all jobs in front of it in the queue are done.

David Genord II

On Fri, May 17, 2013 at 2:53 AM, Arman H <arm...@gmail.com> wrote:

Hello all, new to the list. I appreciate all responses in advance. My question is, when I schedule a new job, how quickly does DJ act upon them? Is there any way to configure this option?

--
 
---
You received this message because you are subscribed to the Google Groups "delayed_job" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delayed_job...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Arman H]

David, thanks for the short and (especially) the long answer! Another question, somewhat unrelated: I noticed DJ processes the que in reverse (e.g. LIFO). Is there a way to change the behavior to FIFO?

[David Genord II]

Technically it is not LIFO or FIFO. Jobs are processed in order of priority, lowest value first, then run_at, earliest time first. This is not configurable.

David Genord II

[Arman H]

Thanks for clarification. How does DJ determine which jobs to run when all jobs have the same priority and no `run_at` time is specified?

P.S. This is purely out of curiosity. I needed to run jobs in sequential order, so setting the `run_at` property to Time.now, got the job done.

[Robert Mathews]

If you need to run them in sequential order, you could queue them with increasing priority.

[David Genord II]

What database backend are you using?

David Genord II

[Arman H]

Robert, thanks for the reply. In order to enqueue them with increasing priority, I would have to keep track of previous jobs' priorities. That's an extra query to the DB every time I'm creating a new job, which I did not want to do. Instead, I'm setting the `run_at` to current time, which seems to work fine ( at least in development environment).

David, I'm running on PostgreSQL (same as Heroku, that's where the app is staged).

[Robert Mathews]

If you were queuing a whole bunch at once the extra query wouldn't matter; If you really need them in order, why don't you have the completion of one of them queue the next calculation?

As for what is wrong with time? Well, you have to choose the granularity of your increment, too fine and the jobs risk running at the same time, too large and it is safe, but slower than optimum. But, hey, whatever works for you.

One difference between development and production might well be the number of queue servers deployed; in development, you have one worker, for us production has 4 queue servers, each with 3 workers.

[David Genord II]

Ok, so you said you were seeing jobs with nil run_at values. That is not suppose to happen. By default DJ sets the run at time to Time.now when the job is created. What gem versions are you using for Rails, DJ, and DJ active record?

David Genord II

[Bill Burcham]

Arman H, Orchestrated (a Ruby Gem) will let you express dependencies between delayed jobs. The syntax is similar to regular delayed job except "delay" is replaced with "orchestrate". The "orchestrate" method (unlike "delay") lets you specify prerequisites.

Imagine you want to "rinse" to run after you "lather". You could do something like this:

B.new.orchestrate( A.new.orchestrate.lather("shampoo") ).rinse("water")

There are primitives for combining prerequisites too. So if you have more than one prerequisite and you want to run after all are complete you use LastCompletion. If you want to run after the first one is complete, use FirstCompletion.