ActiveJob allow passing options to underlying library

[John Negron]

ActiveJob is a great addition to Rails. Many supported backends like DelayedJob support other options than what is being passed into the enqueue adapter methods. It would be great if the "set" method could delegate options into the queue adapters. 

For example, DelayedJob has priority set at the job level and setting the priority could be done without named queues and other background workers.

Are there any plans for that or would the group be open to a pull request for it?

thanks

John

[DHH]

That's still an open question. I'd like to treat each option we'd consider passing on to a specific adapter as an invitation to discuss if we can solve it in an adapter-agnostic way.

What's the purpose of setting priority on the job rather than using separate queues, in your mind?

[John Negron]

Different backend workers/consumers have their own process management, configuration, etc. so relying on queue naming as a sole indicator of processing priority at an abstracted layer like ActiveJob means making some assumptions on the underlying backend worker configuration. 

I was thinking this should be something adapter specific and left to the backend adapters to understand these per backend constraints; hence some delegation into the adapter from ActiveJob.

[Karl Metum]

One example is when passing options such as :retry => false so that e-mail sending would stop if the e-mail failed to be delivered (with Sidekiq)

[DHH]

I'd like to a specific example from a real app where using a separate queue isn't just as well.

[DHH]

That's easily possible already by just swallowing the exception with rescue_from(DeliverException) { } # No retry.

Could think about wrapping that pattern, but it's not an example I find compelling for passing through adapter options.

[Abdelkader Boudih]

That will require to monkey https://github.com/rails/rails/blob/master/actionmailer/lib/action_mailer/delivery_job.rb

[Xavier Noria]

On Thu, Feb 12, 2015 at 4:21 PM, Abdelkader Boudih <term...@gmail.com> wrote:

That will require to monkey https://github.com/rails/rails/blob/master/actionmailer/lib/action_mailer/delivery_job.rb

Yes, at this point the job for mailers is kinda private... guess it needs to be accessible somehow. 

[DHH]

That should be a config flag for ActionMailer, then. 

[Steven Bull]

I like DelayedJob's feature of different "priorities" in the same queue largely for conservation of resources. I don't need to have multiple processes running in order to make some things more important than others.

Use Case A: My site has bursty traffic, and I use DJ to send transactional emails. I usually have 0 jobs in the queue, but occasionally it gets busy. It's much more important to me that a "password reset" email gets picked first, even if there are 20 other jobs that came in all of a sudden which are less important, such as generating a data export, scheduled daily admin reporting, or even sending the "welcome" emails.

Use Case B: I have a low-priority cleanup job that reschedules itself for 5 minutes later after it completes. It doesn't matter if it gets pushed back to a later time based on other jobs that need to happen. I'd rather not have to run some other process to manage this job / this specific queue.

I suppose the argument could be made that the underlying library should have a configuration utility to be able to turn queue names into priorities, but by nature that would involve more configuration (first assign a queue, then assign a priority to that queue), as opposed to just being able to set a priority without configuring the queue name. DJ started with priorities only, and added queue names later (after other libraries became popular). Queues are for sure more sophisticated of a technique, but using priorities is often simpler and "good enough" - until your job management needs have outgrown DJ / SQL-based options, and you need to transition to another solution.

[Seva Orlov]

Hi, another example is sidekiq 'unique: true' option, that is added by https://github.com/mhenrixon/sidekiq-unique-jobs
Quite useful feature because several unicorn workers can add jobs with the same parameters and we want only one unique job to be executed.

For example we have learning plans that users complete, we calculate users score in sidekiq. If learning plan structure was updated, we need to recalculate users scores. If learning plan was updated several times, we don't want to calculate users score several times, we want to calculate it only once, after the last change.

I don't see how we can check jobs uniqueness before we enqueue a job, because workers are parallel. And I don't see how we can check it during job execution, because sidekiq workers are also parallel.

What do you think?

There are also other useful options for jobs uniqueness - either to check for uniqueness on job execution or on job adding to a queue.