How to keep track of worker status in inline mode?

[Philip Postkasten]

Hi,

I am using sidekiq to make calls to a hotel booking api which has throttle limits.
So I use gem 'sidekiq-limit_fetch' in order to set a hard limit in order to not reach the maximum size of allowed requests.
At the same time I am restricted to post an amount of 1000 ids at the same time.
Now I need to receive quotes for e.g. 12000 ids.
So I slice into packages of 1000 like this:

job = QuoteJob.new
$redis.set(job.jid,{:status=>:started})

request = Api.new({
                                           :arrival => Date.today + 30.days,
                                           :departure => Date.today + 36.days,
                                           :ids => sids
                                       },job)


request.ids.each_slice(MAX_RESULTS).to_a.each do |_ids|
         
          #done in sidekiq
          kind_job = search_by_property_numbers(_options.merge({:PropertyNumbers => {:int => _ids}}), request)
          job.dependent_jobs.push(kind_job)
         
        end

The ultimate goal is to get informed when all the jobs have finished.
That's why I create am simple JobObject which knows about all the dependent jobs (kind_job) and there I do call another worker which asks for the sidekiq status
 
 #Job object has a sidekiq worker
 def perform(dependent_jobs,jid)
    while !dependent_jobs.empty?
      dependent_jobs.each do |jid|
        if Sidekiq::Status::complete? jid or Sidekiq::Status::failed? jid or Sidekiq::Status::status(jid).nil?
          dependent_jobs.delete(jid)
        end
      end
    end
    $redis.set(jid, {:status => :completed}) if dependent_jobs.empty?
  end

My initial request created the Job object and triggers all the dependent workers with the api calls. At the same time the Job returns a jid which holds the status.
So I can poll the status e.g. with an ajax request.


 I started writing tests I by doing this in development.rb
 require 'sidekiq/testing/inline'
The problem is that the Sidekig::Status(jid) in perform is allways return :queued.

This confuses me since the doc says : An inline mode that runs the job immediately instead of enqueuing it

So first of all I d be very happy to head your thoughts on the general approach. And last but not least how do I know when the job finishes?
Help and feedback is highly appreciated.
Best regards,
Philip

[Philip Postkasten]

I just noticed that looping the status inside the job class is a bad idea since it's cpu heavy. Already one job puts cpu to 99%.

Furthermore I realized that the idea of my Job class is very similat to sidekiqs pro Batch class which can supervise the status of subsequent workers. The pro gem is no option sinde I dont have budget. Is there another way to monitor related workers in my Job class?

[Mike Perham]

It is very common for people who need the Batch functionality and have no budget to roll their own solution using sidekiq-status.  I don't know the answer to your questions though.

Mike

On Mon, Jun 15, 2015 at 2:40 PM, Philip Postkasten <philipsbr...@gmail.com> wrote:

I just noticed that looping the status inside the job class is a bad idea since it's cpu heavy. Already one job puts cpu to 99%.

Furthermore I realized that the idea of my Job class is very similat to sidekiqs pro Batch class which can supervise the status of subsequent workers. The pro gem is no option sinde I dont have budget. Is there another way to monitor related workers in my Job class?

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To post to this group, send email to sid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/827dcf82-7a2b-4014-8ed7-31d7bfdb34a1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mike Perham – CEO, Contributed Systems

Smart, effective open source infrastructure for your apps.

[Philip Postkasten]

Thanks for your answer. I have some ideas about how to get the status of the child jobs.
Nevertheless, what about the issue with the "queued status" which occurs in inline mode?

"I started writing tests I by doing this in development.rb
 require 'sidekiq/testing/inline'
The problem is that the Sidekig::Status(jid) in perform is allways return :queued."

I also check the status directly on redis which is :queued as well

This confuses me since the doc says : An inline mode that runs the job immediately instead of enqueuing it

[Mike Perham]

That's a sidekiq-status question which I can't answer, knowing nothing about the internals of the gem.

Mike

To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/9ee81131-5dbb-491d-81fb-9384e80a829a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.