How do I setup a callback for on success that spans across projects?

[Altonymous]

I have Project A which needs to create a batch for Project B to process.  When Project B work is complete I want it to call a worker in Project A.  I'm not sure the correct syntax to make this happen?  It seems that batches is expecting all the workers to exist in the same project.

My only alternative thought is to have Project A enqueue work for Project B asynchronously, and when Project B is done asynchronously call a job in Project A.  In other words not rely on batching at all.

Thoughts?  Suggestions?

[Mike Perham]

Your success callback in B should create a job in A.  You can push a Sidekiq job to another Redis or another namespace as explained here:

https://github.com/mperham/sidekiq/wiki/Sharding

Mike

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To post to this group, send email to sid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/5d428c2f-7b51-4515-8d07-72135cbc540b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mike Perham – CEO, Contributed Systems

Smart, effective open source infrastructure for your apps.

[Chris Altman]

Not sure I explained myself correctly.  the on_success is in a batch created by Project A, my expectation is that it should fire when the worker in Project B finishes.  Are you suggesting that I do what I was suggesting in my alternative idea.. which is to have Project B call Project A directly when it's done?  These are using the same redis.  The issue is that Project B is trying to initialize a class in Project A.. and since it doesn't have that class it fails.

To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/CAPjBv7pt8V-gTa_c2hnwKSzN3U62Q3sg3osUwuMrcjmRGW_ufw%40mail.gmail.com.

[Mike Perham]

Oh, I see.  Are you using different namespaces for the different projects?

Batches can't cross Redis boundaries.  Sounds like you are creating a batch in A which has jobs designed to be processed by B?  That won't work unless they are using the same Redis config.  Jobs created in Batch A in Redis A must be processed by Sidekiqs from project A.

To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/CAJU7aEDhCmOKosCo9MPtZKCB6jSz5bGtKeFZOu5wJwmDVOOngQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Chris Altman]

They are both in the same redis and the same namespace.  They are just under two different projects.  i.e. 2 different processes on two different machines.

To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/CAPjBv7rhsiU7_vvpE%3DtfNu%2BKEZ%3D65hGuGbSghbZLny_%3DXYXbSg%40mail.gmail.com.

[Mike Perham]

Ok, do you use named queues to ensure A's sidekiqs don't try to process B's jobs?

Sharing callback code between projects is very tricky.  Perhaps you can create a generic callback handler which can push a job targeted at A:

batch = Sidekiq::Batch.new

batch.on(:success, GenericHandler, 'class' => 'AWorker', 'queue' => 'queueA', 'args' => [])

# This class must be in A and B

class GenericHandler

  def on_success(status, options)

    Sidekiq::Client.push(options)

  end

end

Now the callback can be processed by either A or B but the result will get back to A.

Mike

To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/CAJU7aEDEpv3zqvfnOaNJ6Ae9yPBBhNuNKNs%3DFccNQpCfhJRwbA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Chris Altman]

Mike,

Yes we used name queues to ensure the projects don't process each other's work.  I will give the GenericHandler a try.  Gotta run for now so I won't be able to give a status update, but I'll ask a co-worker to chime in after he tests out the code, or I'll give an update tomorrow.

To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/CAPjBv7q6WsH3iB_Zhiv4CwCx_ZezT5EdWtKWzsG9XV9VP9ahSQ%40mail.gmail.com.

[Chris Altman]

Mike,

Thanks for the help, that appears to get us moving again.  Wish there was a cleaner way to do it, but what works, works... :P

[Scott Steele]

Thanks for the suggestion. Improves my mental model / understanding of how batch callbacks work.

We did need to tweak it a little bit so that a method other than AWorker#perform would be called:

In the AWorker class:

class AWorker

  # ...

  def some_method

    batch = Sidekiq::Batch.new

    batch.on(

      :success, GenericHandler,

      'class' => 'AWorker::OnSuccess',

      'queue' => 'queueA',

      'args' => [argX, argY, argZ]

    )

    # ...

  end

  class OnSuccess < AWorker

    def perform(x, y, z)

      # ...

    end

  end

end