Subscribe workers to topics

[Mike Pastore]

Hi there,

I'd like to subscribe some Sidekiq workers to different topics. For example, I'd like to subscribe FooWorker and BarWorker to a "user_updated" topic. When a user is updated, I'll publish the pk to that topic and any subscribed workers will fire. 

I thought about using Redis Pub/Sub, but I don't want any messages to get lost if a worker isn't listening for whatever reason. My current plan is to decorate each worker with the topics it's interested in, and write a simple wrapper method that enumerates interested workers and enqueues them. 

(Ugly, hasty, brutal) mockup:

       class FooWorker

      include Sidekiq::Worker

      SUBSCRIBED_TOPICS = [:user_updated]

      def perform(pk)

        # ..

      end

    end

    # elsewhere...

    SomeThing.publish(:user_updated, my_user.id)

    # elsewhere...

    module SomeThing

      def publish(topic, *args)

        # of course we could memoize a tree of topics and subscribed workers and use that going forward; this is just to illustrate the (il)logic

        ObjectSpace.each_object(Class) do |klass|

          next unless klass.included_modules.include?(Sidekiq::Worker)

          next unless klass.const_defined?(:SUBSCRIBED_TOPICS) && klass.const_get(:SUBSCRIBED_TOPICS).include?(topic)

          Sidekiq::Client.push('class'=>klass, 'args'=>args)

        end

      end

    end

I'll also need to solve these problems, among others:

1. Let workers define optional preprocessors for topic arguments. 
2. Mask off the caller to break cycles, e.g. allow FooWorker to publish to the "user_updated" topic without re-enqueuing itself. 

Both are easily doable, but I didn't want to clutter up the above mockup. 

Anyway, before I burn a weekend writing a brain-dead Sidekiq plugin, I wanted to ping the group to see if I'm missing a more obvious solution—or if anybody knows of an existing plugin or other design pattern that would meet my needs. Thank you in advance for your time!

Mike

[Mike Perham]

In messaging there are two main patterns: queues and topics. These are fundamentally different patterns; Sidekiq is an implementation of the queue pattern. That's why this feels hacky. If you really want to proceed down this path, I'd suggest you look into another messaging system like RabbitMQ which provides persistent topics. It will make things notably more complex but also allow you to have completely separate application subsystems, a la microservices. As you note, Redis's pub/sub is hampered by the fact that it is not persistent.

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/8d453a0a-d2b8-487a-99c1-ac5f773f7527n%40googlegroups.com.

--

Mike Perham – CEO, Contributed Systems

Smart, effective open source infrastructure for your apps.