PubSub Options

[Paul Tiseo]

So, I am building a server that has to do a fair amount of data integration/munging/reporting. For example, it might have a Task that fetches data from a third-party service, then transforms it, then pushes it into a different system.

Now, I know I can use the built-in node-resque subsystem to enqueue tasks, but I'd like more of an event system. I don't want to enqueue the task transformData(), but rather signal the DataReady event.

Has anyone rolled in a pubsub system into their server? If so, what system (internal library or external system) and how?

[Clay Himmelberger]

Redis has built in pubsub, http://redis.io/topics/pubsub. Which is what is used in actionhero across clusters: https://github.com/evantahler/actionhero/blob/2b003879bfd64256046c9832e71d7b5b296aa1da/initializers/redis.js#L108 You could use that for your needs - create a few channels, and have certain (all?) servers subscribe to that data.

Or similar node libraries to keep that contained: https://www.npmjs.com/package/node-redis-pubsub

Other "MQ" (message queue) systems exist as well. RabbitMQ (https://www.rabbitmq.com/) or ActiveMQ are popular, and a quick search found this node module to help facilitate those: https://www.npmjs.com/package/stomp-client. 

In my opinion, without knowing much of your goals, if it's just a matter of kicking off events, the tasking system would be sufficient. If one event === one task to run. All the messaging stuff is useful if multiple endpoints need to handle the same message, and/or there is a dynamic element to who is getting messages when. If eveything is always online, it feels like tasking. 

Keep us posted to what you find out : )

Clay