Pub/Sub vs. Request/Reply

[Ed Henry]

I'm pretty all in on pub/sub schemas when it comes to dealing with tying multiple systems together and attempting to tool on top of it.

Non-linear integration is something that also comes from a pub/sub model as opposed to request/reply. We can service a single part of the entire system without impacting any other systems that are linearly integrated, directly.

There is a lot of overhead in the request/reply model with respect to tying systems together as well, especially ones with multiple points of administration - like disaggregated network gear. I've seen this in some of the 'polling' integrations that I've built so far.

I suppose there is an active / passive portion to this conversation as well wrt how you want the integrations to function.

So - I know this is pretty high level, but what has everyone else's view points been on something like this when building tooling?

[Matthew Stone]

It's pretty widely accepted to use pub/sub format for time consuming tasks. This doesn't remove the need or value for sync based messaging though. Internally we use a MQ for our desktop management system because most of those tasks are very long running and we don't need an immediate response. For other tasks we use RPC or REST. With PUB/SUB you have to do additional work to confirm the message was received and that action was taken. With sync based com you don't. You can expect a response immediately. So their are tradeoffs on each.

Also remember that in the end sockets are created. Messages are sent. Sockets are closed. Both pub/sub and sync are used at very large scale. It's one of those use case type things. Use the right tool in the toolbox.

I'd like to see some of the code you've written around this. I can't publish our internal code, but I can mock something up easily enough to show high level how we use it. Then we can compare notes.

[Ed Henry]

I can't post any of the code I've written around this either, but I will happily mock something up in the coming days.

There are different consistency / concurrency models and normalization techniques that can be used within a pub/sub vs. request reply as well. 

Policy application point can shift in a pub/sub model as well - something like this is what I've seen : 

Service publishes data to distributed table -> table entries are replicated throughout the system -> subscribers see update for table rows subscribed to -> policy is instantiated according to logic that is defined locally by some sort of policy engine service.

I agree with the right tool in the toolbox sentiment as well.

I'll rough out something in the next few weeks and we can definitely compare notes.

[Matthew Stone]

What type of data are you trying to work with that needs what you've described here? I assume it's not infrastructure type data. Also this sounds like you're wanting to create your own pub/sub infrastructure, correct?

[Ed Henry]

I've built and continue to help build something along those lines.

Curious though as to why your assumption is it not being infrastructure type data?

[Matthew Stone]

Just curious what problem you're trying to solve that's not been solved by existing means without the additional overhead of building your own. RabbitMQ has been shown to handle over 1 million messages/sec. Multi-tier RPC is similar.

[Matt O]

I like turtles.