Looking to change the message queue we use

[Kelly Sommers]

I think I need to start thinking ahead and investigate possible options for changing our current message queue choice that we chose for our communications strategy. I made the choice last year and I went for a pub/sub solution that would help us compose an extremely decoupled communications strategy very easily. Unfortunately my choice has been pretty unstable so I want to start looking for options.

Here are the pros/cons I identify with the current choice:

Pros:

• No message broker. No traffic bottleneck since no broker exists. 
• A subscriber service propagates dynamic subscriptions to all nodes so when a node publishes a message it can send it directly to subscribers queues.
• Since message traffic is peer to peer, if the subscriber service goes down traffic still continues to work normally.
• A node can subscribe to a single message.
• Extremely easy to introduce a new service that needs to subscribe to some messages.
• Each node only needs to be configured to know the subscriber service. No other configuration required.

Cons:

• Subscriber service is a single point of failure if a majority of the services restart while the subscriber service is down.
• If a new node or service is introduced while the subscriber service is down it's essentially "orphaned" because nobody knows it exists.
• Since there is no broker any advanced message filtering has to occur on the receiving endpoint.

I haven't seen many pub/sub message queue middle-ware where publishing is peer to peer and only subscriptions are brokered. Are there any others? I am starting to think that this isn't a common design.

I still value making it extremely easy to compose communications between distributed systems but I'm starting to wonder if the ease of composing communications can be achieved in other ways.. Is service discovery with a directory a better option where services register themselves into the directory?

As far as durability/reliability needs, I can probably follow up with some more questions once I have some choices to investigate.

What have you found works to make composing communications extremely simple? Any questions, feedback or suggestions would be greatly appreciated :)

Thanks!

Kelly

[Joe Stein]

Have you looked into Kafka? http://incubator.apache.org/kafka/design.html

--
You received this message because you are subscribed to the Google Groups "Distributed Systems" group.
To post to this group, send email to distsys...@googlegroups.com.
To unsubscribe from this group, send email to distsys-discu...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop
*/

[Kelly Sommers]

I've heard of it but haven't looked into it yet. Any thoughts on if/how it makes composing communications simple and what it might be good or bad at?

I'll give the link a read. Thanks!

[Joe Stein]

pros:

• very, very, very fast http://incubator.apache.org/kafka/performance.html
• scales out linearly
• automatic discovery of brokers
• lots of support from community
• used in production in lots of places (Tumblr, LinkedIn, Urban Airship, Medialets) https://cwiki.apache.org/confluence/display/KAFKA/Powered+By
• multi language support (Python, Ruby, C, Java, Scala, make your own...)

cons:

• no replication yet (coming in 0.8) however has it has mirroring if required https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring

some more good reading

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations

[Roy Cornelissen]

Hi Kelly,

Would you care to share the current messaging solution you're working with and the instability issues you ran into?

I'm currently looking into different queueing solutions to cover different (decoupled) messaging scenario's. I think it's on a much smaller scale than you're looking at but still the same aspects.

Roy

[Jérémie Chassaing]

I wrote my own P2P pub/sub service based on a subscription service and had originaly the same pro/cons.

 

I made a few changes over time to mitigate the problems :

- There's 1 subscriber service / computer. Connection is always to a local subscriber service that synchronizes subscriptions with other subscriber services.

- Subscriber service persist subscriptions

- Services remain subscribed even when stopped/killed, except if they unsubscribe explicitly. This way they continue to receive messages in their queue even when stopped and can do the processing when restarting.

- Services persist locally the last known subscribers. On start they can continue the job even if the subscriber service is unreachable. A warning is still raised, because it mean they risk not to be notified about new subscribers.

 

For the last point (advanced filtering), Udi would say that requiring it is a sign you have a event/command definition problem...

 

If you have control over evolution of your pub/sub system, it's possible directions.

 

jeremie / thinkbeforecoding

[Kelly Sommers]

What we are using now pretty much does what you said below already but it just hasn't been stable in production especially as we scaled it out to a much larger cluster. We've also found that approach difficult to use to do no down time upgrades where you stand up the new service version in parallel while still handling requests.

I've started to consider that maybe instead of looking for queueing technologies that implement complex pubsub mechanisms that dynamic service discovery would be better and simpler. I stumbled upon this:

https://github.com/mbknor/ddsl

Interesting thing about this approach is, it makes discovering versions of services and standing up parallel services with different versions much more simple.

As far as filtering goes, there's always a time and a place for something. Content based routing can be very helpful in some situations. We don't use this capability at the moment but it's still a pro/con depending on the situation.

Thanks,

Kelly

--
You received this message because you are subscribed to the Google Groups "Distributed Systems" group.
To post to this group, send email to distsys...@googlegroups.com.
To unsubscribe from this group, send email to distsys-discu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/distsys-discuss/-/0FPHRpcaTYcJ.

[Andreas Öhlund]

> Since there is no broker any advanced message filtering has to occur on the receiving endpoint. 

We've been lacking this feature in NServiceBus since day one but In my experience those filtering needs can usually be avoided by a better message design (like  jeremie pointed out)

> Content based routing can be very helpful in some situations. We don't use this capability at the moment but it's still a pro/con depending on the situation. 

The same goes for CBR (probably just me beeing a recovering Biztalk dev:) in my book that is very often a design smell that again can be easily solved with a more explicit message design.

http://www.udidahan.com/2011/03/20/careful-with-content-based-routing/ 

That said I have seen this being somewhat usefull in early stages of brownfield project where you need to "tap in" to existing integrations just to get a foothold for introducing a more well designed messaging solution. I would never do CBR in a greenfield project though.

Cheers,

Andreas

[Nick Telford]

Another con:

• Designed for low-cardinality messaging; i.e. few topics

This is a design trade-off that's primarily driven by the expense of seeking on a disk. The more topics you read/write from concurrently, the worse the performance is going to be.

Also, I wouldn't consider "multi-language support" as a pro: so much of the logic is handled on the client-side that unless you're using the Java/Scala API, you risk things not working properly at all.