Integration with super stream queues

[Shivanshu Kumar]

Hi team, we are trying to integrate with super stream queues.
We have routing keys in the format supported by topic exchange (using *,#)  but the issue identified while integrating with the super stream queues was that we have routing keys for the messages which cannot be changed due to changes required across multiple producer repos.
Super stream architecture appears to bind us with direct exchange prior to the queues. 
(rabbitmq-stream add_super_stream command generates direct exchange by default)

1.  Is there a way by which we can make use of super stream queues without having to make changes in producer code bases?

2. Can manually making topic exchange and attaching the streams to it with the required parameters  be feasible? Will it be performant for load of 50000 rps of avg size around 3Kb per message?

[Arnaud Cogoluègnes]

Why do you need streams and especially super streams in the first place? Streams are already very fast, have replay capabilities, etc. Super streams are meant to scale out streams.

Super streams have been designed and tested with a direct exchange topology in mind, but it's possible *in theory* to make them work with other types of exchanges.

You mention you cannot change the producer code base, but which protocol are you using, AMQP or the stream protocol? Super streams are fully supported only with the stream protocol, because they require some specific commands that are not available in AMQP.

Note it's still possible to keep producers sending messages the same way, bind streams to exchanges, and use the stream protocol to consume, with or without single active consumer and / or super streams.

[Shivanshu Kumar]

Thanks Arnaud for your response,
1. We have multiple consumer groups consuming the same message and as the rps is increasing with time the classic mirrored queues have not been been performing well. As our consumers perform better when horizontally scaled so for parallel consumption of messages the architecture of super stream queues seemed apt.
2. We are using AMQP protocol for now . Can you please suggest the functionalities which would not be available with AMQP protocol ? I had only come across stream queues having with slower throughput with AMQP protocol 

[Arnaud Cogoluègnes]

For 1, you can consider using quorum queues [1] instead of classic mirrored queues (which will be removed in RabbitMQ 4.0 [2]). They provide better data safety and can have higher throughput for some workloads. If you have several applications that need to consume the same message, streams can indeed be more appropriate. See the overview of streams [3].

For 2, see the "stream core" vs "stream plugin" comparison page [4]. "stream core" is basically "stream on AMQP" in this context.

Quorum queues are the closest to your existing solution, so you can consider them first in your migration. If they don't work like you expect, especially if the same messages need to be read by more and more applications (fan-out scenario in the stream overview), you can consider streams on AMQP: same API, but different semantics (applications may have to adapt though, streams are not queues). If you can't reach your throughput objective with streams on AMQP, you can consider using streams with the stream protocol (high throughput scenario in the stream overview), but applications will have to change.

I don't think you need to consider super streams right away, they are a partitioning solution on top of streams. See the blog posts if you want an overview [5] [6].

[1] https://rabbitmq.com/quorum-queues.html

[2] https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#removal-of-classic-queue-mirroring

[3] https://blog.rabbitmq.com/posts/2021/07/rabbitmq-streams-overview/
[4] https://rabbitmq.com/stream-core-plugin-comparison.html

[5] https://blog.rabbitmq.com/posts/2022/07/rabbitmq-3-11-feature-preview-single-active-consumer-for-streams/

[6] https://blog.rabbitmq.com/posts/2022/07/rabbitmq-3-11-feature-preview-super-streams

[Shivanshu Kumar]

Thanks for the detailed response , very helpful indeed
The difficulty for us is that our consumers are slow and hence we need to run multiple instances of  each consumers (Horizontal pod autoscaler enabled) to accommodate for high incoming rps.
As per my understanding , if we use only stream queues then multiple instances of the same service will read the same events and it will lead to duplication in DB and also we are effectively utilising only the capacity of a single instance of the consumer. 
We are using clustered version of rmq with 2 clusters in different zones.
Can you suggest if using some other architecture here might be more beneficial. Your inputs would be very helpful here 
PFA current architecture :-

[Arnaud Cogoluègnes]

OK, so you must use competing consumers because message processing is slow. If you are unhappy with classic mirrored queues (CMQ) performance, you can try out quorum queues (QQ). Again, better data safety and higher throughput for some workloads. And you're more likely to get help optimizing QQ than CMQ.

But switching the data structure won't help if consumers are slow. RabbitMQ can dispatch messages very fast to queue consumers, even faster to stream consumers with the stream protocol, but it'll always be slow if the consumers are slow. So you should try to optimize the processing. And streams won't help out of the box indeed, because you'll end up processing messages several times if you add more consumers.

Something that can get you more parallelism without changes in consumers is the consistent hash exchange [1]. It can be helpful if you reach the limit of one queue and you want an almost transparent way to partition the message across several queues.

[1] https://github.com/rabbitmq/rabbitmq-server/tree/main/deps/rabbitmq_consistent_hash_exchange