subscribe to multiple queues in spring cloud stream

[revooru nischal]

Hi,

   As per the best practice guides of rabbitmq it says maximum of 50000 messages per queue should be available. 

But what does this 50k mean? 50k per second or 50k messages ready to be acknowledged or only total 50k messages which should be published in the queue ?

Ref: https://www.cloudamqp.com/blog/2017-12-29-part1-rabbitmq-best-practice.html#number-of-queues

If maximum is 50,000 that means in case of IOT(mqtt) where millions of devices publish some data which needs to be processed or recorded by a single java server. I will split them to publish into multiple routingKeys(topics) based on some factor.  But spring cloud stream allows only one queue to subscribe at a time ?

is there a way I can subscribe to multiple queues on same exchange ? 

I can use concurrency and parallelise processing if I can read from multiple queues.

[Michael Klishin]

50K messages per second for a single queue which is bound to a single CPU core [1].

How many consumers there are on said queue is somewhat orthogonal, although having, say, 50K consumers

on a queue is not going to achieve any better results than 50 consumers or 500 consumers (the exact number

which saturates the throughput of a queue is a matter of benchmarking and monitoring [2]).

1. https://www.rabbitmq.com/queues.html#runtime-characteristics

2. https://www.rabbitmq.com/blog/category/performance-2/

[Michael Klishin]

You can have multiple consumers on a queue and bind multiple queues to an exchange (or, in case of MQTT, have multiple consumers on a topic).

This is covered in detail in the tutorials [1]. There is nothing Spring-specific about this in most cases, although Spring often introduces its own concepts

and it's important to understand how they map to RabbitMQ topologies.

The post covers rabbitmq-sharding which can be used with Spring projects just like any other client.

With MQTT every consumer gets its own queue.

1. https://www.rabbitmq.com/getstarted.html

On Thursday, October 15, 2020 at 3:39:36 PM UTC+3 revooru...@gmail.com wrote:

[revooru nischal]

but one binding in spring queue allows us to connect to only one queue. So even if I use shrading I need to write those many bindings right ??

Following is my spring config 

server:

  port: 8056

logging:

  level:

    root: ERROR

spring:

  cloud:

    stream:

      rabbit:

        default:

          consumer:

            concurrency: 6

        bindings:

          mqreceiver-in-0:

            consumer:

              bindingRoutingKey: test

              exchangeDurable: false

              queueNameGroupOnly: true

      bindings:

        mqreceiver-in-0:

          binder: rabbit

          destination: mqtt_topic_internal_exchange

          group: messages-consumer-group-1

          content-type: application/json

      binders:

        rabbit:

          type: rabbit

          environment:

            spring:

              rabbitmq:

                host: localhost

                virtual-host: test

                password: guest

                username: guest

spring.cloud.stream.function.definition: mqreceiver

spring.cloud.stream.function.routing.enabled: true