RabbitMQ+Spring Boot getting slow as number of users increases

[Harikrishnan K K]

Hi,

I am developing a Real Time Messaging Server for my organization using Spring Boot as Websocket Server to which users can connect from browser and it uses RabbitMQ as the messaging service. Our primary requirements are minimum latency, fault tolerance and guranteed delivery. Current requirement is around 10k concurrent users, 3k to 5k exchanges.

Each user could connect to at least one of these exchanges. Two message is published to at least 300 of these exchanges per second. So each client connected to the websocket server will receive 2 messages per second.

The RabbitMQ and Spring Boot will be deployed in a Kubernetes cluster for scalability. But, I am testing it using Windows 10 Machine and in a Redhat Machine without using Kubernetes.

Our reconnect threshold is 90 seconds, so if a websocket connection is broken for 90s or less, due to client network or any other issue,  then all realtime data sent during this period should be sent to user when the browser reconnects within 90 seconds. 

That is the main reason we tried using one queue per user subscription so that messages will not be lost.

When ever users connect to websocket server, a unique id will be generated from client, and this will be used to create a queue to connect to a direct exchange. So for each subscription of a session there will be one Queue in the format <uniquesessionid>_<exchange_name> and each queue will have its own listener instance as show below.

Map<String, Object> args = new HashMap<String, Object>();

args.put("x-expires", 90000);

channel.queueDeclare(queueName, false, false, false, args);

channel.queueBind(queueName, exchangeName, routingKey);

        

String tag = channel.basicConsume(queueName, true, 

         new RabbitMQConsumer(simpMessagingTemplate, 

         simpUserRegistry, exchangeKey1,  exchangeKey2 , principalId), consumerTag -> { });

The listener is given below

public class RabbitMQConsumer implements DeliverCallback {

@Override

public void handle(String consumerTag, Delivery message) throws IOException {

DataObj objMessage = fromBytes(message.getBody(), DataObj.class);

String topicToSend = "/topic/exchgdata/"+exchangeKey1+"/"+exchangeKey2;

simpMessagingTemplate.convertAndSendToUser(principalId,

topicToSend, objMessage);

}

The principal id is the unique id sent from front end to identify the user session.

All these set up serves these purposes

1. When a client temporarily disconnects or when a Spring Boot instance goes down, the data is safe in the queue. So when the user reconnects to the same Spring Boot instance or different instance, it will be reconnecting with the same unique id generated in browser (react), so it will connect to the same queue and will get data delivered to the queue during disconnection period.

All these set up works fine, except for performance. Whenever the number of users reaches around 1200 - 1500, the delivery of the messages starts to get delayed. Our requirement is < 3 seconds for 10k concurrent users. But just as number of concurrent users reaches 1200 - 1500 message delivery to client takes > 10 seconds. So it will be much more as we move towards 10k concurrent users.

My question is, is my approach correct? For stopping message loss, is one queue per user per subscription a correct approach? Are there any other recommended approaches for our requirement of minimum latency, fault tolerance and guranteed delivery?