Suitability of Nats-streaming

[Roshan Borale]

Hi All,

I am evaluating NATS-STREAMING for a business use case. Could you please help me with your experts inputs if NATS is a good choice for my problem domain.

1] We have to sent metric data from the application server to the  centrallized server. So for that we will be having nearly 10-20k producers each one on one application server publishing data on same channel.

2] This channel will have queue subscription on the other end to process the message.  

3] The data throughput won't be constant and it will be appox 100 msgs/sec from one producer. Hence It could go to 100*20000= 2 Million and can even go beyond it.

4] I want to know with this scale of load and It can even grow ahead, will nats-streaming be good choice?

We are thinking to deploying it in Fault tollerent mode, with file storage.

Kindly help me with some notes on the feasibility.

thanks, Roshan.

[iv...@synadia.com]

Hi Roshan,

Thank you for considering NATS Streaming.

I don't think you will be able to achieve 2 Million messages per second on a single NATS Streaming Server (I understand you mentioned FT, but the standby is just there to take over should the primary fail). Any chance you could have multiple FT servers, each handling a part of the 20K producers?

You did not mention how big a message is, and that will also come into play. And how fast the storage is going to be. A spin disk is not an option at this type of speed/volume: if your consumer(s) are doing anything with the message, it is likely that they will start to fall behind, which will cause the server to have to read back from disk and having to go earlier in the file to read message to deliver to the subs and then have to go append the new messages at the end of it.

Then will come the issue of channel limits. In NATS Streaming, messages stay in the message log, even after being consumed. They are solely removed based on the administrative limits that are set on the channel.

Forgive the question, but I have to ask: do you need persistence? Do you expect end-to-end confirmation that the message was processed? As you know, NATS Streaming has the publisher and subscriber decoupled. The confirmation that is sent to the producer is simply that the message has been persisted, not that the message has been properly consumed. If you are looking for end-to-end, you may want to user core NATS and use request-reply.

If persistence is required, and if the message-log pattern is not fitting with your use case, you could have a look at JetStream, our new persistence offering that is about to be GA with clustering support (already available from the main branch in single mode). Since you were planning on FT, I think that the single mode is the ok for you so you could also evaluate that. Unlike NATS Streaming, JetStream provides a work-queue semantic where messages are removed once they have been consumed and ack'ed.

To be honest, I am not sure if JetStream would be able to achieve 2M/sec either, but again, you would have to test on your target hardware, with production-like messages and processing time, etc...

Hope this helps,

Ivan.