NATS clustering limits

[tcurtiss]

Hello,

I'm considering NATS for signaling on a network with a large and growing client base.

My concern is the fully-connected mesh topology of NATS clustering, which should cause message traffic to increase quadratically with the number of nodes.

Are there benchmarks that explore the capacity limits of NATS clusters?

I'd like to have an estimate of the number of client connections that a NATS node can handle, on a baseline Linux instance, for example 4 GB RAM and 4 CPU cores, or some other Linux baseline instance.

Also, I'd like to have an estimate of the number of Linux baseline nodes that can be NATS clustered, and fully loaded with message traffic, before the NATS cluster begins to fail.

Thank you.

[Colin Sullivan]

Thank you for your interest in NATS!

We do not provide benchmarks for clustering (we do for basic clients though).  There is so much variability in the physical network and hardware resources, any numbers we provide would end up being specific to a particular environment.  There are other factors to consider as well, such as average message size, messaging patterns, how often clients connect and disconnect, and subscription activity.  So, "your mileage may vary", based on your requirements.

That being said, I suggest you setup a test to reflect your environment.  It is very easy to setup a NATS cluster - details can be found here.  I'd suggest starting simple, then if necessary, there are additional ways to optimize based on subject interest propagation.

Can you share more details about your use case?

Thanks,

Colin

[tcurtiss]

Colin,

Thanks for the prompt reply.

In my use case, NATS would be used for signaling in a media network, with a fairly large amount of signaling traffic per client.

Does all the message traffic coming into one NATS node, get 100% propagated to all the other nodes in the entire NATS cluster?

Or does each NATS node maintain the updated subscription state of all the clients connected to the other NATS nodes in the cluster, so that each NATS node propagates traffic to other NATS nodes, only if the other NATS nodes have open subscriptions to specific message traffic?

If the latter, is the mechanism configurable by the network admin, or is it hardwired into the code?

On that same idea, if one node in the NATS cluster wants to know the open interest in a subject throughout the entire NATS cluster, does it have to poll every other NATS node in the cluster, or does it simply look at its own state record for the open interest in that subject?

If the latter, I couldn't find a way to do this in the NATS documentation, but maybe it's there and I missed it.

Thanks,

TC

[derek]

The interest graph propagates automatically across servers so messages are only sent when they are needed on the remote server and they are sent once regardless of the interest on the other side. Well almost, queue subscribers could cause it to be sent twice but that is the maximum.

Since interest propagates automatically at subscribe time there is no need to poll, servers know which other servers they need to forward the message to.

[tcurtiss]

Derek,

Very well that the interest graph propagates across the server cluster, automatically upon subscribe.

If a client wants to know the total number of clients subscribed to a subject across the server cluster, can the client get that data from the interest graph of its NATS server? And if so, what is the method?

Can the NATS servers be load balanced with an Nginx reverse proxy, via the ip stickiness of Nginx?

If not, what is the preferred method of load balancing the NATS server cluster?

Thanks

[tcurtiss]

Colin, or anyone,

If a client wants to know the total number of clients subscribed to a subject across the NATS server cluster, can the client get that data from the interest graph of its NATS server? And if so, what is the method?

Can the client load to a NATS server cluster be load balanced with an Nginx reverse proxy, via the ip stickiness of Nginx?

If not, what is the preferred method of load balancing the client load on a NATS server cluster?

Thanks

[Ivan Kozlovic]

If a client wants to know the total number of clients subscribed to a subject across the NATS server cluster, can the client get that data from the interest graph of its NATS server? And if so, what is the method?

When starting a server with the monitor port enabled `-m <port number>`, the NATS Server can provide statistics for different endpoints. For instance, if server runs on host `nats_host` and has been started with monitor port 8222 `gnatsd -m 8222`, you could check the subscriptions with `http://nats_host:8222/connz?subs=1`. It would return something like:

{
  "now": "2016-09-23T12:24:49.185477595-06:00",
  "num_connections": 2,
  "total": 2,
  "offset": 0,
  "limit": 1024,
  "connections": [
    {
      "cid": 11,
      "ip": "::1",
      "port": 58903,
      "start": "2016-09-23T12:24:47.705340926-06:00",
      "last_activity": "2016-09-23T12:24:47.705756232-06:00",
      "uptime": "1s",
      "idle": "1s",
      "pending_bytes": 0,
      "in_msgs": 0,
      "out_msgs": 0,
      "in_bytes": 0,
      "out_bytes": 0,
      "subscriptions": 1,
      "lang": "go",
      "version": "1.2.2",
      "subscriptions_list": [
        "foo"
      ]
    },
    {
      "cid": 12,
      "ip": "::1",
      "port": 58904,
      "start": "2016-09-23T12:24:47.705492986-06:00",
      "last_activity": "2016-09-23T12:24:47.70579662-06:00",
      "uptime": "1s",
      "idle": "1s",
      "pending_bytes": 0,
      "in_msgs": 0,
      "out_msgs": 0,
      "in_bytes": 0,
      "out_bytes": 0,
      "subscriptions": 1,
      "lang": "go",
      "version": "1.2.2",
      "subscriptions_list": [
        "foo"
      ]
    }
  ]
}

 

Can the client load to a NATS server cluster be load balanced with an Nginx reverse proxy, via the ip stickiness of Nginx?

If not, what is the preferred method of load balancing the client load on a NATS server cluster?

You can use any type of load balancer.

But note that with NATS Servers running 0.9.2 and above, the servers send to clients (when they connect) the URLs of servers in the mesh clients can connect to. So in case the server a client is connected to fails, it can automatically reconnect to one server from that list.  

Ivan.