Proper usage for large number of queues

[Patrick Morris]

I have a system with about 1000 queues that I am trying to get working with NSQ. 

This system has 5 nodes each running nsqd and each node has about 100 microservices.

Each microservice handles specific work in a handful of queues.

For a random node nsqd.exe has about 1400 TCP connections.

Each microservice has been about 50 to 150 TCP connections.

Overall this is a huge amount of TCP connections and is alarming.

I don't know if it is library related however I am using the standard NSQ binary with the ZeroNSQ c# library.

Does NSQ work for scenarios like this?

[Pierce Lopez]

NSQ requires a separate connection for each (topic,channel) subscription from a consumer to an nsqd node. So if one service creates a consumer object/entity for 10 different (topic,channel) combos then that's 10 connections, even if they're all to the same nsqd. But also, a consumer for one (topic,channel) which finds that the topic exists on 10 different nsqd, will make 10 connections, one to each nsqd. If the service consumes from 10 different (topic,channel) and each topic is present on 10 nsqd then that's 100 connections.

So, "lots of connections" is probably not because the client library is doing something wrong. If each of the 100 microservices can find the topics it wants to consumer on all 5 of your nsqd nodes, then that explains 500 connections to each nsqd ... but only 5 or 6 connections for each service (assuming 1 for writer to localhost nsqd). So I'm guessing each microservice is consuming between 10 and 30 (topic,channel) that could be present on all 5 nsqd, that could explain 50 to 150 connections per microservice.

It's more typical for there to be fewer topics/channels but a higher volume of messages per topic, but this many connections is not necessarily a problem. I'd check on nsqd cpu usage - it maintains timers for each connection, and that can add up to some noticeable overhead. But at this level it might still be quite OK.

- Pierce

[Patrick Morris]

Thanks for the explanation. 

Based on this it appears that the horizontal scaling is limited unless you have a disjoint environment.

Is this correct?

[Pierce Lopez]

Horizontal scaling of just a few queues is pretty good, that was actually one of the main design goals of NSQ. You can have lots of producers and consumers of a single queue over a couple hundred servers, and there is no single bottleneck. If you have 1000 queues and they're all going everywhere then yeah it's not as good for that.

- Pierce

[Patrick Morris]

Certainly makes sense.

Thanks again.

[Matt Reiferson]

Hi Patrick,

Perhaps if you described the use case a bit more we might be able to provide some insight in how to accomplish that with NSQ.

[Patrick Morris]

Sure.

I am building a distributed system based on cassandra.

This system of course has no joins so data needs to be distributed sometimes to several locations to ensure consistency.

In order to achieve this there is a common distributed data service which takes all changes and puts them into 3 initial queues.

EntityChange

EntityFieldChange

EntityMerge

An operation is not complete until it is in the DB and message in queue.

Each of the 3 queues is then read and broken into any number of sub queues.

For instance if EntityFieldChange indicated field X was changed then queue messages for the appropriate tables would be created like:

EFC_Table1_FieldX

EFC_Table2_FieldX

Lastly each of those messages can be broken into further components to help distribute some of the extra processing.

For instance 1 DB change could require 100k records to be altered.

There are:

3 level 1 queues

~400 level 2 queues

~300 level 3 queues

I am currently using the same name for topic/channel which may be a bad thing in NSQ. 

However I didn't want to rely on an implementation detail like that in case I needed to change queues.

Some queues are heavily changed while others are barely touched.

I am open for any suggestions you may have.

Thanks.

[Pierce Lopez]

Channel name doesn't really matter to performance, what you have is fine.

Is this so that services can subscribe to just the fields (columns?) they are interested in? Or so a separate service does the processing for each field? I would consider just a topic per table, and publishing/subscribing at table granularity instead. How much this helps depends on what consumers are doing with these messages ... consider also that you can have multiple instances of a service consuming from a single channel, sharing the work of processing the messages across multiple servers. Also, a consumer filtering out a significant fraction of messages which are not interesting to it (instantly finishing them) can be pretty fast. Also make sure your consumers max-in-flight is high enough.

- Pierce