How to subscribe multiple topic/channel pairs through a single connection?

[Rain Zen]

I have multiple topics, and I want to assemble all messages from them through on connection, is it possible?

[Jehiah Czebotar]

No, currently there is a 1:1 relationship between connections and the topic/channel that connection is subscribed to. This is fairly intentional to avoid implementation complexity.

If you find yourself wanting to subscribe to a large number of topics at the same time, often it's a useful signal that you might be better off consolidating your topics and pushing the more granular description into the messages themselves. For example, at Bitly we have several general data streams that contain a wide variety of messages, but each message has a well structured "action" key in it. Some consumers might need a few of the actions, but some need them all. By consolidating and pivoting on the messages themselves we avoid managing hundreds of discreet topics for each specific event.

-- 

Jehiah

[Brian Dennis]

Sorry to latejack this thread. I take this to mean that a pynsq (or any other?) client that needs to listen to large numbers of topics is upper bounded by socket resources on its host. This is even though the TCP protocol docs do not prohibit multiple SUB requests on a connection, that I can see.

I'm currently in a use case where we want to do some capacity estimation before figuring out how to merge topics or take another approach.

Thanks!

[Matt Reiferson]

The TCP protocol does inhibit multiple SUB, I think (at best it's undefined).

But yes, bound by socket resources.

Practically speaking having that many topics is an NSQ antipattern.

[Brian Dennis]

Is there an actual degradation in nsqd performance with a large number of topics? Seems like this can also lead to a lot of traffic for a set of nsqlookupds.

[Matt Reiferson]

We've done a lot of work to reduce the overhead and ensure scalability (removing lock contention, etc.) so it should mostly be bound by system resources.