topics vs channels?

[SirVon Thomas]

first off this system looks awesomely simplistic to the point where I think you guys might be on to something very big, when more ppl get creative with the use case, its gonna be a go. no pun intended.

But help me on the understanding of topics and channels because im working with the same concepts in my projects, so my psyche has already built up a concept of channels thats baked in.

so help me out with your vision....

whats the advantages/disadvantages of having many topics lets says 10000s of and 1 channel on each VS 1 topic with 10000 channels with in?

the use case is almost giving an individual their own topic vs giving them their own channel.

this almost plays in to the same internal debate/question, I have with myself on collections vs documents in a nosql store.

thank you for your work of tech and your time.

[Matt Reiferson]

Think of topics as distinct streams of messages… They describe the data in the stream.

Topics have one or more channels.

Think of a channel as a logical group of consumers of a specific topic.  The channel gets a copy of each message published to a topic.  Within a channel, messages are distributed to subscribed clients.

[Matt Reiferson]

The following slides should help clarify this:

https://speakerdeck.com/snakes/nsq-nyc-golang-meetup?slide=25

(or start from the beginning)

or this video of a similar talk http://www.youtube.com/watch?v=IkU8JsxdCAM

[SirVon Thomas]

so in my use case of individualizing msgs/data, I would want many unique topics....

so can nsq handle 10,000, 300,000, 500,000, 1million+ topics?

how many instances of nsqd would be suitable production wise for 10,000s of topics?

So the purpose of channels is to evenly distribute those msgs?

if im looking at this system like a cable network, Time Warner.

topics = ESPN, HBO, MTV, TNT,...

channels = ????

 

the part i dont get is that in the slide you have different channels of a topic assigned different titles

topic A ----> channel "videos", channel "pic", channel "text, channel "audio"

but you said that channels get all the msgs, that means that channel "text" is going to get the same msgs as channel "audio"?

so what? on a client thats consuming topic A channel "video", you say, even though all these msgs (text,video,pic,audio) are coming in, i just want "video" related msgs?

thank you for your time and clarity

[Matt Reiferson]

If the messages are truly unique, yes, you would have many individual topics.

Channels make it efficient for independent groups of consumers to subscribe to a given topic.  They also provide isolation from the effects of any neighboring channels that are behind.

As I mentioned earlier… topics describe the data in its stream, so in your examples “videos” “pics” “text” “audio” are topics not channels.

Channels describe the consumer(s).  A more concrete example would be:

“videos” (topic)

      |

      |——— “thumbnail” (channel)

      |

      |——— “archive” (channel)

      |

      |——— “downsample” (channel)

    

In this example the three independent channels each receive a copy of all the messages for the videos topic and each perform some action.  For example, if the cluster that processes thumbnails is having issues the other two channels are isolated and can continue processing normally.

From what I gather you intend to target (or directly route) messages to consumers.  This wasn’t a primary use case for us when we designed NSQ, so it isn’t something that has first class support.  However, there are ways NSQ can be structured to support that.  Is that what you’re looking to do?

[SirVon Thomas]

yes, thats what from the docs and for my needs and from reading alot of other msging suites, it seems this project might excel at.

considering also 2 other ppl in the group desire to use it in a similar fashion.

Is there any way, you being the creator, can fashion it in that direction or atleast guide us to what is missing from it being used in that manner.

Once again, this piece of art seems like a diamond in the whole msging paradigm rough.

I'm setting it up now and plan on testing my use case.

[Matt Reiferson]

Given the fact that a few have asked similar questions, perhaps this warrants a longer blog post on ways this could be implemented (or NSQ improved to facilitate it).

Thanks for your feedback...