NATS for low throughput, millions of topics

[Jack Chen]

Hi,

We're investigating gnatsd/NATS for a pubsub use case where we want to publish and subscribe to millions of topics. Each topic would be relatively low throughput (10/s), but overall thy system needs to handle millions a second. Would gnatsd be suited to this?

I've done some investigating and I'm finding that publishg to many topics is slow and I'm unsure why yet (pprof is broken on OSX, about to try Linux). I've had to modify the Go nats client to decrease the channel buffer size for each subscription for our needs.

Thanks,

Jack

[Derek Collison]

Gnatsd can handle the throughput, and I have tested a server holding hundreds of thousands of subjects, but not millions (normal use the entire subject spectrum is usually in the hundreds, as long as INBOX and ephemeral subjects are unsubscribed). I would be happy to do more testing on this. Was the client subscribing to a large number of subscriptions? Which seemed to be slow, the client or the gnatsd server? Also, any more details you could give on setup, version of Go, etc would be helpful.

--
You received this message because you are subscribed to the Google Groups "nats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to natsio+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jack Chen]

I had two clients, one doing the publishing and one doing the subscribing.

The client seems to be slow (mostly publishing, sending seems fine). gnatsd didn't even seem stressed.

The way I've done it so far, is there's an Overlay interface where I'm trying to implement a NATS-backed Overlay. Overlay provides Subscribe, Unsubscribe and Publish with its own Handler. I already have code using the Overlay interface, so this ideally should be a drop in replacement.

Specifically, our use case is allowing a single client connection to be able to communicate with many devices. Device connections both subscribe and publish to their own topic, and client connections subscribe and publish to all the device topics it has access to. Currently, the NATSOverlay class I've got has two NATS connections for potentially 50k subscriptions. Is this ideal?

Go version is 1.3.1.

[Derek Collison]

Do you have some code that shows the slowdown with the NATS Go client directly, without the overlay? You said publishing, not sending. Can you give me some more information on what you mean there? The publish method on the client, to as many subjects as you want, should be fine and is the least complicated method, so not sure what could be hanging that up at all.

The situation you describe sounds fine and should be easy for the system to handle, but let's dig in more on what might be the performance bottleneck.

[Jack Chen]

Code and benchmark results are here: https://gist.github.com/chendo/4650f71f07dd4d3754f2

Ran it on a Macbook Pro, 2.6Ghz i5, 16GB RAM. Go 1.3.1.

It takes 4.8s~ to subscribe to 50k topics. Profiling showed that it was massive GC overhead due to allocating a 65535 limit channel per subscription. I tried changing the maxChanLen to a smaller value (1024) which dropped the subscription time to 0.5s but it caused the 50k message to a single topic subscription test to stall, so it was probably dropping messages somewhere. Multi topic receive throughput dropped down to 13k though.

Would it make sense to allow the ability to set the maximum channel size per subscription?

[Jack Chen]

I've changed my benchmark and went with measuring the amount of messages received and then measuring the count every second, making it a sustained throughput benchmark. I'm seeing 200k messages a second with 250k topics which looks much better!

Sorry for the trouble.

[Derek Collison]

That seems quite low, thanks for the reference to the code and benchmark, I will take a look.