VerneMQ concurrency model

[PIETRO TOLLOT]

Hi, I am evaluating to use VerneMQ as mitt entry point to a system deployed on kubernetes. I'm load testing the cluster, made of two VerneMQ brokers. Because my load test tools made first the publishing and then the subscribing my question is what will be the real throughput that i will measure in production when there will be both publishing and subscribing in parallel? The question so it is, how VerneMQ manage parallelism,mit has fixed number of threads for receiving and for transmitting so that they not limit each other? Thanks for answering 

[André Fatton]

Hi P.

VerneMQ runs 2 processes for every connected MQTT client: an MQTT state machine, and a queue process. When a MQTT client goes offline but had required a persistent session before, VerneMQ will continue to run an offline queue process for that subscriber.

Those processes are examples of what's known as OTP behaviours/processes. They do not work with threads, but with lightweight processes in the Erlang VM directly.

This does give you complete separation of consumer sessions, and great latency behaviour.

To come to your question: this does not give you independent per-session throughput behaviour. Message throughput is basically CPU bound. So there you go. You cannot just test the maximum load of one pub-sub path  and then suppose you'll get the same for every additional session. OTOH, one pub-sub path's maximum isn't your absolute maximum neither: you will have increased throughput with more concurrent sessions.

Luckily, all of this can be tested quite easily.

Best,

André