Doubts on how many threads and channels should be dedicated for publish/consume.

[Gianluca Trancanella]

Hi,

I'd like to share with you our use case in order to clear some doubts that i currently have.

The problem:
We want to replicate data across multiple applications (every application stores data in a separate -and potentially different- database).
Every application (depending on some configuration) is the "owner" of a certain type of data, so every other application would contain "replicas" of that specific data.

We thought would be a good idea to solve this problem connecting our applications with a Message-oriented Middleware and more specifically we'll try to achieve this goal with RabbitMq.


Environment:
- All our client apps will use the java library;
- The number of applications is dynamic: in some customer environments could be 3 in some others let's say up to 20;
- The "configurations" and the "data realms" are dynamically defined per customer environment needs;
- Each application will consume and will publish according to the before mentioned environment configuration.


The solution we came up with implies the followings:
- Every application will publish messages every S seconds (for instance: 10). Each message describes some data changes to share with other apps.
- Each queue is bound to a certain "data realm" (Example of queues: "CrmApp-commercialDocsRealm", "CrmApp-anotherDataRealm");
- Every application will consume from N queues DEDICATED TO THAT APPLICATION ONLY (N = the number of realms the app is interested in). Let's say N will likely be in a range between 5 to 20 (env-configuration-depenent)


I tried to describe our case in the most generic manner but if something is unclear or some important information is missing please let me know and i'll try my best to integrate.


Since we care more achieving the goal (data replication) rather then doing it as fast as possible, what do you think about the followings?
A) Some application will probably consume from 20 queues. I dont feel i should dedicate one queue on a single channel on a single thread, but i'm worried consuming all queues on a single channel on a single thread. What would you suggest?
B) I thought each application could dedicate a single thread that will publish all the messages (to different Exchanges, different types of messages, ..). Do you see any problem doing that?


If you spot anything else, every advice would be great!

Best regards,
Gianluca

[Witold Szczerba]

As far as I can tell:

• There is no need to create many queues per application, but many queues come handy in some scenarios when order of things matter. You can create one queue with many bindings and the bindings are determining what messages is this particular queue interested in.

• One queue can be consumed by one or more processes and each process can use one or more channels and each channel has its own prefetchCount and autoAck preference, so you can have parallel consumption in one or across many processes.
  

We use many queues per a µservice when we care about order of events:

• Event queues are consumed one by one by a single process and channel, almost always prefetchCount=1 and always autoAck=false.
• More event queues are used when we organize for concurrency, e.g. customer updates are happening in the "backround" (separate queue) while transaction processing has its own independent pace.
• Query queues are living theirs own lives with autoAck=true and prefetchCount=0 (or some big number) and are processed on a thread pool, so they do not block one another and certainly they do not block events.
  

Hope that clarifies a bit :)

Regards,

Witold Szczerba