How to increase the number of processes managing idx files?

[DuaneA]

Hi All,

Evaluating this software for possible use...

I'm looking for a way to increase the number of processes managing the idx files in the message queue...

I saw a reference somewhere stating that only 3 processes run to manage the idx files by default.  I can't seem to find that reference back.

Any help would be appreciated!

Thanks,

Duane

[Luke Bakken]

Hi Duane,

Is there a particular limitation you're trying to address, or some other issue?

Thanks -

Luke

[Michael Klishin]

I'd like to confirm what those "index files" are but on 3.7 and 3.8 each virtual host

has its own pair of message stores. There is no way to increase the count as, well,

there is one store per message persistence type.

[DuaneA]

In our implementation, there could be situations where the generating system may not have internet access for days/weeks.  (We are looking to implement this with a durable queue and persistent messages...  The queue would live on the generating system...)

In my testing, I found as I added more messages that aren't being consumed, the app appeared to slow down...  (My tests were run on Windows with C#...)  I was testing with about 10M messages...  (I realize that is overkill but I wanted to see what would happen...)

I guess I'm confused on how to configure this...  Is there a white paper/book I can reference?

Thanks!

Duane

[Luke Bakken]

Hi Duane,

It's important to remember that RabbitMQ isn't a database or object storage system. While you can let queues grow and grow over time it will have adverse effects that you have to account for.

The first thing you need to do is ensure all of the queues that could back up are lazy queues - https://www.rabbitmq.com/lazy-queues.html

Your app could be slowing down because a memory alarm is being hit.

It sounds as though you're planning on leaving RabbitMQ unattended, un-monitored and with growing queues. That's pretty much a recipe for some sort of problem.

Thanks,

Luke

[DuaneA]

Hi Luke,

Thank you for the reply...

We are trying to send data from robots located all over the world...  (Literally...)  Our model would be to have each robot send messages to a consumer located in a data center in the region.  While this data does come at regular intervals, it is quite light weight...  (We were looking into DB replication initially which is more robust but requires a lot more management and configuration.)  We do want to be able to scale up as we add new robots by adding more consumers to the region...

We do want to be able to monitor if a robot is not sending data and deal with it in a timely manor...  However, sometimes we have issues with local IPs dealing with network connection issues...  Our goal is not to use Rabbit as a database, but as a way to manage the messages coming from the robot.  (The consumer at the regional level would then write the message data to a normalized database.)

Does that make sense?  Are we biting off more than we can chew here?

Thanks,

Duane

On Friday, October 4, 2019 at 4:22:44 PM UTC-5, DuaneA wrote:

[Michael Klishin]

So you are trying to use RabbitMQ as a temporary local queue that then moves messages "on shore" (I use

this term because there are ships in the sea that do this using RabbitMQ and  the Shovel plugin)?

If so you probably want to use a lazy queue for this.

I'm not sure how the queue index file "processing" is relevant here since no index files can be removed until all messages

are consumed and acknowledged when the system supposedly comes online.

A lazy queue would have more predictable throughput with lower peaks and lower average and peak memory usage.

Such offline systems should be mindful of running their local node out of disk space. Message TTL or queue length limit

are two ways to control runaway growth.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/271dedd8-1a46-4c27-9639-6a07f53a559c%40googlegroups.com.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ