Could I use rabbitmq as an intermediary store?

[Damien R]

I'm rewiring my current architecture and have introduced RMQ.

I have:

Publisher -> RMQ Broker

I then have multiple consumers spread across different regions. Each message is processed (in this case performance checking a particular url) and a result is produced.

My question is about the results. Because they are distributed across many applications, I need a way to aggregate and flush to disk (bulk insert queries to postgres, in this case).

Would it be absurd for me to publish these results to a RMQ queue, and have a set of consumers pull the results down and process them?

[Michael Klishin]

It's something that plenty of people use messaging technologies for ;)

Specifically due to the geographically distributed nature of the problem

I'd recommend researching a star topology where local brokers

move messages around using Shovels [1]. There are discussions

of such architecture in list archives.

Note that RabbitMQ is generally not designed for long term storage. While there are

features [2] that make it handle longer backlogs e.g. when consumers have been down for an extended period of time,

this is something your design should take into account.

1. http://www.rabbitmq.com/shovel.html

2. http://www.rabbitmq.com/lazy-queues.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Damien R]

Thanks! I'll do some research. 

I did run into shovel (when looking at clustering options) and lazy queues.

The other side is batching. Can I batch consume a bunch of messages? 

The alternative there, and appears common, is to instead code up the batching logic in the consumer.

On Wednesday, August 8, 2018 at 5:11:01 AM UTC+7, Michael Klishin wrote:

It's something that plenty of people use messaging technologies for ;)

Specifically due to the geographically distributed nature of the problem

I'd recommend researching a star topology where local brokers

move messages around using Shovels [1]. There are discussions

of such architecture in list archives.

Note that RabbitMQ is generally not designed for long term storage. While there are

features [2] that make it handle longer backlogs e.g. when consumers have been down for an extended period of time,

this is something your design should take into account.

1. http://www.rabbitmq.com/shovel.html

2. http://www.rabbitmq.com/lazy-queues.html

On Wed, Aug 8, 2018 at 12:46 AM, Damien R <dami...@gmail.com> wrote:

I'm rewiring my current architecture and have introduced RMQ.

I have:

Publisher -> RMQ Broker

I then have multiple consumers spread across different regions. Each message is processed (in this case performance checking a particular url) and a result is produced.

My question is about the results. Because they are distributed across many applications, I need a way to aggregate and flush to disk (bulk insert queries to postgres, in this case).

Would it be absurd for me to publish these results to a RMQ queue, and have a set of consumers pull the results down and process them?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

There is no batching operation in any of the protocols we support.

However, you can limit the number of messages in flight [1] and can

do batching in the publisher and consumer. Some projects (I believe Spring AMQP is one of them)

even provide such feature out of the box.

HTH.

1. http://www.rabbitmq.com/confirms.html

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Damien R]

Cheers. Will have a look. Appreciate the help.