[rabbitmq-discuss] Query - Batch Consume

[Abhishek K]

I am just checking out RabbitMQ for use at a small company.

I need to know if there is Batch Consume Option and even batch publish .

(Basically instead of sending messages one by one, the messages can be sent to the Consumer in one network transfer)

Is there any plugin available for it.

It would be great if you point me to some links.

Also can you give me some links /guides for Clustering ?

Thanks 

Abhishek Kona

[Andreas Jung]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Abhishek K wrote:
> I am just checking out RabbitMQ for use at a small company.
> I need to know if there is Batch Consume Option and even batch publish .
>
> (Basically instead of sending messages one by one, the messages can be
> sent to the Consumer in one network transfer)
> Is there any plugin available for it.
> It would be great if you point me to some links.
>
>

Does one really need that? RabbitMQ is very, very fast and batching is
usually not needed...how many data do you need to store in "batch"?

- -aj
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxRYf0ACgkQCJIWIbr9KYzMoACfaKRpLmpGn7n+jkceCQYJGod2
sBgAn0VQf7jFkIs64/iwsBh8MnLAOhu3
=0ltN
-----END PGP SIGNATURE-----

[David Wragg]

Hi Abhishek,

Abhishek K <abhish...@gmail.com> writes:
> I am just checking out RabbitMQ for use at a small company.
> I need to know if there is Batch Consume Option and even batch publish .
>
> (Basically instead of sending messages one by one, the messages can be sent
> to the Consumer in one network transfer)
> Is there any plugin available for it.
> It would be great if you point me to some links.

Batch operations are normally useful in a protocol with synchrnous
operations, so that rather than having to wait for a response on each
operation, a client can perform many operations and only wait for one
response.

But the publish and consume operations in AMQP are not synchronous, so
it's not clear what efficiency improvements could be allowed by batch
operations.

> Also can you give me some links /guides for Clustering ?

See the clustering guide at <http://www.rabbitmq.com/clustering.html>.
If you are interested in clustering, you probably also want to look at
the active/passive failover guide at
<http://www.rabbitmq.com/pacemaker.html>.

David

--
David Wragg
Staff Engineer, RabbitMQ
SpringSource, a division of VMware
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq...@lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[David Wragg]

Abhishek K <abhish...@gmail.com> writes:
> But will not batching reduce network transfers?
> I mean if there are 10 items in a queue. Can a consumer consume them in one
> network transfer?
> Can something like that be done.

AMQP already permits an AMQP server to deliver many messages to a
consumer in a single network transfer (unless the consumer does
something like setting qos which would prevent this).

The RabbitMQ broker does not currently strive to make this happen - it
looks like it will do one write to the socket for each AMQP frame. But
the operating system TCP stack might coalesce several socket writes into
a single IP packet. I haven't looked at packet traces to check whether
this actually happens.

So RabbitMQ is either already doing what you suggest, or could be
optimised to do so, without introducing batch operations. This is a key
advantage of an asynchronous protocol.

[Matthew Sackman]

On Thu, Jul 29, 2010 at 01:25:11PM +0100, David Wragg wrote:
> The RabbitMQ broker does not currently strive to make this happen - it
> looks like it will do one write to the socket for each AMQP frame. But
> the operating system TCP stack might coalesce several socket writes into
> a single IP packet. I haven't looked at packet traces to check whether
> this actually happens.

We turn Nagel off by default to reduce latency. You can always turn it
back on to allow the kernel to minimise packets.

Matthew

[Maxima 120]

I do but for different purpose. Imagine consumer does a long running work on a message (say 1sec). And imagine in the system quite often 10 events happen in a span of 10ms. All of these 10 involve the same object to process. So these events will result in doing exactly same work 10 times. Because Rabbit will deliver 10 messages one by one..

So there are 2 solution - write complex messy logic with measuring timestamps, making batches to coalesce messages which involve the same work.. Maintain and support it, copy / paste in future similar apps...

OR

Rabbit team writes a new feature. Say another BasicQos(int batchTimeout) or whatever and implement it once and for all...

Which way do you think is 'appropriate' on many levels?

[Nathan Long]

I have a use case for batch processing, also.

System A sends about 1k messages per second to System B. The main thing System B needs to do is to write those raw messages to a durable database.

PostgreSQL is unable to write 1k records per second individually in our tests. It *is* able to write that may records if we insert multiple rows at a time. So our options are:

- Receive a message, ACK it even though we haven't actually stored it yet, keep it in a buffer, and when we have N messages, write them all to PostgreSQL in one shot. This is bad because we're ACKing things we haven't reliably stored, so we may lose them.

- Switch databases to something that can write faster

- Switch message queues to something that can give us a batch of N messages which we can store and then ACK as a group.

[Agha Usman]

Hi all, 

Has this been resolved? 

I too have a situation where I need to consume messages in bulk so I can process them at once. I tried using BasicQos and set the limit to 100 which works fine if I run 2 worker process one after another but if both processes are running, the messages split up between the two. 

Although I'm still getting message one by one, I reckon it is how it is. I've written logic in my consumer code to create a batch of messages and then process it all.