MassTransit bulk publish to RabbitMQ

2,980 views
Skip to first unread message

Annie Ju

unread,
Jun 22, 2015, 4:49:50 PM6/22/15
to masstrans...@googlegroups.com
Hi,

I'd like to send messages of type ABC to publish via MassTransit using C# interface to a remote RabbitMQ.
Since there may be many messages coming in through different threads, I'd like to group these messages together and send in one batch at a time, e.g., a list of 10 to 20 ABC's.
Is there a way in MT to configure it to send it in batches, rather than one at a time?

How do you handle large number of publish requests in the publisher? 
I can write the batch processing code, e.g., put in a queue and batch them up when they reach a certain size.

Thanks,
Annie

Dru Sellers

unread,
Jun 22, 2015, 5:04:56 PM6/22/15
to masstrans...@googlegroups.com
What goal do you want to achieve by batching them? What pain are you hoping to solve for?

-d

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/9014c5d2-cefc-4001-95d6-ec6cc7d59267%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Annie Ju

unread,
Jun 22, 2015, 5:12:28 PM6/22/15
to masstrans...@googlegroups.com
Mainly for performance.  
I'm concerned each publish request will take some time over the network, and there could be many messages per second.


On Monday, June 22, 2015 at 2:04:56 PM UTC-7, Dru wrote:
What goal do you want to achieve by batching them? What pain are you hoping to solve for?

-d
On Mon, Jun 22, 2015 at 3:49 PM, Annie Ju <a...@twistbioscience.com> wrote:
Hi,

I'd like to send messages of type ABC to publish via MassTransit using C# interface to a remote RabbitMQ.
Since there may be many messages coming in through different threads, I'd like to group these messages together and send in one batch at a time, e.g., a list of 10 to 20 ABC's.
Is there a way in MT to configure it to send it in batches, rather than one at a time?

How do you handle large number of publish requests in the publisher? 
I can write the batch processing code, e.g., put in a queue and batch them up when they reach a certain size.

Thanks,
Annie

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

Dru Sellers

unread,
Jun 22, 2015, 5:48:38 PM6/22/15
to masstrans...@googlegroups.com
the RabbitMQ client only supports sending one message at a time see - https://www.rabbitmq.com/releases/rabbitmq-dotnet-client/v3.5.3/rabbitmq-dotnet-client-3.5.3-client-htmldoc/html/type-RabbitMQ.Client.IModel.html#method-M:RabbitMQ.Client.IModel.BasicPublish(RabbitMQ.Client.PublicationAddress,RabbitMQ.Client.IBasicProperties,System.Byte%5B%5D)

Message publish throughput will depend largely on typical network issues such as distance and payload size. we have seen people achieve something like 3000+ messages per second on production hardware. That would be one message per 0.33 milliseconds.

so in terms of is it possible? directly no. You could create a message which has a list of messages on it and bundle it up that way. You would increase you payload size and be at risk for an app crash that would lose messages. Due to the speed of RabbitMQ I would suggest publishing your messages as fast as you get them.




On Mon, Jun 22, 2015 at 4:12 PM, Annie Ju <a...@twistbioscience.com> wrote:
Mainly for performance.  
I'm concerned each publish request will take some time over the network, and there could be many messages per second.

On Monday, June 22, 2015 at 2:04:56 PM UTC-7, Dru wrote:
What goal do you want to achieve by batching them? What pain are you hoping to solve for?

-d
On Mon, Jun 22, 2015 at 3:49 PM, Annie Ju <a...@twistbioscience.com> wrote:
Hi,

I'd like to send messages of type ABC to publish via MassTransit using C# interface to a remote RabbitMQ.
Since there may be many messages coming in through different threads, I'd like to group these messages together and send in one batch at a time, e.g., a list of 10 to 20 ABC's.
Is there a way in MT to configure it to send it in batches, rather than one at a time?

How do you handle large number of publish requests in the publisher? 
I can write the batch processing code, e.g., put in a queue and batch them up when they reach a certain size.

Thanks,
Annie

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/0a5976b5-510f-46ae-9c55-bd26aab454f4%40googlegroups.com.

Cezar Gradinariu

unread,
Feb 18, 2019, 5:15:54 PM2/18/19
to masstransit-discuss
Hi, 

A little bit late, but RMQ allows now the publish bulk:  https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/master/projects/client/Unit/src/unit/TestBasicPublishBatch.cs . I used it and it is insanely fast.

Regards,
Cezar G.

On Tuesday, June 23, 2015 at 7:48:38 AM UTC+10, Dru wrote:
the RabbitMQ client only supports sending one message at a time see - https://www.rabbitmq.com/releases/rabbitmq-dotnet-client/v3.5.3/rabbitmq-dotnet-client-3.5.3-client-htmldoc/html/type-RabbitMQ.Client.IModel.html#method-M:RabbitMQ.Client.IModel.BasicPublish(RabbitMQ.Client.PublicationAddress,RabbitMQ.Client.IBasicProperties,System.Byte%5B%5D)

Message publish throughput will depend largely on typical network issues such as distance and payload size. we have seen people achieve something like 3000+ messages per second on production hardware. That would be one message per 0.33 milliseconds.

so in terms of is it possible? directly no. You could create a message which has a list of messages on it and bundle it up that way. You would increase you payload size and be at risk for an app crash that would lose messages. Due to the speed of RabbitMQ I would suggest publishing your messages as fast as you get them.



On Mon, Jun 22, 2015 at 4:12 PM, Annie Ju <a...@twistbioscience.com> wrote:
Mainly for performance.  
I'm concerned each publish request will take some time over the network, and there could be many messages per second.

On Monday, June 22, 2015 at 2:04:56 PM UTC-7, Dru wrote:
What goal do you want to achieve by batching them? What pain are you hoping to solve for?

-d
On Mon, Jun 22, 2015 at 3:49 PM, Annie Ju <a...@twistbioscience.com> wrote:
Hi,

I'd like to send messages of type ABC to publish via MassTransit using C# interface to a remote RabbitMQ.
Since there may be many messages coming in through different threads, I'd like to group these messages together and send in one batch at a time, e.g., a list of 10 to 20 ABC's.
Is there a way in MT to configure it to send it in batches, rather than one at a time?

How do you handle large number of publish requests in the publisher? 
I can write the batch processing code, e.g., put in a queue and batch them up when they reach a certain size.

Thanks,
Annie

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

Chris Patterson

unread,
Feb 19, 2019, 11:57:03 AM2/19/19
to masstrans...@googlegroups.com
Cool, something that might make sense to add. I'll create an issue.

Mainly for performance.  
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/3c31901b-dbf4-4191-8a2c-4b7da9d70318%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages