Queuing messages for batch processing
398 views
Skip to first unread message
Tommy Jakobsen
Sep 23, 2015, 3:46:20 AM9/23/15
to masstransit-discuss
We have situation where we need to synchronize orders and order states to an ERP system. Everything is built around MassTransit 3 and RabbitMQ. An application consumes order related events and talks to the ERP systems API. For example, orders are created when consuming the OrderApproved event, and there’s events consumed for changing order states, and managing the whole order flow as well. So far so good.
Now we need to batch order creation and processing in the ERP system. The ERP systems API supports batching functionality, so the question is how to pick up all these order related events and build the API request to begin batch importing the orders. The batch import job should run every X minutes, importing a maximum of Y orders.
I have considered two scenarios.
Consume each message as we are today, but store the messages in a SQL table, ready to be processed by the batch import job, taking takes the oldest Y records, and builds the API request.
The other option is to utilize RabbitMQ. We could setup a binding that places order related messages into a separate queue, waiting for the batch job to execute. The batch job should then take the oldest Y messages (RMQ queues are FIFO based, right?) and build the API request.
I like this idea the best, but I am unsure about how to implement such functionality with MassTransit. I am thinking that the batch job should start a bus control with a receive endpoint pointing at the queue. However, how can I make sure that I only retrieve Y messages? Is there any functionality to retrieve “sets of messages”, or should I just use regular message handlers and then stop the bus control when some shared counter reaches Y? How would you approach this problem?
Now we need to batch order creation and processing in the ERP system. The ERP systems API supports batching functionality, so the question is how to pick up all these order related events and build the API request to begin batch importing the orders. The batch import job should run every X minutes, importing a maximum of Y orders.
I have considered two scenarios.
Consume each message as we are today, but store the messages in a SQL table, ready to be processed by the batch import job, taking takes the oldest Y records, and builds the API request.
The other option is to utilize RabbitMQ. We could setup a binding that places order related messages into a separate queue, waiting for the batch job to execute. The batch job should then take the oldest Y messages (RMQ queues are FIFO based, right?) and build the API request.
I like this idea the best, but I am unsure about how to implement such functionality with MassTransit. I am thinking that the batch job should start a bus control with a receive endpoint pointing at the queue. However, how can I make sure that I only retrieve Y messages? Is there any functionality to retrieve “sets of messages”, or should I just use regular message handlers and then stop the bus control when some shared counter reaches Y? How would you approach this problem?
Tommy Jakobsen
Sep 28, 2015, 4:34:05 AM9/28/15
to masstransit-discuss
Anyone? I'm open to other suggestions as well.
Travis Smith
Sep 28, 2015, 8:17:57 AM9/28/15
to masstransit-discuss
MassTransit doesn't really have a way to batch consume messages, so you either write that out of bounds of MT or use SQL. Just schedule a Quartz job that activates the batch event.
The big problem with doing this via messages is managing which messages to ACK or not; I think you'll have an easier time with this doing it with SQL.
--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/1ea7e4c8-ca03-4393-ba29-1fb55558def2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
-Travis
Reply all
Reply to author
Forward
0 new messages