Non-competing consumers (sticky-routing) in MassTransit + RabbitMQ

[mar...@stefaniuk.info]

In my integration solution I would have tens od thousands objects for which be handled messages. Each object would should have consumed messages in order and aggregate them somehow within defined time window. To achieve that I'm planning to dispatch messages to different machines for fixed set of objects.

When I looked on RabbitMQ feature it looks easy - all messages would go to one exchange (C# type) but would be enriched with routing key (naive objectId modulo 3) and would go to different queues / workers. Is it possible in MassTransit?

Regards
Marcin

[Chris Patterson]

It isn't in there yet because the ability to monitor the queues isn't there yet either to determine if a queue for a particular routing queue has lost its worker. I know RabbitMQ would likely not handle thousands of consumers like that so it would need to be a mix of routing and consumer load balancing with affinity by your ID. 

____
Chris Patterson

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/01b5d466-de28-4d77-bf8e-2982cbf11de5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marcin Stefaniuk]

Ok. I can simulate work partitioning by competing consumers and selecting defined subset on each machine (by predefined configuration or by sending control messages to worker).

Second thing is: how to fetch 3 or 15 messages by one thread to build a analysis window and ACK them after analysis (one by one) while getting newer (one by one)?

Regards

Marcin Stefaniuk

Wiadomość napisana przez Chris Patterson <ch...@phatboyg.com> w dniu 21 wrz 2015, o godz. 15:41:

You received this message because you are subscribed to a topic in the Google Groups "masstransit-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/masstransit-discuss/AobzKpcymzE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/28202EC55A35A71B.29787E27-79AD-4A1A-9A84-4AB25FA453CE%40mail.outlook.com.

[Chris Patterson]

Any sort of windowing or buffering on the inbound would be some type of coordination built into your consumers. With the async support in MT3, it should be possible with a minimal of stack overhead, but I haven't done it myself. 

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CD81557D-AC3C-43D7-AE39-F903C8F83AC2%40stefaniuk.info.

[mar...@stefaniuk.info]

Ok. Please correct if I'm wrong but I thought that MT is acking automagically. Should I keep tasks in memory untill processing is not finished and use NotifyConsumed (for success)?

Regards
Marcin

[Chris Patterson]

Just don't complete the Consume(...) Task until the composition of messages is completed. You are using RabbitMQ right? Make sure you set the PrefetchCount high enough to account for the number of pending (awaiting acknowledgement) consumers.

The same as if you were batching with a timeout, if you want a batch of 100, set the prefetch count to something like 101, and have a timeout on the batch combiner for the Tasks. I think the TPL Dataflow has some good examples here.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/a4cca01a-0ed5-409d-9312-ab832e5f9002%40googlegroups.com.