Strategy for replaying messages for CQRS read model rebuilding

[Tim Gebhardt]

Hello,

Our system has an endpoint that listens to all the messages published to the system (a Message Store in the EIP parlance) so that we are able to bootstrap new endpoints or update existing endpoints with messages that they may not have been listening to in the past.  I'm wondering if anyone has any good strategies for achieving this with MT + RabbitMQ?

I figured this would be easiest if each of our read-model denormalizers had their own MT endpoint (unique RabbitMQ url-per-read-model) so that we can easily wipe out a read model, dump all the messages from the message store directly to the read-model's queue, and then just let it work like normal.

The tricky thing is the cut-over from replay to live updates.  I figured the easiest thing to do would be to some how suspend the real subscription, spin up a separate copy of the same denormalizer subscribed to a temporary queue for the replay, then when that's done unsuspend the real subscription and just check for duplicates.

Am I on the right track here?  Is anyone else doing something different for message replay and MT?  Any caveats or helpful things that MT can do for me in this situation?

[Dru Sellers]

I have not done this at all. Am I understanding correctly that you are looking for the ability to collect messages at your endpoint but not process them until a specific time??

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msg/masstransit-discuss/-/tRfpJFtdohYJ.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Tim Gebhardt]

Not quite.  Imagine a system where we record all the business events that get published by all of our systems.  Now say at some point our stake holder says "Oh you know what would be awesome?  If we had such and such report organized like so".  Well we have the raw data to produce what they've asked for (because we have the entire history of what's happened to our system), we just didn't anticipate their request from day one.

And actually what they've asked for is really just an existing report but with a few extra columns.

So the easiest way to produce this report would be to update our code to listen for the extra events, blow away the existing report, dump what's in our message history into the report's queue (RabbitMQ queue-per-report to make this easy), and then listen for live updates once the backfill has been completed.

Any hints on how MassTransit could handle this situation?

On Saturday, May 4, 2013 11:34:53 AM UTC-5, Dru wrote:

I have not done this at all. Am I understanding correctly that you are looking for the ability to collect messages at your endpoint but not process them until a specific time??

On Fri, May 3, 2013 at 1:53 PM, Tim Gebhardt <t...@gebhardtcomputing.com> wrote:

Hello,

Our system has an endpoint that listens to all the messages published to the system (a Message Store in the EIP parlance) so that we are able to bootstrap new endpoints or update existing endpoints with messages that they may not have been listening to in the past.  I'm wondering if anyone has any good strategies for achieving this with MT + RabbitMQ?

I figured this would be easiest if each of our read-model denormalizers had their own MT endpoint (unique RabbitMQ url-per-read-model) so that we can easily wipe out a read model, dump all the messages from the message store directly to the read-model's queue, and then just let it work like normal.

The tricky thing is the cut-over from replay to live updates.  I figured the easiest thing to do would be to some how suspend the real subscription, spin up a separate copy of the same denormalizer subscribed to a temporary queue for the replay, then when that's done unsuspend the real subscription and just check for duplicates.

Am I on the right track here?  Is anyone else doing something different for message replay and MT?  Any caveats or helpful things that MT can do for me in this situation?

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

[Tim Gebhardt]

So I'm making pretty good progress on this, but one part I can't seem to get past is how to tell MassTransit to ignore a message in the routing or pre-delivery phase.

A few details: I have a helper class that'll know how to do the backfill and then after backfill switch over to live updating.  I want the backfill to maintain a list of the last X (100 or so) MessageIds so that if it sees one of those MessageIds on the live update stream it doesn't process that message because it's a duplicate.

Is there somewhere in MassTransit that I'm missing that can do that?  I don't want to have all my projections to need to be able to do this because that code will get duplicated everywhere.

[Tim Gebhardt]

Or I guess to ask another way: Is there a way to either wireup MassTransit not to process duplicate messages (based on message id), or to cancel a message processing in the pipeline?  Or is this something I'll need to handle in each consumer?

[Travis Smith]

http://en.wikipedia.org/wiki/Idempotence

In a system using MassTransit you are pretty much required to have idempotance. There are no ordering assurances provided. You can someone come close with only having one thread but that's it. 

However, it would appear you just want a backup? Just bind another queue to the exchange in question. Don't have anyone read off it...

 

-Travis

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/masstransit-discuss/-/0txduIYhdGQJ.

[Dru Sellers]

No 'message ids' are considered a concern of the consumer. :(

We are working on making this kind of stuff easier to do in future releases, but that doesn't help you right now.

Chris and I have often talked about MT Consumers as being the glue to pushing messages into a 'business pipeline' its this pipeline that (in your case) could handle these kinds of concerns.

So, alas, we have a bunch of stuff in the fire that's going to help but nothing today.

Depending on your container usage, you could get fancy with AOC style programming to handle it. But then you'd be fancy. ;)

-d

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/masstransit-discuss/-/0txduIYhdGQJ.