Transferring large messages using MSMQ transport

[René M. Andersen]

We are using Rhino-ESB with MSMQ as transport. MSMQ has been chosen instead of Rhino Queues due to the tooling available to inspect the queues.

We have several small messages transferred continously and in addition to that one message send once a day which can be quite large and exceed the MSMQ 4MB limit.

To deal with this we are considering a databus/claim check approach where the message content of the large message is stored in a document database (MongoDB) and only an identifier is transferred using Rhino-ESB.

We would like to be able to handle this before the message is serialized (since it will be serialized again when saved in the database).

One solution would be to introduce a "pre-sending" event in the transport implementation where such cases as the above can be handled. The handler of this event should be able to signal that it has taken responsibility for the message. This way we can implement a message module to determine if a message content should be transferred normally or using the database approach.

How does that sound? Any other/better ideas?

[Jason Meckley]

if the message is that large it may be a good candidate to refactor into smaller messages and a saga.
1. send start message
2. send data in small chunks
3. send end message

the start and end message could also contain checksums to ensure all the "middle" messages arrived.

[René M. Andersen]

Hi Jason (and others), Thanks for the input.

We are also considering that option, but in our scenariowe have some concerns:

• We have several independent consumers of the messages incl. the large one (pub-sub scenario) each consumer with its own message processing logic. This means that every consumer should implement the sequencing saga. This could to some degree be resolved by providing some common saga handling with hooks for the custom parts.
  
• What if one of the sequencing messages fail? Can I retry that one message? I guess perhaps I can if I can make the message arrival order indepedent

I feel the databus approach would be a more general solution to the problem. We have a few more message types which may exceed the limit in some scenarios.

[René M. Andersen]

Corey and others, any thoughts on the suggestion about the pre-send event?

[Corey Kaylor]

First off, sorry for the delay responding.

Have you had a chance to look at ICustomElementSerializer? I think it should be able to handle what you've described.

On Sat, Sep 3, 2011 at 4:54 AM, René M. Andersen <renemygin...@gmail.com> wrote:

Corey and others, any thoughts on the suggestion about the pre-send event?

--
You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group.

To view this discussion on the web visit https://groups.google.com/d/msg/rhino-tools-dev/-/aquXaGrI-s8J.

To post to this group, send email to rhino-t...@googlegroups.com.
To unsubscribe from this group, send email to rhino-tools-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en.

[René M. Andersen]

I haven't thought about the ICustomElementSerializer interface. I think it will suite my needs just fine. 

Thanks for the reply.