Message data external storage middleware

[Egor Chikunov]

Hello, everyone!

Some of the messages on our project are really big (and by 'big' i mean huge: 1+ gb size.). So, the idea is to send simple message with PayloadId and Type of the data, store huge initial message in the db and retrieve it on the receiving side by id.

It's somewhat close to existing UseMessageData, but IMessageDataRepository allows only to save/get stream of data based on uri.

I tried to write own middleware, sending special message type (MessageWithPayload), but got stuck on passing message from database to consuming pipe.

Send method of the filter looks like this:

        public async Task Send(ConsumeContext<T> context, IPipe<ConsumeContext<T>> next)

        {

            ConsumeContext<MessageWithPayload> consumeContext;

            if (!context.TryGetMessage(out consumeContext))

            {

                await next.Send(context);

            }

            else

            {

                var message = await _repository.RetrieveMessage<T>(consumeContext.Message.PayloadId);

                await next.Send(???)

            }

        }

Could someone help me with the way to pass message to the right pipe or suggest some better overall approach?

Thanks

[Chris Patterson]

The use of Message Data is pretty poorly documented, but the URI is returned by the repository when the message data is stored. And it's not an HTTP reachable URI, it's just a way to store the identifying information for the message data. The repository itself uses the data in the URI to load the message data. 

It sounds like you need to create your own message data repository implementation, and use that to store and load your large payloads. The URI format is owned by your repository, so you can use it as you like to store identifiers, etc. For instance, we store them as urn:data:base32hashofdata when saving large payloads.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/027c8373-cf9c-4c7e-b337-8d0c237160c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Egor Chikunov]

Ah, I see. Yeah, It's logical - Unique Resource Identifier.

Thanks a million, Chris

понедельник, 1 февраля 2016 г., 21:00:11 UTC+3 пользователь Chris Patterson написал:

The use of Message Data is pretty poorly documented, but the URI is returned by the repository when the message data is stored. And it's not an HTTP reachable URI, it's just a way to store the identifying information for the message data. The repository itself uses the data in the URI to load the message data. 

It sounds like you need to create your own message data repository implementation, and use that to store and load your large payloads. The URI format is owned by your repository, so you can use it as you like to store identifiers, etc. For instance, we store them as urn:data:base32hashofdata when saving large payloads.

On Mon, Feb 1, 2016 at 8:56 AM, Egor Chikunov <chikun...@gmail.com> wrote:

Hello, everyone!

Some of the messages on our project are really big (and by 'big' i mean huge: 1+ gb size.). So, the idea is to send simple message with PayloadId and Type of the data, store huge initial message in the db and retrieve it on the receiving side by id.

It's somewhat close to existing UseMessageData, but IMessageDataRepository allows only to save/get stream of data based on uri.

I tried to write own middleware, sending special message type (MessageWithPayload), but got stuck on passing message from database to consuming pipe.

Send method of the filter looks like this:

        public async Task Send(ConsumeContext<T> context, IPipe<ConsumeContext<T>> next)

        {

            ConsumeContext<MessageWithPayload> consumeContext;

            if (!context.TryGetMessage(out consumeContext))

            {

                await next.Send(context);

            }

            else

            {

                var message = await _repository.RetrieveMessage<T>(consumeContext.Message.PayloadId);

                await next.Send(???)

            }

        }

Could someone help me with the way to pass message to the right pipe or suggest some better overall approach?

Thanks

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

[sonst...@googlemail.com]

The use of Message Data is pretty poorly documented

It does not appear in the documentation at all. I just found it by chance in a blog article.
Maybe you can at least copy the information from your blog to the documentation or link to it.

https://lostechies.com/chrispatterson/2015/06/16/masstransit-v3-update/
https://blundell89.github.io/data/2016/02/16/sharing-large-message-between-your-services-with-masstransit-and-mongodb.html