Using Azure Service Bus with Deduplication Enabled

[Steve Winward]

Is it possible to use NServiceBus and leverage Azure Service Bus's deduplication feature?  From some preliminary research it appears it may not be possible, but I thought I would reach out and see if anyone else has done this.

It would be nice if there was a way to set the MessageId property in code and allow Azure Service Bus to handle deduplication logic, however

a) I'm not sure there is a way in NServiceBus to manually set this

b) If we could set the MessageId manually, would the NServiceBus retry logic also break since the same MessageId would be resubmitted to the queue?

If this can't be done, what are people using as an alternative to the build in Azure Service Bus deduplication feature?

[Sean Feldman]

Hi Steve,

Could you please provide a little more details how those duplicates get generated?

[Adam Tybor]

A) You are correct, there is no way to set the message id manually from the send / publish unless this is getting resolved in NSB v6 & NSB.ASB v7 but its still unplanned on the backlog after over 2 years, so I don't have high hopes.

https://github.com/Particular/NServiceBus/issues/1779

B) Depending on how you are doing retries, yes, this could cause a problem. For our setup we require all endpoints to send to a single topic where we fan out and forward the messages to the appropriate subscribers / receivers (very different than the default topology), because of this we are able to put dedup logic on the topics and keep dedup off the queues.  This ensures we can still replay and retry messages but also take advantage of the built in dedup logic of ASB.  However, until (A) happens there is not much you can do unless you roll your ASB Transport.

Adam

On Tue, Jan 12, 2016 at 2:51 PM Sean Feldman <sean.f...@particular.net> wrote:

Hi Steve,

Could you please provide a little more details how those duplicates get generated?

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/91e983ac-6728-4218-a107-42855d0b1878%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sean Feldman]

It is part of NSB v6 Adam

https://github.com/Particular/NServiceBus/blob/eb97f0b32e15b29e9b6f6b5e1c37801d7c114528/src/NServiceBus.Core/Pipeline/Outgoing/MessageIdExtensions.cs#L15

And will work with NSB.ASB v7

[Sean Feldman]

Steve,

You should be able to accomplish this in NSB v5 as well, if you follow the example Ramon has provided in the issue shared in Adam`s response (there are new comments added recently).

More specifically, using IMutateOutgoingMessages.

[Sean Feldman]

To add to that, you can also achieve this by using IMutateOutgoingTransportMessages and setting message ID header to the new value.

Note that TransportMessage has an ID property that is a getter w/o setter. Behind the scenes it's just returning the header value for message id.

[Steve Winward]

Thank you all for your responses.  This helps me out a ton!

[Steve Winward]

In case anyone is curious, here is the code required to get this working.  In addition to this code, you also need to make sure that the queue you are using has been created with deduplication enabled.  Once you create the queue, you cannot change this.  It must be set on creation.

public class MessageIdMutator : IMutateOutgoingTransportMessages

    {

        public void MutateOutgoing(LogicalMessage logicalMessage, TransportMessage transportMessage)

        {

            var msg = logicalMessage.Instance as Message1;

 

            if (msg != null && !string.IsNullOrEmpty(msg.MessageId))

            {

                transportMessage.Headers[Headers.MessageId] = msg.MessageId;

            }

        }

    }

 

public class MutatorConfig : INeedInitialization

    {

        public void Customize(BusConfiguration configuration)

        {

            configuration.RegisterComponents(components =>

            {

                components.ConfigureComponent<MessageIdMutator>(DependencyLifecycle.InstancePerCall);

            });

[Ramon Smits]

Steve,

Small improvement, as the mutator is stateless you can make the component registration a single instance.

Regards,

Ramon

--

You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.
Visit this group at https://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/589c2b40-1e35-4432-8a8e-d66db3105d10%40googlegroups.com.

[Steve Winward]

Hey Ramon,

That's a great suggestion.  Thank you for sharing that!

On Thursday, March 24, 2016 at 6:00:05 PM UTC-4, Ramon Smits wrote:

Steve,

Small improvement, as the mutator is stateless you can make the component registration a single instance.

Regards,

Ramon

On Wed, Mar 16, 2016 at 7:21 PM Steve Winward <steve....@gmail.com> wrote:

In case anyone is curious, here is the code required to get this working.  In addition to this code, you also need to make sure that the queue you are using has been created with deduplication enabled.  Once you create the queue, you cannot change this.  It must be set on creation.

public class MessageIdMutator : IMutateOutgoingTransportMessages

    {

        public void MutateOutgoing(LogicalMessage logicalMessage, TransportMessage transportMessage)

        {

            var msg = logicalMessage.Instance as Message1;

 

            if (msg != null && !string.IsNullOrEmpty(msg.MessageId))

            {

                transportMessage.Headers[Headers.MessageId] = msg.MessageId;

            }

        }

    }

 

public class MutatorConfig : INeedInitialization

    {

        public void Customize(BusConfiguration configuration)

        {

            configuration.RegisterComponents(components =>

            {

                components.ConfigureComponent<MessageIdMutator>(DependencyLifecycle.InstancePerCall);

            });

       }

    }

On Thursday, January 14, 2016 at 9:58:41 AM UTC-5, Steve Winward wrote:

Thank you all for your responses.  This helps me out a ton!

On Wednesday, January 13, 2016 at 5:26:29 PM UTC-5, Sean Feldman wrote:

To add to that, you can also achieve this by using IMutateOutgoingTransportMessages and setting message ID header to the new value.

Note that TransportMessage has an ID property that is a getter w/o setter. Behind the scenes it's just returning the header value for message id.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.