Reliable messaging without DTC in Azure

[Marqus Landin]

Helpful information to include

Product name: NserviceBus

Version: > 5.0

http://docs.particular.net/nservicebus/no-dtc

Hi, I have read the above but I don't really get how I should make use of this in in Azure :(

Could someone please explain if I am on my own creating reliable messaging in Azure using NSB Version 5? 

If NSB has some feature for this how do I make use of it?

If it is built in by default in Azure Transport please explain how it works?

In the docs link on top, you only can use NHibernate, can I use it with Entity Framework?

Best regards

Marqus Landin

[petr....@gmail.com]

Marqus, i'm sorry for injecting but your Q is interesting me too.

I have two cloud services in Azure. Each of them using SQL Azure through Entity Framework for business data. I need reliable messaging between them.

Am i correct if say that only right and safest solution with Azure Service Bus as Transport is make all my business logic idempotent ?

Something like:

public void Handler(MyEvent event)
{
   IdempotentAction();
   Bus.Publish<MyEvent2>();
}

[Sean Feldman]

Hi Marqus,

With Azure there's no DTC. Azure transports (Storage Queeus and ServiceBus) are built to on top of services that don't support it. These transports support retries (FLR and SLR) that allow to resolve transient communication issues. Outbox currently is not available with Azure transports, though please chime in here to provide your feedback.

[Marqus Landin]

Hi Sean, 

Then I guess I need to implement something like the diagram on this page http://docs.particular.net/nservicebus/no-dtc

on my own, or is that not the way to go in Azure? I guess you guys would have included it for Azure transports already then. 

What are the main obstacles for having the same solution to the problem? I thought this was the way to go after listening to this https://vimeo.com/111998645 from Udi, but that might just apply to on-premise.

Regards 

Marqus 

[Yves Goeleven]

Hi all,

The no-dtc support that we have requires that the business logic & send intents of an endpoint, can be sent to a single database in a single transaction.

That means that it is possible using SQL in azure (for both sql database as sql on a VM), you don't need to build it, you can use the one mentioned in the article above.

However, applicability is very limited. If you use any other azure service that doesn't support transactions (which is all of them to be honest), or you need more than 1 transactional resource (like 2 db connections, or 2 sb queue senders, etc...), then it is probably better to accept the fact that azure is a non transactional world and build idempotency into your application.

Kind regards,

Yves

[petr....@gmail.com]

On Thursday, 19 February 2015 11:24:23 UTC-8, Yves Goeleven wrote:

However, applicability is very limited. If you use any other azure service that doesn't support transactions (which is all of them to be honest), or you need more than 1 transactional resource (like 2 db connections, or 2 sb queue senders, etc...), then it is probably better to accept the fact that azure is a non transactional world and build idempotency into your application.

Yves, did i understand right that if i using Azure Service Bus as Transport and SQL Azure as business storage i should implement idempotency logic?

OR

Use NHibernate as ORM for business data and NServiceBus OutBox feature ?

[Marqus Landin]

Thanks Yves for your response, I will try to make a POC on this and verify my thoughts.

/Marqus

[Marqus Landin]

Hi again, I wonder if you guys have a roadmap for when Entity Framework will be supported?

/marqus

[John Simons]

Hi Marqus,

Sorry missed the initial question regarding EF.

You can write all your business access logic using EF, NH is only needed by the NServiceBus internals.

The thing to watch out is to ensure that EF reuses the internal IDbConnection opened by the NserviceBus framework.

You can access that connection by injecting `NHibernateStorageContext` and reading `Connection`, possibly you also want to enlist in the same ITransaction.

Hope this helps, let us know?

Cheers

John

[Marqus Landin]

Would be awesome if there was a sample for that EF thingy :) or maybe just documentation on it.

Cheers 

Marqus

[John Simons]

Hi Marqus,

I have raised an issue to include a sample on how to do this, see https://github.com/Particular/docs.particular.net/issues/576

In the meantime please feel free to hit us with any questions.

Cheers

John

[Morten]

Hi Yves,

I assume that the Azure Service Bus transport was created when we only had the Basic tier features on Azure Service Bus. Now with the Standard tier which supports topics, transactions, de-duplication and a bunch of other goodies, does that mean that we can do reliable only-once message delivery without outbox now?

If not, are there any plans to support the features in the new Standard tier?

Thanks,

Morten

[Yves Goeleven]

Hi Morten,

Standard tier is not new, basic is new and we only support Standard as we need all those features. We already leverage almost everything that is available, or you can turn it on using the AzureServiceBusQueueConfig configuration section.

But whatever MS marketing says, there are no transactions in azure. ASB does not support true transactions, it has a retry mechanism that looks like transactional rollback on the receive end, but it's not a true transaction. It won't let you rollback sends and it won't play ball with other services in a transactionally consistent way, which means we retries are required all over the place and thus lead to duplicates. The dedupe mechanism can be turned on if you like, but it is timebound, meaning that if a duplicate is sent outside that time window, you will still get duplicates...

So getting an end to end exactly once delivery story cannot be guaranteed

[Morten Petterøe]

Ah,

 

Thanks for clearing that up for me. I guess I was looking for any hope of an easy way out for my specific scenario and rushed to the wrong conclusion. I am implementing a credit card acquiring feature, and the provider (as most providers do) makes it very hard to create an idempotent handler on my end.

 

 

Morten

---

This message is private and confidential and it is intended exclusively for the addressee named above. If you receive this message by mistake, you should not disseminate, distribute or copy this e-mail. Please inform the sender and delete the message and attachments from your system. Neither confidentiality nor any privilege regarding the information is waived or lost by any mistransmission or malfunction. Any views or opinions contained in this message are solely those of the author, and do not necessarily represent those of Santander Consumer Bank AS, unless otherwise specifically stated and the sender is authorised to do so. E-mail transmission cannot be guaranteed to be secure, confidential, or error-free, as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or contain viruses. Santander Consumer Bank AS does not accept responsibility for any changes in the contents of this message after it has been sent, or for any viruses. If the addressee of this message does not consent to the use of internet e-mail, please communicate it to us.

[Yves Goeleven]

There is a good reason why service providers make it so hard on you (MS storage service or whatever is also a service provider and is no exception to this)

Resources locked by external parties for an extended/unknown period of time (which is what transactions actually are), make great targets for denial of service attacks, just grab enough of them and the service provider will stop working or its operations become extremely expensive which will also put them out of business...

So the more we go to a decentralized software system where parts of the system are operated by different parties (cloud, mobile, iot, all seem to point in this direction), the more this will become an issue and force us to change the way we think of building our systems...

[Morten Petterøe]

Oh, I wasn’t even dreaming of something sophisticated like that. I would be satisfied if I could just retry an auth request, and if it was already processed (based on a payment request ID I set), I would get a message that this was a duplicate along with the result from the actual processing. But I am even having a hard time getting a method where I can check the status of a payment based on the ID I set, and if it doesn’t exist go ahead and submit a request.

 

 

Morten Petterøe

 

From: particula...@googlegroups.com [mailto:particula...@googlegroups.com] On Behalf Of Yves Goeleven
Sent: 26. februar 2015 10:52
To: particula...@googlegroups.com
Subject: Re: [particularsoftware] Re: Reliable messaging without DTC in Azure

 

There is a good reason why service providers make it so hard on you (MS storage service or whatever is also a service provider and is no exception to this)

 

Resources locked by external parties for an extended/unknown period of time (which is what transactions actually are), make great targets for denial of service attacks, just grab enough of them and the service provider will stop working or its operations become extremely expensive which will also put them out of business...

 

So the more we go to a decentralized software system where parts of the system are operated by different parties (cloud, mobile, iot, all seem to point in this direction), the more this will become an issue and force us to change the way we think of building our systems...

 

[Yves Goeleven]

Sounds like they really make it hard on you, should at least be able to check the side effects of a command before retrying...

[mariu...@gmail.com]

I'm sorry, but we are not getting this to work (NServiceBus 5, Azure Service Bus transport and local sql server using nhibernate), not even when trying out the official sample (https://github.com/Particular/NServiceBus.NHibernate/tree/c52edf9c1ebde807e9178925009041654735b8d0/Sample.NHibernate.Outbox).

Problems:

            configuration.UseTransport<AzureServiceBusTransport>();

This sets the default serialization level to "Serialized", which enforces pessimistic locking. Why is this the case? This will never scale in production.

            configuration.EnableOutbox();

Calls "DoNotWrapHandlersExecutionInATransactionScope", which means that you can throw an exception in a handler and the data will still be commited to the database. I think the problem is that the dbconnnection has enlisted in the outermost transaction scope because it will enlist with Transaction.Current when opened. Just for debugging purposes we rolled a "IManageUnitOfWork" implementation to test this tesis:

            configuration.RegisterComponents(c => c.ConfigureComponent<UoW>(DependencyLifecycle.InstancePerCall));

    public class UoW : IManageUnitsOfWork

    {

        private TransactionScope _scope;

        public NHibernateStorageContext NHibernateStorageContext { get; set; }       

        public void Begin()

        {

            _scope = new TransactionScope(TransactionScopeOption.Required);

            // this hack will make failing handlers fail as expected again

            (NHibernateStorageContext.Connection as DbConnection).EnlistTransaction(Transaction.Current);

        }

        public void End(Exception ex = null)

        {

            if (ex == null)

                _scope.Complete();

            _scope.Dispose();

        }

    }

As you can see we are a bit confused as to what exactly is the problem, the documentation around Azure and outbox is somewhat sparse, and the answers in this threads are contradicting. So far I'm leaning towards that the implementation is broken, but I am hoping that someone can show a working sample of Azure Service Bus transport, outbox and sql server local storage.

With regards

Marius Ingjer

[Marqus Landin]

I am also still trying to get this to work with AzureServiceBusTransport and as stated in David Boikes book: 

Outbox

One of the features in NServiceBus 5.0 is important to understand, especially if you plan to run NServiceBus in the cloud. The Outbox feature is designed as a replacement for the Distributed Transaction Coordinator (DTC).

If this is so important to understand, then it would be nice with a sample that is pure Azure. Preferably the VideoStore.AzureServiceBus sample with outbox and both NH and EF if possible otherwise document how to do this. 

For my purpose I would like the sample to be with AzureServiceBusTransport, Outbox feature, EF, SQL Azure, and Databus feature

I hope I'm not totally of topic here, and that I have understood what the Outbox feature should be used for. 

Regards,

Marqus 

[Marqus Landin]

Hi, is anyone working on a pure Azure - Outbox sample or am I on my own? 

Please let me know so that I know what to do.

Best regards,

Marqus