Understanding NServiceBus in Azure

[item...@hotmail.com]

Hi,

Some of the things I came to like about using NSB when talking application design, was the transactional nature (using DTC while locking it makes life easier), and the Saga features. Now the powers that be wants to embrace cloud and in particular Azure. But as I understand it, I should not use MSMQ in the cloud, but Azure queues or Azure service queues doesn't support DTC so I'll find myself missing the transactional nature, won't I? 

I know that NSB supports Azure queues as a transport but what about Azure service queues (a better alternative for msmq as I read it)? 

Also, when I read about Azure, it seems that much of the same issues that we use NSB for today, are solved in other ways in Azure (e.g. distributor/workers), so does it even make sense to think NSB if everything is running in Azure? 

Any thoughts or actual experience is much appreciated! Even if I haven't mentioned it in this post - anything missed in this post just means I didn't think about it yet!

--
Werner
 

[Jimmy Bogard]

This came up for us the other day when a client was looking at RabbitMQ. The best description I've heard is from Udi, something along the lines of "<insert queue/broker here> is apples, NServiceBus is apple pie". With NServiceBus, I don't have to worry about developing hosts, queue pollers, poison messages, dispatchers, transactions and on and on. To really see the difference, just go through a tutorial on Azure Service Bus, then take a look at the samples. That gap is what you'd have to develop yourself.

As far as DTC in the cloud, no, it's not supported, but instead there are other ways of dealing with that issue. Azure has some built-in ways, or you can develop your own approaches (though if you really examined it, your situation might not require such high guarantees of once-and-only-once messaging).

Check out Clemens' post on the subject for more details: http://blogs.msdn.com/b/clemensv/archive/2012/07/30/transactions-in-windows-azure-with-service-bus-an-email-discussion.aspx

We've turned of DTC in a client or two for various reasons, and honestly, it's not that big of a deal. You just have to do a bit more up-front design and thinking about your services. DTC removes the burden. It's kinda like doing Serializable transactions - it removes the need to think about concurrency, at a cost.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/e25adf63-f19d-4237-b1bf-69e4f13909dc%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Afif Mohammed]

Also please refer to this post for more details on expected NSB behaviour on Azure with no DTC.

[Afif Mohammed]

Hi Jimmy,

Clemen's post relies heavily on the duplicate detection mechanism built into Azure. As I know from where things stand, NSB provides no way of giving you control over setting the Message's ID (which is required for duplicate detection). And if you gravitate towards the out box solution he cited, I understand you would be in some order working around NSB's infrastructure of how it (transparently) manages transactions around message handlers. 

Also given how the current Azure transport is implemented over NSB, I am told, all bus activity in a handler is queued in memory, to happen outside the unit of work(s) that wraps the handler. Which means out of the box, there will always be a window (albeit small) where messages will be consumed by the handler (if all database transactions succeed), but any bus activity (send, publish) that happens as part of the handler can disappear into the ether, if the bus or the worker go down before the Azure Service Bus transactions complete.

I would appreciate some insight on what compensation mechanisms to build to accommodate for this small window of loss.

Cheers,

Afif

On Tuesday, 26 November 2013 01:07:02 UTC+11, Jimmy Bogard wrote:

This came up for us the other day when a client was looking at RabbitMQ. The best description I've heard is from Udi, something along the lines of "<insert queue/broker here> is apples, NServiceBus is apple pie". With NServiceBus, I don't have to worry about developing hosts, queue pollers, poison messages, dispatchers, transactions and on and on. To really see the difference, just go through a tutorial on Azure Service Bus, then take a look at the samples. That gap is what you'd have to develop yourself.

As far as DTC in the cloud, no, it's not supported, but instead there are other ways of dealing with that issue. Azure has some built-in ways, or you can develop your own approaches (though if you really examined it, your situation might not require such high guarantees of once-and-only-once messaging).

Check out Clemens' post on the subject for more details: http://blogs.msdn.com/b/clemensv/archive/2012/07/30/transactions-in-windows-azure-with-service-bus-an-email-discussion.aspx

We've turned of DTC in a client or two for various reasons, and honestly, it's not that big of a deal. You just have to do a bit more up-front design and thinking about your services. DTC removes the burden. It's kinda like doing Serializable transactions - it removes the need to think about concurrency, at a cost.

On Mon, Nov 25, 2013 at 1:55 AM, <item...@hotmail.com> wrote:

Hi,

Some of the things I came to like about using NSB when talking application design, was the transactional nature (using DTC while locking it makes life easier), and the Saga features. Now the powers that be wants to embrace cloud and in particular Azure. But as I understand it, I should not use MSMQ in the cloud, but Azure queues or Azure service queues doesn't support DTC so I'll find myself missing the transactional nature, won't I? 

I know that NSB supports Azure queues as a transport but what about Azure service queues (a better alternative for msmq as I read it)? 

Also, when I read about Azure, it seems that much of the same issues that we use NSB for today, are solved in other ways in Azure (e.g. distributor/workers), so does it even make sense to think NSB if everything is running in Azure? 

Any thoughts or actual experience is much appreciated! Even if I haven't mentioned it in this post - anything missed in this post just means I didn't think about it yet!

--
Werner
 

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

[Andreas Öhlund]

It sort of a hack but you can control the message id (in v4) by setting the "NServiceBus.MessageId" header. That would of course only be useful if the transport actually uses that id as the "transport id".

Sent from my iPhone

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/84df66df-13fa-4f6e-a435-3bb7d0fc6752%40googlegroups.com.

[Yves Goeleven]

Afif,

the servicebus transport does set the ASB messageId = to NSB messageId, and you can enable duplicate detection using the azure servicebus queue config. So in theory, you can close this gap by enabling this feature and set the messageId (as Andreas says it is possible using the header). I haven't tested this scenario yet though, but in theory you should be able to delegate this concern off to ASB.

Kind regards,

Yves

[Andreas Öhlund]

Here is a code sample on how to set the header:

https://github.com/Particular/NServiceBus/blob/7332dda935a87c4c974905a4d6b7ea01fb55e0cb/src/NServiceBus.AcceptanceTests/NonDTCOperations/When_receiving_a_message.cs#L42

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/84c72629-2f58-4031-8f74-2b9f6c76f0e3%40googlegroups.com.

[Afif Mohammed]

The duplicate detection feature in most cases only helps when a message that was successfully delivered to the bus, is sent again because the handler retried, possibly because of a database transaction failure. With the current Azure Service Bus Transport implementation, the chances of this happening are NIL. As all bus activity happens out side the unit of work, a retry will not result in a re-queue. 

Hence the gap I cited earlier remains the same regardless of the duplicate detection and the Message ID setting feature. If the transport were to work the usual way, where a handler encompasses both database and bus activity, if the bus succeeds and database fails, handler retries, bus sends the message again, duplicate detection comes to the rescue, and the developer is only left with ensuring any database transactions are idempotent. 

But with the current Azure Transport implementation, there is no way out. The bus activity happens outside the unit of work, so if the worker or the bus were to go offline, there's nothing to save you. Hence my original question, on what other compensation mechanisms can I look to build here? 

[Yves Goeleven]

I don't think there is a way out... we can't bring the servicebus activity in the uow either as it will try to promote your local transaction to a distributed one (inside asb client) and blow up. 

In azure there are 2 behaviors that combined do not allow us to mitigate all points of potential message loss (lack of DTC and lack of non partitionable & durable storage), all we can do is try our best to get the message there, but there are no absolute guarantees.

[Afif Mohammed]

So does that mean, for some one directly writing apps for Azure service bus (not NSB), if from inside a handler, they do some database activity and then send a message to the bus, that handler will not see the light of the day in production? It means you either must do database activity or bus activity from within a handler. That's a pretty hard constraint.

On 26 November 2013 19:56, Yves Goeleven <yv...@goeleven.com> wrote:

I don't think there is a way out... we can't bring the servicebus activity in the uow either as it will try to promote your local transaction to a distributed one (inside asb client) and blow up. 

In azure there are 2 behaviors that combined do not allow us to mitigate all points of potential message loss (lack of DTC and lack of non partitionable & durable storage), all we can do is try our best to get the message there, but there are no absolute guarantees.

--
You received this message because you are subscribed to a topic in the Google Groups "Particular Software" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/particularsoftware/5DtjlNlNnEM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/c48bd36e-d22f-4dd2-9591-cbc236ddc83e%40googlegroups.com.

[Yves Goeleven]

Correct,

Opening a transaction and doing work against both sql azure and azure servicebus in the same scope will blow up, you need to put each in a separate transaction.

On Tuesday, November 26, 2013 10:03:57 AM UTC+1, Afif Mohammed wrote:

So does that mean, for some one directly writing apps for Azure service bus (not NSB), if from inside a handler, they do some database activity and then send a message to the bus, that handler will not see the light of the day in production? It means you either must do database activity or bus activity from within a handler. That's a pretty hard constraint.

On 26 November 2013 19:56, Yves Goeleven <yv...@goeleven.com> wrote:

I don't think there is a way out... we can't bring the servicebus activity in the uow either as it will try to promote your local transaction to a distributed one (inside asb client) and blow up. 

In azure there are 2 behaviors that combined do not allow us to mitigate all points of potential message loss (lack of DTC and lack of non partitionable & durable storage), all we can do is try our best to get the message there, but there are no absolute guarantees.

--
You received this message because you are subscribed to a topic in the Google Groups "Particular Software" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/particularsoftware/5DtjlNlNnEM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to particularsoftware+unsub...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.

[Afif Mohammed]

Opening a transaction? You mean using TransctionScope yourself inside the handler? Or do you mean the handler regardless will open a transaction and enlist it with the Distributed transaction coordinator? I would have imagined as a developer I can choose not to wrap them in a transaction and manage idempotency guarantees myself.

[Yves Goeleven]

I meant opening a transaction scope in any program (non nsb)

In nsb the default is that we wrap the handler in a transaction scope, but you can turn this of using Configure.Transactions if you like. But the side effect of turning it off is that the read from ASB won't use peek lock anymore either, so the message will be removed from the queue immediatly. 

[Afif Mohammed]

Thank you for the clarification Yves. I was going through the NServiceBus.Azure code base and couldn't find the code that does the bus activity outside the unit of work. 

Can you please point me to the relevant part(s)? 

Thank you 

On Tuesday, 26 November 2013, Yves Goeleven wrote:

I meant opening a transaction scope in any program (non nsb)

In nsb the default is that we wrap the handler in a transaction scope, but you can turn this of using Configure.Transactions if you like. But the side effect of turning it off is that the read from ASB won't use peek lock anymore either, so the message will be removed from the queue immediatly. 

--

You received this message because you are subscribed to a topic in the Google Groups "Particular Software" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/particularsoftware/5DtjlNlNnEM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/8b75332c-d87b-4936-a176-005d815c31f0%40googlegroups.com.

[Yves Goeleven]

We use IEnlistmentNotification to perform the asb send with the transaction commit

https://github.com/Particular/NServiceBus.Azure/blob/develop/src/NServiceBus.Azure.Transports.WindowsAzureServiceBus/SendResourceManager.cs

We chain it in here

https://github.com/Particular/NServiceBus.Azure/blob/develop/src/NServiceBus.Azure.Transports.WindowsAzureServiceBus/AzureServiceBusMessageQueueSender.cs#L53

And retries are handled by the send method

https://github.com/Particular/NServiceBus.Azure/blob/develop/src/NServiceBus.Azure.Transports.WindowsAzureServiceBus/AzureServiceBusMessageQueueSender.cs#L57

And just to be complete, we have plans to use a recently released feature in ASB that allows you to fail over to another namespace when your main namespace is down, instead of just retrying like we do now, but we haven't come around implementing it yet.

https://github.com/Particular/NServiceBus.Azure/issues/37

If you see other potential improvements to mitigate this risk further, let me know and we'll discuss the viability (I'm always open to suggestions, but as this is all pretty theorethical (haven't seen the ASB go down at that exact moment yet), this isn't all that high on the priority list)

On Tuesday, November 26, 2013 10:35:29 AM UTC+1, Afif Mohammed wrote:

Thank you for the clarification Yves. I was going through the NServiceBus.Azure code base and couldn't find the code that does the bus activity outside the unit of work. 

Can you please point me to the relevant part(s)? 

Thank you 

On Tuesday, 26 November 2013, Yves Goeleven wrote:

I meant opening a transaction scope in any program (non nsb)

In nsb the default is that we wrap the handler in a transaction scope, but you can turn this of using Configure.Transactions if you like. But the side effect of turning it off is that the read from ASB won't use peek lock anymore either, so the message will be removed from the queue immediatly. 

--
You received this message because you are subscribed to a topic in the Google Groups "Particular Software" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/particularsoftware/5DtjlNlNnEM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particularsoftware@googlegroups.com.

[Afif Mohammed]

Thanks a many for the links, succinct explanation and the prompt reply Yves. 

Cheers

Afif

To unsubscribe from this group and all its topics, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.

Visit this group at http://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/f73b21c1-f38b-4bb9-8f85-f072faad07e0%40googlegroups.com.

[Udi Dahan]

As a part of our work on removing dependencies on the DTC, we will be addressing the issue mentioned about message IDs. While we can’t provide any guarantees on when that will be ready, we have been making some very nice progress on it and it is one of our top priorities going forwards.

 

Thanks,

 

Udi

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/e25adf63-f19d-4237-b1bf-69e4f13909dc%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

 

--

You received this message because you are subscribed to the Google Groups "Particular Software" group.

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.

To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/74a26c56-7e60-4d6e-ab1a-c01a6d9c4113%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

---

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2014.0.4158 / Virus Database: 3629/6862 - Release Date: 11/24/13

[item...@hotmail.com]

Thanks Afif, already been there :)

[item...@hotmail.com]

Thanks Jimmy.

I understand completely what you say. I've been reading about Azure the last 2 days and even small things are just very hard to accomplish. The promise of course, is that it will pay off in the end with regional scalability, high availability etc etc. 

But you know, as I have been using NSB since 1.8, I'm having a hard time leaving the explicitness (is that a word?) of transactional messaging. But I need to examine Azure in detail - I can't go to my CEO and talk about apple pie :)

-- 
Werner

Den mandag den 25. november 2013 15.07.02 UTC+1 skrev Jimmy Bogard:

This came up for us the other day when a client was looking at RabbitMQ. The best description I've heard is from Udi, something along the lines of "<insert queue/broker here> is apples, NServiceBus is apple pie". With NServiceBus, I don't have to worry about developing hosts, queue pollers, poison messages, dispatchers, transactions and on and on. To really see the difference, just go through a tutorial on Azure Service Bus, then take a look at the samples. That gap is what you'd have to develop yourself.

As far as DTC in the cloud, no, it's not supported, but instead there are other ways of dealing with that issue. Azure has some built-in ways, or you can develop your own approaches (though if you really examined it, your situation might not require such high guarantees of once-and-only-once messaging).

Check out Clemens' post on the subject for more details: http://blogs.msdn.com/b/clemensv/archive/2012/07/30/transactions-in-windows-azure-with-service-bus-an-email-discussion.aspx

We've turned of DTC in a client or two for various reasons, and honestly, it's not that big of a deal. You just have to do a bit more up-front design and thinking about your services. DTC removes the burden. It's kinda like doing Serializable transactions - it removes the need to think about concurrency, at a cost.

On Mon, Nov 25, 2013 at 1:55 AM, <item...@hotmail.com> wrote:

Hi,

Some of the things I came to like about using NSB when talking application design, was the transactional nature (using DTC while locking it makes life easier), and the Saga features. Now the powers that be wants to embrace cloud and in particular Azure. But as I understand it, I should not use MSMQ in the cloud, but Azure queues or Azure service queues doesn't support DTC so I'll find myself missing the transactional nature, won't I? 

I know that NSB supports Azure queues as a transport but what about Azure service queues (a better alternative for msmq as I read it)? 

Also, when I read about Azure, it seems that much of the same issues that we use NSB for today, are solved in other ways in Azure (e.g. distributor/workers), so does it even make sense to think NSB if everything is running in Azure? 

Any thoughts or actual experience is much appreciated! Even if I haven't mentioned it in this post - anything missed in this post just means I didn't think about it yet!

--
Werner
 

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.

To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

[item...@hotmail.com]

Yves,

Is the Azure queue support in NSB with "Windows Azure Queues" or "Service Bus Queues" as defined here:

http://msdn.microsoft.com/en-us/library/windowsazure/hh767287.aspx

And - just to hear it from someone with both Azure and NSB experience, I got 2 questions: 

1)

If we had to install NSB services in Azure, would you embrace the Azure service role layout/pattern, or would just install windows services in a traditional VM?

2)

Would you consider using MSMQ in Azure to support DTC between queue and storage (RavenDB or SQL server VM). I understand that doing so would effectively mean that you might aswell host your services outside Azure - but I need to hear this.

-- 
Werner

[Yves Goeleven]

Hi Werner,

We support both, so you can pick whichever you like most.

I'm a very big fan of cloudservices, but I also understand that it is quite hard to grasp the impact of 'no state' and 'multiple instances of the same machines' on most software designs. But once you 'get it', cloudservices allows you to grow very very big.

Never, ever, use DTC (or anything that depends on it like msmq) in windows azure (or amazon for that matter). The reason is that the networks in these cloud environments are so huge that network partitioning is very very likely, even when you only run a couple of machines, these may be far away from each other (in a network perspective). As in the 2 phase commit protocol every node needs to acknowledge the transaction twice in a limited time frame, you will run into timeouts / in doubt states very often here. So just don't do it! PS: using msmq for local communication on the same machine is fine, as long as you don't rely on the DTC.

Hope that helps,

Yves

[item...@hotmail.com]

Yves, 

I think I'm already hooked. Was reading articles (including yours) all night. And without any sleep, got to work all full of energy and anticipation on what I would discover today. That hasn't happened for some time!

I'll go talk to my team and advocate the NSB + Azure combination. 

Will probably return with more questions when we get our hands dirty :)

Thanks for now...

-- 
Werner

[Sean Feldman]

Hi Werner,

We are currently implementing both. A few things to consider:

1) Monitoring and ProdOps operations - either using ServicePulse/ServiceMatrix or rolling your own (not a good idea), but no other tools

2) Cloud services vs VMs on Azure - I'm with Yves on this, cloud services are better and easier to scale than VMs. Not to mention how easy it is to upgrade to a new OS (by switching a flag in your config) and never ever do system updates (unless team enjoys that) :)

Sean