Azure Queues Transport speed very slow with zero load

[Eric Swann]

Helpful information to include

Product name: NServiceBus

Version: 5.2.6

I've got a demo web api that is sending a command and waiting for a response.  Behind the scenes its doing a mini-saga calling two other services and returning some minimal data.  When I use MSMQ as the transport, it takes < 2 seconds to return a response to the client calling the web api.  This is pretty slow to begin with but not my main issue.  When I use the Azure storage emulator and switch to Azure Queues the response returns in > 6 seconds. So switching to that transport adds over ~ 4 seconds.  This is all running on a local VM.  However, when I actually deploy to Azure as use real azure queues I see similarly poor performance. The service is running on an A1 azure VM which doesn't have a lot of power, but this is with zero load on the system.

Any idea why using the Azure Queues transport would be so crazy slow?  Any help is appreciated.

Thanks,

Eric

[Sean Feldman]

Hi Eric,

Are you running with debugger attached? Try to run w/o debugger, as debugger does slow down things. 

When you're using ASQ and Azure Storage persistence, transport and persistence are making HTTP calls to the Azure data centre for message sending/receiving and storage operations. It is expected to be slower than MSMQ.

If you'd like to share your code, I could go through it and post my time and findings.

[Yves Goeleven]

Hi Eric,

Can you try setting following .net runtime properties before starting the bus? (it must be done before any http call is made)

ServicePointManager.UseNagleAlgorithm = false;
ServicePointManager.Expect100Continue = false;
ServicePointManager.CheckCertificateRevocationList = false;
ServicePointManager.DefaultConnectionLimit = 1000;

Your solution is very Http heavy, all queue interactions, saga interactions and webservice interactions use web requests, yet .net limits the number of web requests to 2 by default and queues up the rest.

Hope this helps,

Yves

[Eric Swann]

Well, actually it didn't help...I was still using MSMQ when I ran the last test.  Once I switched it, back to very slow performance.  I'll try to put together a sample as I can't post my entire project.  Thanks.

[Eric Swann]

Thanks Sean, running w/o debug did speed things up a little...like from 6.5 seconds to 5.7 seconds...but that's still way slow.  I'm getting 2 secs with MSMQ by comparison.  I'll try to dig deeper and determine where the bottleneck is.  FYI, I am using SQL Server for persistence (saga/subscription) and Azure Queues for transport.

[Sean Feldman]

Hi Eric,

If you could share the sample, I'll have a look. SQL server persistence, is it local or Azure as well?

A quick test with sending a command and processing it (w/o persistence) is about 2-3 seconds with self-host. 

Having a sample would be much easier to try to  help. 

[Eric Swann]

So that this is fully understood.  I'm running locally with the Azure Storage Emulator (so it's not running on Azure) and SQL Express locally.  That being said, I've also deployed to Azure where I am using the "real" Azure Queues and SQL Azure.  I tried switching to SQL Server Transport and I'm also getting very slow results locally using sql express.  I was going to try to push it to my dev Azure to see what happens out there using SQL Storage.  I'll try to put a sample together tomorrow.

Thanks,

Eric

[Geert Depickere]

I believe NServiceBus polls the azure storage queues with a back-off mechanism (to preserve transactions costs, slowing down polling rate while queue in empty, and then switching back to fast polling when it finds a message in the queue). To make sure the extra delay you see is not caused by this, please perform multiple measurements with multiple consecutive messages (expecting the first one to be slower).

Geert.

Op woensdag 30 september 2015 23:59:05 UTC+2 schreef Eric Swann:

[Eric Swann]

Hi Sean, I have posted a sample at the following location:

https://github.com/eswann/NSB_Perf

I've set up a demo to make it easy to switch between 4 transports (msmq, sql, azure queues, rabbit).  I explain the setup and results I've found in the readme.  Basically I have a reqeust response that runs a saga which contains two commands.  The logic of each command is just a Task.Delay(100).

The times I'm seeing are pretty dramatically different when running locally.  This includes some warm-up requests.  Time is per request for a full reqeust/response cycle that completes the saga.

Rabbit : < 200 ms 

MSMQ: 300-350 ms

SQL: 1500-3500 ms

Azure Queues: 4500-6000 ms

Any help is appreciated...thanks in advance.

[Eric Swann]

Sure, this isn't just for the first command.  I typically run several in succession and Azure Queues are very slow for me even after warmup.

[Eric Swann]

Hi Yves, please check out my sample and results here:

https://github.com/eswann/NSB_Perf

I explain everything in the readme.  I'm seeing dramatic differences between transports.  Hopefully the very slow speeds I'm seeing on Azure Queues (and even SQL) indicate something with my setup and not the transports...that being said, Rabbit is very fast...so it seems like the slowness is dependent on the transport itself.

On Tuesday, September 29, 2015 at 2:12:45 AM UTC-5, Yves Goeleven wrote:

[Yves Goeleven]

Did you perform the test as well hosted in azure using a real azure storage account and with the settings I mentioned applied before any outbound connection is made?

To be frank, storage emulator is a real piece of junk, which I never use..

[Yves Goeleven]

PS: I do not expect azure storage queues to outperform any of the other  transports, it has been designed to be cheap, not fast... in fact it will go to sleep if it sees no messages on the queue. (How many ms are there between testruns?)

If you need a more performance azure native transport, I suggest you look into the azure servicebus one (or just use rabbit if it suits your architecture), or even try dialing down the sleep interval

[Eric Swann]

Thanks for the reply Yves,

Yes, even when deployed out to Azure the azure queues performed very slowly (even with nagling turned off and the connection count bumped etc...). Understood that the emulator isn't the same thing..but I think it's clear this is somewhere in the NSB transport adapter, not the transport itself.  I've used Azure queues in the past and they have latency ~10ms in the same virtual network with nagling off. We're running on pretty basic A1 VM's on Azure...but it's not like we're loading them either.  Even the SQL transport was substantially slower going against both a local sql express or Sql Azure when hosted. 

I'm guessing some of this is due to the nature of how those transports poll for messages.  One of my coworkers stated that they used SQL as a transport in the past and didn't remember seeing that level of latency.  It seems like SQL is definitely faster when it's getting a constant stream of messages (perhaps due to connection/polling backoff when no messages are present) but even in that case I was seeing pretty slow perf a good bit of the time when I fed it a constant stream of messages.

Rabbit on the other hand was very performant (both locally and hosted in azure), more along the lines of what I'd expect.  I installed Rabbit in Azure via a docker on a low power A1 VM running linux.  It blew away the other transports...hands down.  Don't get me wrong, I love rabbit, but we were hoping to use Azure Queues to keep costs down as we'd need to either go with an expensive Rabbit Azure 3rd party service or create our own rabbit clusters via our own VM's.  Using Azure queues is way cheaper.

I'm also going to try with Azure Service Bus to see what that yields...unfortunately there isn't really a good way to test that locally and get any real idea of perf.

Best,

Eric

[Eric Swann]

Understood, I'm going to also try with Azure Service Bus and see what happens.  I haven't used ASB before...is it a polling receive mechanism like Azure Queues or is it a push from ASB?

Cheers,

Eric

[Yves Goeleven]

Eric,

Azure storage transport is extremely simple and has almost no overhead. So I'm pretty sure following thread.sleep is what you're waiting for https://github.com/Particular/NServiceBus.AzureStorageQueues/blob/master/src/Transport/AzureMessageQueueReceiver.cs#L102; 

This design is intentional, it reads the queue once for a batch, if there is no next batch it goes to sleep

You can circumvent it by putting a large amount of messages on the queue, or dial down the MaximumWaitTimeWhenIdle

On Friday, October 9, 2015 at 10:57:14 PM UTC+2, Eric Swann wrote:

Thanks for the reply Yves,

Yes, even when deployed out to Azure the azure queues performed very slowly (even with nagling turned off and the connection count bumped etc...). Understood that the emulator isn't the same thing..but I think it's clear this is somewhere in the NSB transport adapter, not the transport itself.  I've used Azure queues in the past and they have latency ~10ms in the same virtual network with nagling off. We're running on pretty basic A1 VM's on Azure...but it's not like we're loading them either.  Even the SQL transport was substantially slower going against both a local sql express or Sql Azure when hosted. 

I'm guessing some of this is due to the nature of how those transports poll for messages.  One of my coworkers stated that they used SQL as a transport in the past and didn't remember seeing that level of latency.  It seems like SQL is definitely faster when it's getting a constant stream of messages (perhaps due to connection/polling backoff when no messages are present) but even in that case I was seeing pretty slow perf a good bit of the time when a fed it a constant stream of messages.

Rabbit on the other hand was very performant (both locally and hosted in azure), more along the lines of what I'd expect.  I installed Rabbit in Azure via a docker on a low power A1 VM running linux.  It blew away the other transports...hands down.  Don't get me wrong, I love rabbit, but we were hoping to use Azure Queues to keep costs down as we'd need to either go with an expensive Rabbit Azure 3rd party service or create our own rabbit clusters via our own VM's.  Using Azure queues is way cheaper.

I'm also going to try with Azure Service Bus to see of what that yields...unfortunately there isn't really a good way to test that locally and get any real idea of perf.

[Yves Goeleven]

ASB could be considered push based (in fact it's long polling on an open tcp connection), so latency is expected to be low

[Eric Swann]

Cool...will do

[Eric Swann]

Yeah, changing the max wait got me to ~650ms locally...so that's a big improvement.  I'll try it out on azure and see what happens.