Intermittent issues with C# driver on Azure

[Joe Waller]

Hi folks,

I am experiencing intermittent errors with my application on Azure. It happens every few minutes

System.AggregateException: One or more errors occurred. ---> MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

   --- End of inner exception stack trace ---

   at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)

   at System.Net.Security.SslStream.EndRead(IAsyncResult asyncResult)

   at System.IO.Stream.<>c__DisplayClass0.<BeginEndReadAsync>b__9(Stream stream, IAsyncResult asyncResult)

   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at MongoDB.Driver.Core.Misc.StreamExtensionMethods.<ReadBytesAsync>d__0.MoveNext()

Here are my connection settings

ssl=true

connectTimeoutMS=30000

socketTimeoutMS=300000

maxPoolSize=50

maxIdleTimeMS=45000

How do I go about debugging this? Is it a possible connection pool issue? 

[Joe Waller]

Also the driver version is 2.0.1

[Craig Wilson]

It would appear that there is some form of networking issue: 

    System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 

This indicates to me that a connection that was alive has stopped functioning.  Can you turn on Network tracing for System.Net.Sockets: https://msdn.microsoft.com/en-us/library/ty48b824(v=vs.110).aspx?

[Joe Waller]

Hi Craig,

I will try to get the traces you requested. A few more follow up questions

1. Is maxIdleTimeMS implemented in the c# driver?

2. Is it possible to add to the documentation the default values for various parameters in the driver - connectiontimeout, sockettimeout, pool size, max idle time ms

3. How does maxIdleTimeMS interact with tcp keep alive? If I set TCP keep alive to a lower value like 40s will that prevent from maxIdleTimeMS from kicking in and removing and idle connection?

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/OlT0kecHBzY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/b1872bf8-0a40-479a-b283-665326df64d7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Craig Wilson]

1. Yes, maxIdleTimeMS is implemented in the driver. Are you seeing evidence that it isn't being honored?

2. Yes. I've added https://jira.mongodb.org/browse/CSHARP-1397

3. We don't currently set TCP keep alive. While it is possible for you to set it manually, there is no interaction between them.

[Joe Waller]

So maxIdleTimeMS is based on mongo activity and not TCP activity? Even if the TCP keep alive setting keeps the TCP connective alive , if there is no mongo activity the connection will be torn down after maxIdleTimeMS?

Since we are seeing this problem withe active connections (that may have been idle for some time)  I was wondering if maxIdleTimeMS is implemented. Are there any logs for the underlying connection pool operations performed by the driver so we can confirm that?

I will have the socket logs for you soon.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/e33e8bfe-4b5f-40d0-aeb1-627e03e55485%40googlegroups.com.

[Stephen Steneker]

On Thursday, 10 September 2015 05:24:52 UTC+10, Joe Waller wrote:

I will try to get the traces you requested. A few more follow up questions

1. Is maxIdleTimeMS implemented in the c# driver?

2. Is it possible to add to the documentation the default values for various parameters in the driver - connectiontimeout, sockettimeout, pool size, max idle time ms

3. How does maxIdleTimeMS interact with tcp keep alive? If I set TCP keep alive to a lower value like 40s will that prevent from maxIdleTimeMS from kicking in and removing and idle connection?

Hi Joe, 

Are your MongoDB servers also deployed on Azure? The Azure load balancer is a common culprit for connection timeouts; you should reduce the TCP keepalive on your MongoDB servers to be lower than the Azure timeout: http://docs.mongodb.org/master/administration/production-notes/#azure.

Regards,

Stephen

[Joe Waller]

Hi Stephen - my mongodb servers are not on Azure. So this should not be an issue. Only my application code is running on Azure.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/OlT0kecHBzY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/b34c80c2-c63f-49f3-8bcd-8b520c830c61%40googlegroups.com.

[Craig Wilson]

Joe,

Yes, MaxIdleTime is based on mongo activity. Even if TCP keep-alive is set, no mongo activity would cause the connection to be torn down.

To prove that MaxIdleTime is working you can do the following... This is for the 2.0.x versions of the driver and uses an experimental API that will be changed in 2.1. The below sets MaxIdleTime to 5 seconds and then runs a query every 7 seconds. In the console, you'll notice that the connections are opened and closed every ~7 seconds. If your problems seem to occur with long lived connections, you might also want to play with MaxLifeTime.

using System;

using System.Threading.Tasks;

using MongoDB.Bson;

using MongoDB.Driver;

using MongoDB.Driver.Core.Events.Diagnostics;

namespace ConsoleApplication1

{

    class Program

    {

        static void Main(string[] args)

        {

            MainAsync().GetAwaiter().GetResult();

        }

        static async Task MainAsync()

        {

            var settings = new MongoClientSettings

            {

                MaxConnectionIdleTime = TimeSpan.FromSeconds(5),

                ClusterConfigurator = cb =>

                {

                    var writer = Console.Out;

                    cb.AddListener(new LogListener(writer, LogLevel.Info));

                }

            };

            var client = new MongoClient(settings);

            var db = client.GetDatabase("test");

            var col = db.GetCollection<BsonDocument>("foo");

            while (true)

            {

                var results = await col.Find("{}").ToListAsync();

                // throw them away...

                await Task.Delay(TimeSpan.FromSeconds(7));