StackOverflowException in C# driver on Azure

838 views
Skip to first unread message

Ron Rejwan

unread,
May 29, 2014, 4:59:21 AM5/29/14
to mongod...@googlegroups.com
Hello,

We are running a production WCF Rest service on Azure that uses MongoDB (using the C# driver).
Recently we've began getting strange random service hangs on the server.

We have deployed 3 duplicated instances of the service, which all run fine for days until suddenly one of the instances would begin massively failing to connect to the DB.
The other 2 instances continue working perfectly well, and looking in the actual MongoDB logs - we can see nothing wrong (We are hosted on MongoHQ and they verified this).

Simply restarting the hanged instance immediately solves the issue, until it begins occurring again.

Our logs show a lot of socket timeouts to the DB, and looking at the IIS debug dump I found several interesting exceptions in our exception heap:





Previous .NET Exceptions Report (Exceptions in all .NET Heaps)

Exception Type   Count   Message   Stack Trace
System.Net.Sockets.SocketException   1   A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond   System.Net.Sockets.Socket.DoConnect(System.Net.EndPoint, System.Net.SocketAddress)
   System.Net.Sockets.Socket.Connect(System.Net.EndPoint)
   System.Net.Sockets.TcpClient.Connect(System.Net.IPEndPoint)
   MongoDB.Driver.Internal.MongoConnection.Open()
   MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoDB.Bson.IO.BsonBuffer, Int32)
   MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoDB.Driver.Internal.MongoRequestMessage)
   MongoDB.Driver.Operations.CommandOperation`1[[System.__Canon, mscorlib]].Execute(MongoDB.Driver.Internal.MongoConnection)
   MongoDB.Driver.MongoServerInstance.Ping(MongoDB.Driver.Internal.MongoConnection)
   MongoDB.Driver.MongoServerInstance.Connect()
   MongoDB.Driver.Internal.MultipleInstanceMongoServerProxy+<>c__DisplayClassd.<ConnectInstance>b__c(System.Object)
   
System.Exception   1   <none>   System.Net.Sockets.Socket.DoConnect(System.Net.EndPoint, System.Net.SocketAddress)
   System.Net.Sockets.Socket.Connect(System.Net.EndPoint)
   System.Net.Sockets.TcpClient.Connect(System.Net.IPEndPoint)
   MongoDB.Driver.Internal.MongoConnection.Open()
   MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoDB.Bson.IO.BsonBuffer, Int32)
   MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoDB.Driver.Internal.MongoRequestMessage)
   MongoDB.Driver.Operations.CommandOperation`1[[System.__Canon, mscorlib]].Execute(MongoDB.Driver.Internal.MongoConnection)
   MongoDB.Driver.MongoServerInstance.Ping(MongoDB.Driver.Internal.MongoConnection)
   MongoDB.Driver.MongoServerInstance.Connect()
   MongoDB.Driver.Internal.MultipleInstanceMongoServerProxy+<>c__DisplayClassd.<ConnectInstance>b__c(System.Object)
   
System.OutOfMemoryException   1   <none>   System.Net.Sockets.Socket.DoConnect(System.Net.EndPoint, System.Net.SocketAddress)
   System.Net.Sockets.Socket.Connect(System.Net.EndPoint)
   System.Net.Sockets.TcpClient.Connect(System.Net.IPEndPoint)
   MongoDB.Driver.Internal.MongoConnection.Open()
   MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoDB.Bson.IO.BsonBuffer, Int32)
   MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoDB.Driver.Internal.MongoRequestMessage)
   MongoDB.Driver.Operations.CommandOperation`1[[System.__Canon, mscorlib]].Execute(MongoDB.Driver.Internal.MongoConnection)
   MongoDB.Driver.MongoServerInstance.Ping(MongoDB.Driver.Internal.MongoConnection)
   MongoDB.Driver.MongoServerInstance.Connect()
   MongoDB.Driver.Internal.MultipleInstanceMongoServerProxy+<>c__DisplayClassd.<ConnectInstance>b__c(System.Object)
   
System.StackOverflowException   1   <none>   System.Net.Sockets.Socket.DoConnect(System.Net.EndPoint, System.Net.SocketAddress)
   System.Net.Sockets.Socket.Connect(System.Net.EndPoint)
   System.Net.Sockets.TcpClient.Connect(System.Net.IPEndPoint)
   MongoDB.Driver.Internal.MongoConnection.Open()
   MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoDB.Bson.IO.BsonBuffer, Int32)
   MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoDB.Driver.Internal.MongoRequestMessage)
   MongoDB.Driver.Operations.CommandOperation`1[[System.__Canon, mscorlib]].Execute(MongoDB.Driver.Internal.MongoConnection)
   MongoDB.Driver.MongoServerInstance.Ping(MongoDB.Driver.Internal.MongoConnection)
   MongoDB.Driver.MongoServerInstance.Connect()
   MongoDB.Driver.Internal.MultipleInstanceMongoServerProxy+<>c__DisplayClassd.<ConnectInstance>b__c(System.Object)
   
System.ExecutionEngineException   1   <none>   System.Net.Sockets.Socket.DoConnect(System.Net.EndPoint, System.Net.SocketAddress)
   System.Net.Sockets.Socket.Connect(System.Net.EndPoint)
   System.Net.Sockets.TcpClient.Connect(System.Net.IPEndPoint)
   MongoDB.Driver.Internal.MongoConnection.Open()
   MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoDB.Bson.IO.BsonBuffer, Int32)
   MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoDB.Driver.Internal.MongoRequestMessage)
   MongoDB.Driver.Operations.CommandOperation`1[[System.__Canon, mscorlib]].Execute(MongoDB.Driver.Internal.MongoConnection)
   MongoDB.Driver.MongoServerInstance.Ping(MongoDB.Driver.Internal.MongoConnection)
   MongoDB.Driver.MongoServerInstance.Connect()
   MongoDB.Driver.Internal.MultipleInstanceMongoServerProxy+<>c__DisplayClassd.<ConnectInstance>b__c(System.Object)
   
System.Threading.ThreadAbortException   2   <none>   System.Net.Sockets.Socket.DoConnect(System.Net.EndPoint, System.Net.SocketAddress)
   System.Net.Sockets.Socket.Connect(System.Net.EndPoint)
   System.Net.Sockets.TcpClient.Connect(System.Net.IPEndPoint)
   MongoDB.Driver.Internal.MongoConnection.Open()
   MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoDB.Bson.IO.BsonBuffer, Int32)
   MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoDB.Driver.Internal.MongoRequestMessage)
   MongoDB.Driver.Operations.CommandOperation`1[[System.__Canon, mscorlib]].Execute(MongoDB.Driver.Internal.MongoConnection)
   MongoDB.Driver.MongoServerInstance.Ping(MongoDB.Driver.Internal.MongoConnection)
   MongoDB.Driver.MongoServerInstance.Connect()
   MongoDB.Driver.Internal.MultipleInstanceMongoServerProxy+<>c__DisplayClassd.<ConnectInstance>b__c(System.Object) 

craiggwilson

unread,
May 29, 2014, 10:07:16 AM5/29/14
to mongod...@googlegroups.com
Hi Ron, Sorry you're having some issues. As all these exceptions seem to be related to socket issues, could you enable the diagnostics for System.Net.Sockets? Here is the msdn reference: http://msdn.microsoft.com/en-us/library/ty48b824(v=vs.110).aspx. Note that there has been others with issues running a client on Azure and a host on AWS (I believe this is where MongoHQ is hosted) due to networking issues.

What version of the driver are you using?

Ron Rejwan

unread,
May 29, 2014, 10:35:42 AM5/29/14
to mongod...@googlegroups.com
Hey Craig,
Problem is we get roughly around 12,000,000 requests per day (and growing) and this issue happens every 1-2 days, I'm afraid it will choke our server's space.
Can you suggest a way we could turn it on without choking our server?

We're using the newest C# driver (1.9.1.211), however, it also occurred using the old driver (1.8.x).

craiggwilson

unread,
May 29, 2014, 1:30:35 PM5/29/14
to mongod...@googlegroups.com
I don't have any suggestions about turning it on.  We really only want to see it when there is a problem going on. My guess is that there is some form of DNS issue going on and without this, we can't prove it. Perhaps ping microsoft or their forum's related to turning it on? Sorry that's not more helpful.

Ron Rejwan

unread,
May 29, 2014, 5:40:50 PM5/29/14
to mongod...@googlegroups.com
Alright thanks Craig, I'll add the logs and post once the service hangs again.

Ron Rejwan

unread,
May 30, 2014, 6:47:28 PM5/30/14
to mongod...@googlegroups.com
Hey Craig,
I hadn't set up the socket logs yet - Will warning and above be sufficient? I turned it on verbose, and it grew to 1GB in size within 10 minutes.

However, I did analyze a dump once the hangs started and found some interesting data.
All my service threads are locked in the same place inside the MognoDB C# driver, as their callstack shows:
[[GCFrame]]
[[GCFrame]]
[[HelperMethodFrame] (System.Threading.Monitor.Enter)] System.Threading.Monitor.Enter(System.Object)
MongoDB.Driver.Internal.DiscoveringMongoServerProxy.EnsureInstanceManager(System.TimeSpan)+34
MongoDB.Driver.Internal.DiscoveringMongoServerProxy.ChooseServerInstance(MongoDB.Driver.ReadPreference)+1f
MongoDB.Driver.MongoServer.AcquireConnection(MongoDB.Driver.ReadPreference)+12b
MongoDB.Driver.MongoCollection.InsertBatch(System.Type, System.Collections.IEnumerable, MongoDB.Driver.MongoInsertOptions)+95
MongoDB.Driver.MongoCollection.Insert(System.Type, System.Object, MongoDB.Driver.MongoInsertOptions)+56
mscorlib_ni!System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].InnerInvoke()+4f
mscorlib_ni!System.Threading.Tasks.Task.Execute()+6e
mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+285
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+9
mscorlib_ni!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)+250
mscorlib_ni!System.Threading.Tasks.Task.ExecuteEntry(Boolean)+85
mscorlib_ni!System.Threading.ThreadPoolWorkQueue.Dispatch()+1ea
[[DebuggerU2MCatchHandlerFrame]]
[[ContextTransitionFrame]]
[[DebuggerU2MCatchHandlerFrame]]

And all these threads are basically locked and alive, in effect clogging our server (over 97.5% thread lock for over 30 minutes).

Basically it looks as if some static lock inside the C#'s connection handler doesn't get released, and just locks up any other access to the DB.

Thanks,
Ron

Ron Rejwan

unread,
May 30, 2014, 6:50:19 PM5/30/14
to mongod...@googlegroups.com
And more specifically - this lock:
// private methods
        private void EnsureInstanceManager(TimeSpan timeout)
        {
            if (_serverProxy == null)
            {
                lock (_lock)
                {
                    if (_serverProxy == null)
                    {
                        _connectionAttempt++;
                        _state = MongoServerState.Connecting;
                        Discover(timeout);
                    }
                }
            }
        }

craiggwilson

unread,
May 31, 2014, 11:59:25 AM5/31/14
to mongod...@googlegroups.com
Thanks Ron, I'll take a look at this.

You can avoid the "DiscoveringMongoServerProxy" class completely by being more specific about what type of servers you are connecting to. For instance, if you are connecting to a replica set, put the setName into the connection string. This not only let's us avoid having to discover the type of cluster you're talking to, it also provides some validation to you that you are indeed talking to the replica set you thought you were.

If you are talking to a mongos pool, you can similarly specify connect=shardrouter.

Ron Rejwan

unread,
May 31, 2014, 12:21:35 PM5/31/14
to mongod...@googlegroups.com
Hello Craig, 

I've looked here (http://docs.mongodb.org/manual/reference/connection-string/) and found nothing on the setName option.
Do you happen to have a link explaining how to use it correctly?

Thanks

craiggwilson

unread,
Jun 1, 2014, 6:15:22 PM6/1/14
to mongod...@googlegroups.com
My apologies.  setName is what it is called when a mongod is started up as a replica set parameter.  The correct name for the option is replicaSet. It exists in the docs here -> http://docs.mongodb.org/manual/reference/connection-string/#uri.replicaSet.

Ron Rejwan

unread,
Jun 2, 2014, 11:16:08 AM6/2/14
to mongod...@googlegroups.com
Hey Craig,

I've sent over the dumps to Azure's tech support - here is what they had to say about it:

I checked the dumps and see exactly what we expected after looking at WCF traces. I see that many threads (handling new requests) are waiting to own a lock which is owned by another thread. The owning thread is stuck and not releasing the lock and causing the contention/hang and eventually service hits the MaxConcurrentCalls limit.

 

Both the responsible callstacks are related to Mongo DB here i.e. Thread which are waiting to own a lock and thread callstack owning the lock is from Mongo DB module. Please see the callstack here,

 

Sample Thread callstack waiting to own a .Net Lock

 

878  Id: 60c.16c4 Suspend: 0 Teb: 000007f6`9f312000 Unfrozen

Child-SP          RetAddr           Call Site

000000a9`a36acff8 000007f8`9522129e ntdll!ZwWaitForMultipleObjects+0xa

000000a9`a36ad000 000007f8`8f4614f2 KERNELBASE!WaitForMultipleObjectsEx+0xe5

000000a9`a36ad2e0 000007f8`8f4613a7 clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x62

000000a9`a36ad340 000007f8`8f461236 clr!Thread::DoAppropriateAptStateWait+0x53

000000a9`a36ad380 000007f8`8f46108d clr!Thread::DoAppropriateWaitWorker+0x186

000000a9`a36ad470 000007f8`8f461327 clr!Thread::DoAppropriateWait+0x7d

000000a9`a36ad4f0 000007f8`8f3c584e clr!CLREventBase::WaitEx+0xc0

000000a9`a36ad580 000007f8`8f3c5762 clr!AwareLock::EnterEpilogHelper+0xc6

000000a9`a36ad640 000007f8`8f387abb clr!AwareLock::EnterEpilog+0x62

000000a9`a36ad6a0 000007f8`8f3c5bdb clr!AwareLock::Contention+0x1e3

000000a9`a36ad760 000007f8`2fd1c5d4 clr!JITutil_MonContention+0xaf

000000a9`a36ad8f0 000007f8`2fd1c55f MongoDB_Driver!MongoDB.Driver.Internal.DiscoveringMongoServerProxy.EnsureInstanceManager(System.TimeSpan)+0x34

000000a9`a36ad940 000007f8`2fd1c4cb MongoDB_Driver!MongoDB.Driver.Internal.DiscoveringMongoServerProxy.ChooseServerInstance(MongoDB.Driver.ReadPreference)+0x1f

000000a9`a36ad980 000007f8`3002d815 MongoDB_Driver!MongoDB.Driver.MongoServer.AcquireConnection(MongoDB.Driver.ReadPreference)+0x12b

000000a9`a36ad9e0 000007f8`3002d6e6 MongoDB_Driver!MongoDB.Driver.MongoCollection.InsertBatch(System.Type, System.Collections.IEnumerable, MongoDB.Driver.MongoInsertOptions)+0x95

000000a9`a36ada90 000007f8`8e0410df MongoDB_Driver!MongoDB.Driver.MongoCollection.Insert(System.Type, System.Object, MongoDB.Driver.MongoInsertOptions)+0x56

000000a9`a36adae0 000007f8`8e03c1de mscorlib_ni!System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].InnerInvoke()+0x4f

000000a9`a36adb30 000007f8`8e0ed0b5 mscorlib_ni!System.Threading.Tasks.Task.Execute()+0x6e

 

 

Sample Thread callstack owning a .Net Lock (and taking time to release it)L

 

0:117> kL

Child-SP          RetAddr           Call Site

000000a9`86d2d008 000007f8`9522129e ntdll!ZwWaitForMultipleObjects+0xa

000000a9`86d2d010 000007f8`8f4614f2 KERNELBASE!WaitForMultipleObjectsEx+0xe5

000000a9`86d2d2f0 000007f8`8f4613a7 clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x62

000000a9`86d2d350 000007f8`8f461236 clr!Thread::DoAppropriateAptStateWait+0x53

000000a9`86d2d390 000007f8`8f46108d clr!Thread::DoAppropriateWaitWorker+0x186

000000a9`86d2d480 000007f8`8f461327 clr!Thread::DoAppropriateWait+0x7d

000000a9`86d2d500 000007f8`8f3d1b5e clr!CLREventBase::WaitEx+0xc0

000000a9`86d2d590 000007f8`8f3d1b38 clr!Thread::Block+0x1e

000000a9`86d2d5c0 000007f8`8f3d1983 clr!SyncBlock::Wait+0x150

(Inline Function) --------`-------- clr!Object::Wait+0xb

000000a9`86d2d700 000007f8`2fd1cad5 clr!ObjectNative::WaitTimeout+0xc7

000000a9`86d2d880 000007f8`2fd1c73d MongoDB_Driver!MongoDB.Driver.Internal.BlockingQueue`1[[System.__Canon, mscorlib]].Dequeue(System.TimeSpan)+0xd5

000000a9`86d2d910 000007f8`2fd1c5f8 MongoDB_Driver!MongoDB.Driver.Internal.DiscoveringMongoServerProxy.Discover(System.TimeSpan)+0xdd

000000a9`86d2d980 000007f8`2fd1c55f MongoDB_Driver!MongoDB.Driver.Internal.DiscoveringMongoServerProxy.EnsureInstanceManager(System.TimeSpan)+0x58

000000a9`86d2d9d0 000007f8`2fd1c4cb MongoDB_Driver!MongoDB.Driver.Internal.DiscoveringMongoServerProxy.ChooseServerInstance(MongoDB.Driver.ReadPreference)+0x1f

000000a9`86d2da10 000007f8`3002d815 MongoDB_Driver!MongoDB.Driver.MongoServer.AcquireConnection(MongoDB.Driver.ReadPreference)+0x12b

000000a9`86d2da70 000007f8`3002d6e6 MongoDB_Driver!MongoDB.Driver.MongoCollection.InsertBatch(System.Type, System.Collections.IEnumerable, MongoDB.Driver.MongoInsertOptions)+0x95

000000a9`86d2db20 000007f8`8e0410df MongoDB_Driver!MongoDB.Driver.MongoCollection.Insert(System.Type, System.Object, MongoDB.Driver.MongoInsertOptions)+0x56

000000a9`86d2db70 000007f8`8e03c1de mscorlib_ni!System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].InnerInvoke()+0x4f

000000a9`86d2dbc0 000007f8`8e0ed0b5 mscorlib_ni!System.Threading.Tasks.Task.Execute()+0x6e

 

 

Here’s the code where above thread (owing the lock) is stuck and causing the contention. The timeout passed to Monitor.Wait is 30 sec.

 

// MongoDB.Driver.Internal.BlockingQueue<T>
public T Dequeue(TimeSpan timeout)
{
    object syncRoot;
    Monitor.Enter(syncRoot = this._syncRoot);
    T result;
    try
    {
        DateTime d = DateTime.UtcNow + timeout;
        while (this._queue.Count == 0)
        {
            TimeSpan timeSpan = d - DateTime.UtcNow;
            if (!(timeSpan > TimeSpan.Zero))
            {
                result = default(T);
                return result;
            }
            Monitor.Wait(this._syncRoot, timeSpan);        <<<< All the dumps shows that here Dequeue method seems to be waiting for long and causing contention
        }

 

 

 

I also see following exceptions in the memory which seems to be related as thrown from same method causing contention.

 

MongoDB.Driver.MongoConnectionException: Unable to connect in the specified timeframe of '00:00:30'.        <<<< There’re around 23 instance of this exception in memory

   MongoDB.Driver.Internal.DiscoveringMongoServerProxy.Discover(System.TimeSpan) 
   MongoDB.Driver.Internal.DiscoveringMongoServerProxy.EnsureInstanceManager(System.TimeSpan) 
   MongoDB.Driver.Internal.DiscoveringMongoServerProxy.ChooseServerInstance(MongoDB.Driver.ReadPreference) 
   MongoDB.Driver.MongoServer.AcquireConnection(MongoDB.Driver.ReadPreference) 
   MongoDB.Driver.MongoCollection.InsertBatch(System.Type, System.Collections.IEnumerable, MongoDB.Driver.MongoInsertOptions) 
   MongoDB.Driver.MongoCollection.Insert(System.Type, System.Object, MongoDB.Driver.MongoInsertOptions) 
   System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].InnerInvoke() 
   System.Threading.Tasks.Task.Execute()

 

System.ObjectDisposedException

Cannot access a disposed object.

   MongoDB.Driver.Internal.DiscoveringMongoServerProxy.Discover(System.TimeSpan) 
   MongoDB.Driver.Internal.DiscoveringMongoServerProxy.EnsureInstanceManager(System.TimeSpan) 
   MongoDB.Driver.Internal.DiscoveringMongoServerProxy.ChooseServerInstance(MongoDB.Driver.ReadPreference) 
   MongoDB.Driver.MongoServer.AcquireConnection(MongoDB.Driver.ReadPreference) 
   MongoDB.Driver.MongoCollection.InsertBatch(System.Type, System.Collections.IEnumerable, MongoDB.Driver.MongoInsertOptions) 
   MongoDB.Driver.MongoCollection.Insert(System.Type, System.Object, MongoDB.Driver.MongoInsertOptions) 
   System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].InnerInvoke() 
   System.Threading.Tasks.Task.Execute()

 

 

I think the former exception in above list MongoDB.Driver.MongoConnectionException is thrown by DiscoveringMongoServerProxy.Discover because BlockingQueue`1[[System.__Canon, mscorlib]].Dequeue didn’t finish in specified Timeout (30 sec) and eventually Discover threw the exception. So, it’s confirmed that BlockingQueue`1[[System.__Canon, mscorlib]].Dequeue looks to be taking time to finish.

 

Summary

=========

The cause of the hang is confirmed as we expected that Mongo DB module is causing the contention and slow processing of request. In the meantime, requests keep piling up and eventually hitting the MaxConcurrentCalls limit and client application receives SocketExceptions.

 

What can be done

=================

Please connect with Mongo DB support team and check with them why above code could be creating contention. Do they’ve any better suggestions to avoid the contention.

Ron Rejwan

unread,
Jun 5, 2014, 2:31:57 AM6/5/14
to mongod...@googlegroups.com
Hello Craig,
Looking at the Net.Socket logs I can't see anything special besides the sockets timing out
System.Net.Sockets Error: 0 : [3508] Socket#30945470::UpdateStatusAfterSocketError() - TimedOut
System.Net.Sockets Error: 0 : [3508] Exception in Socket#30945470::Connect - A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond XXXX.
System.Net.Sockets Error: 0 : [3516] Socket#45214701::UpdateStatusAfterSocketError() - TimedOut
System.Net.Sockets Error: 0 : [3516] Exception in Socket#45214701::Connect - A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond XXXX.
System.Net.Sockets Error: 0 : [3376] Socket#8780571::UpdateStatusAfterSocketError() - TimedOut
System.Net.Sockets Error: 0 : [3376] Exception in Socket#8780571::Connect - A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond XXXX.
System.Net.Sockets Error: 0 : [3520] Socket#43356036::UpdateStatusAfterSocketError() - TimedOut
...

craiggwilson

unread,
Jun 5, 2014, 1:36:31 PM6/5/14
to mongod...@googlegroups.com
Sorry for the belated reply, been traveling.

As I mentioned earlier, you can completely avoid this code path by setting the "replicaSet" option on your connection string. We won't use the DiscoveringMongoServerProxy in this. Other than that, I very much appreciate you digging into this. We'll work on getting it fixed so that others don't have the same problem.

Craig

Ron Rejwan

unread,
Jun 5, 2014, 1:43:46 PM6/5/14
to mongod...@googlegroups.com
Hey Craig,
I've turned on the replicaSet parameter, we'll wait a few days and see how it goes.

Thanks,
Ron

Ron Rejwan

unread,
Jun 7, 2014, 5:27:59 PM6/7/14
to mongod...@googlegroups.com
Hello Craig,

I'm afraid while the 'replicaSet' did stop the thread lock, the problem has migrated elsewhere.
Every 1-2 days we will suddenly get hundreds to thousands of timeouts to the DB, a thing which a simple restart will solve.

Looking in the socket logs I see nothing out of the ordinary, only a simple socket connecion timeout.

Do you have any other suggestion on how to proceed?

craiggwilson

unread,
Jun 9, 2014, 2:00:31 PM6/9/14
to mongod...@googlegroups.com
Hi Ron,
  Please go to jira.mongodb.com and open a CSHARP ticket.  We can continue this discussion there. I'm currently taking the stance something is going on with Azure as we have seen this behavior before, and only when running on Azure. I'd like to have this in our system so we can correlate the tickets together.

Sorry you are having issues,
Craig

Ron Rejwan

unread,
Jun 10, 2014, 6:28:15 AM6/10/14
to mongod...@googlegroups.com
Hey Craig,

Here is a link to the Jira I've opened - please let me know if you need any more information:
Reply all
Reply to author
Forward
0 new messages