write concern error (unable to connect to shard) when router is under load

[bot head]

Hello,

I am experiencing a exception that I don't understand and am hoping for some guidance on what the issue could be.

First my environment:

• I have a router setup with 2 shards (each has a replica set of 2 secondary).
• I am using a connection string: mongodb://ipaddress/db?w=majority&journal=true
• I have 8 worker roles that are spinning and hitting the router with a query to collectionA and then an insert to collection, over and over again based on messages from Azure Event Hub that are coming into the worker roles.
• I am hitting about 5K inserts per second, 5K queries per second, and 5K ops per second.
• I am using the C# mongodb driver in my worker roles with a singleton instance of the client/server/collection in the application.

When I ramp up and start hitting the throughput metrics I previously mentioned I start seeing the following exception occur in my application logs (full exception at end of post):

WriteConcern detected an error 'write results unavailable from DevAShd2:27017 :: caused by :: Location13328 sharded connection pool: connect failed DevAShd2:27017 : couldn't connect to server DevAShd2:27017 (10.0.2.16), connection attempt failed'. (Response was { "ok" : 1, "code" : 83, "err" : "write results unavailable from DevAShd2:27017 :: caused by :: Location13328 sharded connection pool: connect failed DevAShd2:27017 : couldn't connect to server DevAShd2:27017 (10.0.2.16), connection attempt failed", "n" : NumberLong(0) }

I do not see this exception under low load.  Does anyone have any suggestions about what may be going on here?

Full Exception:

<TraceSource>General</TraceSource><Object>Timestamp: 11/5/2014 6:07:57 PMMessage: HandlingInstanceID: cc3deb56-d8ce-4a4c-a164-e82d2a2d42f1An exception of type '.Common.Data.Repository.RepositoryException' occurred and was caught.-------------------------------------------------------------------------------------------------11/05/2014 18:07:57Type : .Common.Data.Repository.RepositoryException, .Common.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMessage : WriteConcern detected an error 'write results unavailable from DevAShd2:27017 :: caused by :: Location13328 sharded connection pool: connect failed DevAShd2:27017 : couldn't connect to server DevAShd2:27017 (10.0.2.16), connection attempt failed'. (Response was { "ok" : 1, "code" : 83, "err" : "write results unavailable from DevAShd2:27017 :: caused by :: Location13328 sharded connection pool: connect failed DevAShd2:27017 : couldn't connect to server DevAShd2:27017 (10.0.2.16), connection attempt failed", "n" : NumberLong(0) }).Source : .Common.Data.MongoDBHelp link : Repository : .Common.Data.MongoDB.DflplatformMongoRepository`2Method : InsertArguments : entity ( .Common.Data.Entities.Series.NumericSeriesplatform ) - { "_t" : "NumericSeriesplatform", "s" : CSUUID("f952f445-d3cd-4c71-a9fe-394fcfe58822"), "l" : null, "t" : ISODate("2014-11-05T18:02:53.687Z"), "v" : 54971.0 }Data : System.Collections.ListDictionaryInternalTargetSite : T Insert(T)HResult : -2146233088Stack Trace :    at .Common.Data.MongoDB.DflplatformMongoRepository`2.Insert(T entity) in d:\Sandbox\myapp\platform\DFL\src\.Common\.Common.Data.MongoDB\DflplatformMongoRepository.cs:line 258   at System.Threading.Tasks.Task`1.InnerInvoke()   at System.Threading.Tasks.Task.Execute()--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()   at DH.Services.EventHub.DataLayer.Message.Sender.DflplatformSender.&lt;PersistMessage&gt;d__5`3.MoveNext() in d:\Sandbox\myapp\platform\DFL\src\DH\DH.Services.EventHub.DataLayer\Message\Sender\DflplatformSender.cs:line 130--- End of stack trace from previous location where exception was thrown ---   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at .Common.Core.ExceptionHandling.EntLibExceptionHandler.&lt;ProcessAsync&gt;d__0.MoveNext() in d:\Sandbox\myapp\platform\DFL\src\.Common\.Common.Core\ExceptionHandling\EntLibExceptionHandler.cs:line 87Additional Info:MachineName : RD00155D3A7AEDTimeStamp : 11/5/2014 6:07:57 PMFullName : Microsoft.Practices.EnterpriseLibrary.ExceptionHandling, Version=6.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35AppDomainName : RdRuntimeThreadIdentity : WindowsIdentity : NT AUTHORITY\NETWORK SERVICEInner Exception---------------Type : MongoDB.Driver.WriteConcernException, MongoDB.Driver, Version=1.9.2.235, Culture=neutral, PublicKeyToken=f686731cfb9cc103Message : WriteConcern detected an error 'write results unavailable from DevAShd2:27017 :: caused by :: Location13328 sharded connection pool: connect failed DevAShd2:27017 : couldn't connect to server DevAShd2:27017 (10.0.2.16), connection attempt failed'. (Response was { "ok" : 1, "code" : 83, "err" : "write results unavailable from DevAShd2:27017 :: caused by :: Location13328 sharded connection pool: connect failed DevAShd2:27017 : couldn't connect to server DevAShd2:27017 (10.0.2.16), connection attempt failed", "n" : NumberLong(0) }).Source : MongoDB.DriverHelp link : WriteConcernResult : MongoDB.Driver.WriteConcernResultCommandResult : MongoDB.Driver.WriteConcernResultData : System.Collections.ListDictionaryInternalTargetSite : System.Collections.Generic.IEnumerable`1[MongoDB.Driver.WriteConcernResult] Execute(MongoDB.Driver.Internal.MongoConnection)HResult : -2146233088Stack Trace :    at MongoDB.Driver.Operations.InsertOpcodeOperationEmulator.Execute(MongoConnection connection)   at MongoDB.Driver.Operations.InsertOpcodeOperation.Execute(MongoConnection connection)   at MongoDB.Driver.MongoCollection.InsertBatch(Type nominalType, IEnumerable documents, MongoInsertOptions options)   at MongoDB.Driver.MongoCollection.Insert(Type nominalType, Object document, MongoInsertOptions options)   at .Common.Data.MongoDB.DflplatformMongoRepository`2.Insert(T entity) in d:\Sandbox\myapp\platform\DFL\src\.Common\.Common.Data.MongoDB\DflplatformMongoRepository.cs:line 251Category: GeneralPriority: 0EventId: 100Severity: ErrorTitle:Enterprise Library Exception HandlingMachine: RD00155D3A7AEDApp Domain: RdRuntimeProcessId: 1440Process Name: E:\base\x64\WaWorkerHost.exeThread Name: Win32 ThreadId:2948Extended Properties: </Object>

[bot head]

Correction: query to collectionA and then an insert to collectionB