C# driver. EndOfStreamException: Attempted to read past the end of the stream. Possibly malformed responses from mongo server.

[Nikhil Sinha]

Hello,

I'm using version 2.2.4.26 of the C# driver to query Mongo.  My mongo version is 3.2.  I'm running it on CentOS.  The replica set has two members, no sharding.

I've posted the relevant stack trace below.  This seem to be localized at a certain time, and if I wait a while, they do go away.  It appears mongo sends the message length, but then no message.  It was an addToSet query, and when I look at the data, it looks like the write was successful.  It's just the response that's bad.  There aren't any errors in the mongo log on the server.  How do I debug these?

MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server. ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.

   at MongoDB.Driver.Core.Misc.StreamExtensionMethods.ReadBytes(Stream stream, Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)

   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer()

   --- End of inner exception stack trace ---

   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer()

   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer(Int32 responseTo, CancellationToken cancellationToken)

   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)

   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquiredConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)

   at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)

   at MongoDB.Driver.Core.Servers.ClusterableServer.ServerChannel.ExecuteProtocol[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)

   at MongoDB.Driver.Core.Servers.ClusterableServer.ServerChannel.Command[TResult](DatabaseNamespace databaseNamespace, BsonDocument command, IElementNameValidator commandValidator, Func`1 responseHandling, Boolean slaveOk, IBsonSerializer`1 resultSerializer, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)

   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase.ExecuteProtocol(IChannelHandle channel, BsonDocument command, Func`1 responseHandling, CancellationToken cancellationToken)

   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase.ExecuteBatch(IChannelHandle channel, BatchableSource`1 requestSource, Int32 originalIndex, CancellationToken cancellationToken)

   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase.ExecuteBatches(IChannelHandle channel, CancellationToken cancellationToken)

   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase.Execute(IChannelHandle channel, CancellationToken cancellationToken)

   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.ExecuteBatch(IChannelHandle channel, Run run, Boolean isLast, CancellationToken cancellationToken)

   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.Execute(IWriteBinding binding, CancellationToken cancellationToken)

   at MongoDB.Driver.OperationExecutor.ExecuteWriteOperation[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)

   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperation[TResult](IWriteOperation`1 operation, CancellationToken cancellationToken)

   at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)

   at MongoDB.Driver.MongoCollectionBase`1.UpdateOne(FilterDefinition`1 filter, UpdateDefinition`1 update, UpdateOptions options, CancellationToken cancellationToken)

[Wan Bachtiar]

Hi Nikhil,

It’s been a while since you posted the question, have you found a solution for the issue?

The replica set has two members

This seem to be localized at a certain time, and if I wait a while, they do go away

The minimum recommended configuration for a replica set is three nodes, which consists of a primary, a secondary and an arbiter. However, most deployments will keep three nodes that store data : a primary and two secondaries. See also Three node replica sets.

This is because voting works on a quorum basis, where a strict majority of voting nodes must be available in order to elect or maintain a primary. In your case of two nodes, if one of the node is unavailable the primary will step down to become a secondary. Only when the other node is reachable again, a replica set election will be performed for a new primary.

I would recommend to add an extra node to your replica set for high availability.

It was an addToSet query, and when I look at the data, it looks like the write was successful.

MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server. -→ System.IO.EndOfStreamException: Attempted to read past the end of the stream.

Considering it’s a MongoConnectionException error during a write operation and recovered on it’s own, it’s a possibility that it was caused by a brief lost of the primary node. The connections were most likely disconnected when a node changed from primary and to secondary. The driver was then expecting a response that never comes, triggering an end of steam exception.

There aren’t any errors in the mongo log on the server. How do I debug these?

You could try searching for log entries related to heartbeat failure, or similar to:

 I REPL     [ReplicationExecutor] Stepping down from primary in response to heartbeat
 I REPL     [replExecDBWorker-1] transition to SECONDARY

If you are still seeing the same issue with a three nodes replica set, could you provide the following relevant information:

• MongoDB specific server version. i.e. 3.2.8.
• Snippet C# code containing the addToSet query
• The value of your replication write concern

Kind regards,

Wan.

​