"An existing connection was forcibly closed by the remote host" I'm seeing many of these errors since switching to the 10Gen supported c# driver

Tom DeMille

unread,

Dec 27, 2010, 12:56:07 PM12/27/10

to mongodb-user

I'm seeing a ton of these errors

"An existing connection was forcibly closed by the remote host"

in the Mongo log since I switched from the open source c# driver to
the 10Gen supported one. Once this error happens my websites become
unresponsive. I have a feeling it has something to do with the way
connection pooling is implemented. See log below, I'm deleting
indexes from a collection, removing all items, then 'recreating' the
collection by adding all the rows I want and then re indexing the
collection. A few moments after that is all done I invariably get the
error : 'An existing connection was forcibly closed by the remote
host'

[conn46] run command admin.$cmd { ismaster: 1 }
[conn46] query admin.$cmd ntoreturn:1 command: { ismaster: 1 }
reslen:64 0ms
[conn46] query BtSearchCache.system.indexes reslen:1407 nreturned:
7 0ms

drop indexes
[conn46] run command BtSearchCache.$cmd { deleteIndexes:
"Board_41", index: "ix_QuickSearch" }
[conn46] CMD: dropIndexes BtSearchCache.Board_41
[conn46] dropIndexes: ix_QuickSearch not found
[conn46] query BtSearchCache.$cmd ntoreturn:1 command:
{ deleteIndexes: "Board_41", index: "ix_QuickSearch" } reslen:81 1ms

remove all data from collection
[conn46] remove query: {} 1428ms

insert data (thousands of rows, omitted here)

[conn46] insert BtSearchCache.Board_41 0ms
[conn46] insert BtSearchCache.Board_41 0ms
[conn46] insert BtSearchCache.Board_41 1ms
[conn46] insert BtSearchCache.Board_41 0ms
[conn46] insert BtSearchCache.system.indexes 0ms

create indexes
[conn46] run command admin.$cmd { getlasterror: 1 }
[conn46] query admin.$cmd ntoreturn:1 command: { getlasterror: 1 }
reslen:65 0ms

ERROR a few minutes later
[conn46] MessagingPort recv() errno:10054 An existing connection
was forcibly closed by the remote host. 192.168.100.216:51111
[conn46] SocketException: 9001 socket exception
[conn46] end connection 192.168.100.216:51111

Tom DeMille

unread,

Dec 27, 2010, 1:10:07 PM12/27/10

to mongodb-user

A bit more information:

I don't believe this is a server connectivity issue as nothing else
has changed in our environment. We just pushed up new code on the web
server. Mongo is on the DB server. Nothing changes with firewalls,
mongo has been running fine, I can connect via the local console
although after this issue happens a bunch of times I will notice that
the local console is doing a 'reconnect'.

Looking through the old logs there are no instances of this until we
went live with the new software. It seems like it has something to do
with the connection pooling. Perhaps I'm using the driver
incorrectly.

This is being called from a single threaded console application in
some instances and in other instances from our ASP.Net web
application, which gets 3-4000 hits a minute.

In the web app, the code is thus:

var mongoServer =
MongoServer.Create(Common.SearchCacheConnectionString);
var mongoDB = mongoServer.GetDatabase("BtSearchCache");
MongoCollection btSearchCache = mongoDB.GetCollection("Board_" +
boardID);
return btSearchCache.Count(BuildMongoQuery(query));

I'm not using any using statements or explicitly calling close
anywhere, is that correct?

There could be many instances of the mongoserver in the application as
each search runs the code above. I assume the connection pooling is
meant to handle that?

Robert Stam

unread,

Dec 27, 2010, 1:18:27 PM12/27/10

to mongodb-user

What version of the 10gen C# driver are you using?

What exceptions are you seeing client side?

The driver will close a connection if an error occurs on the
connection (which might explain why the server is seeing "connection
was forcibly closed"). But if that's what is happening we need to
figure out what errors the client is seeing.

Robert Stam

unread,

Dec 27, 2010, 1:28:07 PM12/27/10

to mongodb-user

It's OK to call MongoServer.Create as many times as you want, as long
as the value for the connection string is the same. A new instance of
MongoServer will be created the first time it is called, and all
future calls to MongoServer.Create with the same connection string
return the same instance. Therefore there should only be one
connection pool.

It's OK that you don't have any using statements or explicit calls to
Close. The only time a using statement is recommended is when calling
RequestStart (to ensure that RequestDone is called automatically).

It might be helpful to know what query you are using. You could
convert the query to JSON using:

var json = BuildMongoQuery(query).ToJson();

Also, if there is any way to find out what exceptions (if any) the
client side is seeing that would be helpful.

Robert Stam

unread,

Dec 27, 2010, 1:40:54 PM12/27/10

to mongodb-user

Are you calling MongoServer.Disconnect or MongoServer.Reconnect from
anywhere in your client code? That's another way the client might
close all of its connections.

The driver also closes connections that haven't been used for 10
minutes or more, so it might be completely normal that you are seeing
closed connections. The slowness you are seeing in your web site might
have a different cause.

Tom DeMille

unread,

Dec 27, 2010, 1:41:22 PM12/27/10

to mongodb-user

I'm seeing errors like this: It's not a firewall or connectivity
issue because we run Sql Server on the same box and if SqlServer goes
down our entire site goes down.
===============================================

System.Web.HttpUnhandledException: Exception of type
'System.Web.HttpUnhandledException' was thrown. --->
System.IO.IOException: Unable to write data to the transport
connection: An existing connection was forcibly closed by the remote
host. ---> System.Net.Sockets.SocketException: An existing connection

was forcibly closed by the remote host

at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset,
Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32
offset, Int32 size)
--- End of inner exception stack trace ---
at
BoomTown.DataServices.MongoWrapper.ListingService.GetListingSearchCount(Int32
boardID, NameValueCollection query)
--- End of inner exception stack trace ---
at
BoomTown.DataServices.MongoWrapper.ListingService.GetListingSearchCount(Int32
boardID, NameValueCollection query)

MongoDB.Driver.MongoConnectionException: Unable to connect to server
---> System.Net.Sockets.SocketException: No connection could be made
because the target machine actively refused it 192.168.100.186:27017
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot,
SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.Net.Sockets.TcpClient.Connect(IPEndPoint remoteEP)
at
MongoDB.Driver.Internal.MongoConnection..ctor(MongoConnectionPool
connectionPool, IPEndPoint endPoint)
at MongoDB.Driver.Internal.DirectConnector.Connect(IPEndPoint
endPoint, TimeSpan timeout)
at MongoDB.Driver.Internal.DirectConnector.Connect(TimeSpan
timeout)
--- End of inner exception stack trace ---
at
BoomTown.DataServices.MongoWrapper.ListingService.GetListingSearchCount(Int32
boardID, NameValueCollection query)

Tom DeMille

unread,

Dec 27, 2010, 1:47:20 PM12/27/10

to mongodb-user

Here is a typical query: We use short column names in mongo to keep
the collection small, so 'e' might equate to something like
Description

"{ \"e\" : { \"$gte\" : 2 }, \"i\" : { \"$gte\" : 1 }, \"h\" : { \"$gte
\" : 750 }, \"g\" : { \"$gte\" : 60000, \"$lte\" : 60000 }, \"RN\" :
{ \"$ne\" : 1 }, \"CL\" : { \"$ne\" : 1 }, \"$or\" : [{ \"aj\" : 1 },
{ \"ai\" : 1 }] }"

Robert Stam

unread,

Dec 27, 2010, 1:51:57 PM12/27/10

to mongodb-user

Are you seeing "connection was forcibly closed by the the remote host"
on BOTH the client and the server?

For the first exception: the client can't close the connection on the
server side, so it's hard to see what the driver might have done to
cause this exception. I'll ask what might cause the server to
spontaneously close a connection.

For the second exception: assuming the IP address is correct this
clearly indicates that the server is refusing connections. How are you
hosting mongodb? Is it being restarted and we are seeing refused
connections until the server finishes restarting?

Don't know if you are running with safe mode turned on (add ".../?
safe=true" to the end of your connection string). SafeMode can help
detect problems earlier.

Are you using replica sets?

Tom DeMille

unread,

Dec 27, 2010, 1:53:32 PM12/27/10

to mongodb-user

Well the thing is... We have a switch in our config file to turn on/
off the usage of mongo. If off, the web site does things the old way
using the Sql Server DB. If on then it sends the query to Mongo
instead. So I can turn Mongo on and off at will.

Since we pushed live. Much downtime, 3,4,5 minutes at a time.

If I turn Mongo OFF. Zero downtime. I turned it off on the 24th and
not a minute of downtime until this morning. I turned it back on and
we had 3 outages of 2,3, and 5 minutes within the 1st 1/2 hour. I
turned it back off and no outages.

As a long time developer, 15 plus years, I'm pretty good at
eliminating environmental causes and narrowing my debugging to a
scenario that enables me to concentrate on the issue at hand. I'm
just about certain that this is being caused by sockets hanging
asp.net threads, we had a similar issue with nCache 6 months ago and
it was due to the connection pooling that I had written. When the app
pool recycled, nCache did not release it's sockets and it was causing
mayhem. I had to hook into the IRegisteredObject interface and make
sure I was explicitly cleaning up resources when the app pool
recycled..

System.AppDomain.CurrentDomain.DomainUnload += new
EventHandler(CurrentDomain_DomainUnload);
System.AppDomain.CurrentDomain.ProcessExit += new
EventHandler(CurrentDomain_ProcessExit);
System.AppDomain.CurrentDomain.UnhandledException += new
UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);

To me this problem feels very similar to that issue, the symptoms
anyway, not necessarily the cause, but related in that it's a
connection pool using sockets..

Tom DeMille

unread,

Dec 27, 2010, 1:57:40 PM12/27/10

to mongodb-user

not running with safe mode..

so my connection string is this:

mongodb://192.168.0.88/BtSearchCache

it should be this for safe mode? mongodb://192.168.0.88/BtSearchCache/safe=true

On the server I see this error:

[conn46] MessagingPort recv() errno:10054 An existing connection
was forcibly closed by the remote host. 192.168.100.216:51111
[conn46] SocketException: 9001 socket exception
[conn46] end connection 192.168.100.216:51111

On the client I see

System.IO.IOException: Unable to write data to the transport
connection: An existing connection was forcibly closed by the remote
host.
---> System.Net.Sockets.SocketException: An existing connection
was forcibly closed by the remote host

So, yes, I see the same exact error in the ASP.net error reporting and
the MongDB Log file..

Tom DeMille

unread,

Dec 27, 2010, 2:00:17 PM12/27/10

to mongodb-user

Not using replicat sets, pretty vanilla, once instance...

Robert Stam

unread,

Dec 27, 2010, 2:04:45 PM12/27/10

unread,

Dec 27, 2010, 3:16:24 PM12/27/10

to mongodb-user

Robert, perhaps this is a clue. The issue at times seems to happen
after I run this code. The purpose of the code is to remove all
indexes, remove everything in the collection, then rebuild the
collection from scratch and then re-index it. Once a day, we rebuild
each collection from SQL Server this way. Deleting the indexes causes
errors to be logged in Mongo at times because sometimes one or the
other index does not exist for that particular collection.

var mongoServer =
MongoServer.Create(Common.SearchCacheConnectionString);
var mongoDB = mongoServer.GetDatabase("BtSearchCache");
MongoCollection btSearchCache =
mongoDB.GetCollection("Board_" + boardID);

if (btSearchCache.GetIndexes().Count() > 1)
{
try
{
btSearchCache.DropIndexByName("ix_QuickSearch");
btSearchCache.DropIndexByName("ix_LatLong");
}
catch
{
//current version of mongo driver doesn't have
index.exists, this errors out if index does not exist
}
}

btSearchCache.RemoveAll();

populate again..

using (dr)
{
while (dr.Read())
{
//add doc to mongo
}
}

then add indexes

btSearchCache.EnsureIndex(IndexKeys.Ascending(mapBankList.ToArray()),
IndexOptions.SetName("ix_LatLong"));

btSearchCache.EnsureIndex(IndexKeys.Ascending(mapLatLongList.ToArray()),
IndexOptions.SetName("ix_LatLong"));

usually an error occurs after this has completed..

Robert Stam

unread,

Dec 27, 2010, 3:34:53 PM12/27/10

to mongodb-user

I assume that this once a day process runs from a command line
application?

Are these same collections being accessed in real time by the web
application? How is the web application supposed to handle the fact
that this collection might be in the process of being rebuilt? The
collection might be incomplete or locked for an extended period of
time while the indexes are being rebuilt.

A few notes on the code:

- MongoCollection has a method called IndexExists (are you running an
old version of the driver?)
- You could call DropAllIndexes instead of dropping one by one
- You might want to call CreateIndex instead of EnsureIndex

Please verify that you are running with v0.9 or newer of the driver.

Robert Stam

unread,

Dec 27, 2010, 4:01:31 PM12/27/10

to mongodb-user

I don't think:

"An existing connection was forcibly closed by the remote host"

is actually an error. This is normal text that is printed every time
the client closes a connection (I just hadn't seen it before because I
wasn't running -v on mongod).

The fact that connections are being closed a lot is just a symptom
that either:

1. The currently hard coded connection pool size of 10 is not big
enough
2. The server is very slow for awhile for some reason causing a large
backlog of queries (and therefore connections)

If you assume that "An existing connection was forcibly closed" is
just a normal part of operation what other errors remain?

Thomas DeMille

unread,

Dec 27, 2010, 4:20:29 PM12/27/10

to mongod...@googlegroups.com

it doesn't seem like a normal type of log entry because it has Exception and errno all over it

MessagingPort recv() errno:10054 An existing connection was forcibly closed by the remote host. 192.168.100.216:56149

Mon Dec 27 15:57:23 [conn28] SocketException: 9001 socket exception

Mon Dec 27 15:57:23 [conn28] end connection 192.168.100.216:56149

So I'm not sure that is normal or it would probably just say something like 'connection closed'

But, even if that WAS normal, the driver client is throwing a similar exception, as I noted in my pasting in of the client exceptions, and this is in turn throwing up error pages on my web site and then eventually freezing the site.

So since MongoDB is supposed to support extremely high volume, it seems as though there is an issue here that needs to be solved either via the new c# driver or via the mongo server product. I wouldn't consider my little 2,3, or 4000 hits per minute HUGE volume, and only a percentage of those result in mongo queries, so I'm probably only running 300 mongo queries a minute. If Mongo can't support that without freezing my site then why am I using it?

And since this never happened before, I suspect the c# driver, as that is the only thing that has changed in my environment.

Lastly, I might try editing the driver and changing the connection pool settings but that should be configurable or I'm going to lose it every time you push a new release

Tom DeMille

unread,

Dec 27, 2010, 4:22:26 PM12/27/10

to mongodb-user

I have it turned on for all sites now and I'm seeing current
connections in the range of 4 to 10, so I don't think mongo is running
out of connections

Robert Stam

unread,

Dec 27, 2010, 4:31:44 PM12/27/10

to mongodb-user

Sorry, I should have been more clear. "An existing connection was
forcibly closed" is only normal on the server side. I didn't mean it
was normal at the client side. I agree that it doesn't look like a
normal log entry, but I get the exact same entries in my server log
when I close a connection cleanly:

Mon Dec 27 15:56:52 [conn1] MessagingPort recv() errno:10054 An
existing connect
ion was forcibly closed by the remote host. 127.0.0.1:35733
Mon Dec 27 15:56:52 [conn1] SocketException: 9001 socket exception
Mon Dec 27 15:56:52 [conn1] end connection 127.0.0.1:35733

What I meant is that I think we can concentrate on looking at other
errors.

Can we correlate any client side exceptions with the *exact* query
that was issued and resulted in an exception?

It's OK to suspect the C# driver... it seems as likely an explanation
as anything else at this point. We just need to get to a point where
we understand exactly what is happening.

And the size of the connection pool should DEFINITELY be configurable.
It's on my todo list for the near future.

On Dec 27, 4:20 pm, Thomas DeMille <tdemi...@gmail.com> wrote:
> it doesn't seem like a normal type of log entry because it has Exception and
> errno all over it
>
> MessagingPort recv() errno:10054 An existing connection was forcibly closed
> by the remote host. 192.168.100.216:56149
> Mon Dec 27 15:57:23 [conn28] SocketException: 9001 socket exception
> Mon Dec 27 15:57:23 [conn28] end connection 192.168.100.216:56149
>
> So I'm not sure that is normal or it would probably just say something like
> 'connection closed'
>
> But, even if that WAS normal, the driver client is throwing a similar
> exception, as I noted in my pasting in of the client exceptions, and this is
> in turn throwing up error pages on my web site and then eventually freezing
> the site.
>

> So since MongoDB is supposed to support *extremely high *volume, it seems as

Thomas DeMille

unread,

Dec 27, 2010, 4:40:23 PM12/27/10

to mongod...@googlegroups.com

I have added logging to log the query when this error happens on the client side

I also added /safe=true to the end of the connection string

actually, since I added that... no issues.. but it has only been 1/2 hour so maybe I'm being overly optimistic!

I will update this thread after some time passes

T

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

Robert Stam

unread,

Dec 28, 2010, 1:01:30 AM12/28/10

to mongodb-user

Someone else reported a reproducible scenario resulting in this
exception at the client:

System.IO.IOException: Unable to write data to the transport

connection: An existing connection was forcibly closed by the remote
host.

See:

http://jira.mongodb.org/browse/CSHARP-143

I don't think this other scenario is the same as yours (it is about
large batch inserts), but it does illustrate that the server can/will
close a connection if presented with bad data.

If there was any way to correlate the exact client code statement that
ran just before the failure with any error messages in the mongod log
that would identify why the server closed the connection maybe we can
progress a bit further.

Of course, there's always packet sniffing if necessary, but that's
tedious.

On Dec 27, 4:20 pm, Thomas DeMille <tdemi...@gmail.com> wrote:

> it doesn't seem like a normal type of log entry because it has Exception and
> errno all over it
>
> MessagingPort recv() errno:10054 An existing connection was forcibly closed
> by the remote host. 192.168.100.216:56149
> Mon Dec 27 15:57:23 [conn28] SocketException: 9001 socket exception
> Mon Dec 27 15:57:23 [conn28] end connection 192.168.100.216:56149
>
> So I'm not sure that is normal or it would probably just say something like
> 'connection closed'
>
> But, even if that WAS normal, the driver client is throwing a similar
> exception, as I noted in my pasting in of the client exceptions, and this is
> in turn throwing up error pages on my web site and then eventually freezing
> the site.
>

> So since MongoDB is supposed to support *extremely high *volume, it seems as

Thomas DeMille

unread,

Dec 28, 2010, 8:16:24 AM12/28/10

to mongod...@googlegroups.com

Actually, mine occurs during large 'batch' imports, where I drop all indexes, then iterate a SQL DataReader and insert 10k rows, then create indexes again.

In fact, looking at that code in the link below... I'm doing something very similar.

Whats the next step, should I put a pause or something in the loop to allow mongo breathing room?

Perhaps instead of emptying the collection and then rebuilding it I should update rows if necssary, in more of a SQL way, insert/update or delete as necessary. What I'm doing now is removing all then adding all back in for simplicity sake.

This seems to effect my site more with the 10Gen driver than with the open source driver...

T

Thomas DeMille

unread,

Dec 28, 2010, 8:17:45 AM12/28/10

to mongod...@googlegroups.com

My code inserts one row at a time, not a giant BSON doc but otherwise very similar

On Tue, Dec 28, 2010 at 1:01 AM, Robert Stam <rstam...@gmail.com> wrote:

Nat

unread,

Dec 28, 2010, 9:11:43 AM12/28/10

to mongodb-user

Why do you need to drop and recreate indexes? If you don't do that,
does the problem still persist?

> > mongodb-user...@googlegroups.com<mongodb-user%2Bunsubscribe@google groups.com>
> > .

Thomas DeMille

unread,

Dec 28, 2010, 9:22:27 AM12/28/10

to mongod...@googlegroups.com

I'll try that. In SQL Server, it is sometimes faster to drop the index, do bulk operations, and then recreate....

if I have a number of indexes created on a collection, and I drop all the rows, and then add them all back in again, would I then call ensureindex? Would that be different than dropping and recreating the index?

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

nat....@gmail.com

unread,

Dec 28, 2010, 9:41:58 AM12/28/10

to mongod...@googlegroups.com

If you just drop all the rows but not dropping the collection, you don't need to recreate index again. Dropping all rows is like DELETE FROM TableName

I'm not 100% but dropping index might cause the in progress cursors that use that index to fail

Sent via BlackBerry from SingTel!

From: Thomas DeMille <tdem...@gmail.com>

Sender: mongod...@googlegroups.com

Date: Tue, 28 Dec 2010 09:22:27 -0500

To: <mongod...@googlegroups.com>

ReplyTo: mongod...@googlegroups.com

Subject: Re: [mongodb-user] Re: "An existing connection was forcibly closed by the remote host" I'm seeing many of these errors since switching to the 10Gen supported c# driver

Robert Stam

unread,

Dec 28, 2010, 9:46:00 AM12/28/10

to mongodb-user

Next step: ideally we would find a reproducible scenario that would
provide us some information we can act upon.

Here's my summary of where we stand:

1. The "error" messages on the server side that say "connection was
forcibly closed" are a false alarm. This is just what is printed out
when the client closes a connection.

2. The similar error messages on the client side that say "connection
was forcibly closed" are a concern, because the server should not
spontaneously close a connection. My research (and CSHARP-143) show
that the server can close a connection when it receives bad data, so
what I'd like to find out is whether the driver is sending bad data to
the server. The problem is that if this is the case the bad data would
have been sent slightly earlier, so by the time we get the "connection
was forcibly closed" error we are beyond the error.

3. Make sure safe mode is on, that can help detect problems sooner.

4. If possible provide me with an exception message and stack trace
that provides more information to act on. I know this is difficult, or
we would already have it! If we can also correlate statements on the
client side with error messages in the server log that would provide
helpful clues as well.

On Dec 28, 9:22 am, Thomas DeMille <tdemi...@gmail.com> wrote:
> I'll try that. In SQL Server, it is sometimes faster to drop the index, do
> bulk operations, and then recreate....
>
> if I have a number of indexes created on a collection, and I drop all the
> rows, and then add them all back in again, would I then call ensureindex?
> Would that be different than dropping and recreating the index?
>

> > > > mongodb-user...@googlegroups.com<mongodb-user%2Bunsu...@googlegroups.com>

> > <mongodb-user%2Bunsubscribe@google groups.com>
> > > > .
> > > > For more options, visit this group at
> > > >http://groups.google.com/group/mongodb-user?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongod...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > mongodb-user...@googlegroups.com<mongodb-user%2Bunsu...@googlegroups.com>

Thomas DeMille

unread,

Dec 28, 2010, 9:54:20 AM12/28/10

to mongod...@googlegroups.com

Thanks Robert,

What does SafeMode do?

1. The "error" messages on the server side that say "connection was
forcibly closed" are a false alarm. This is just what is printed out
when the client closes a connection.

OK

2. The similar error messages on the client side that say "connection
was forcibly closed" are a concern, because the server should not
spontaneously close a connection. My research (and CSHARP-143) show
that the server can close a connection when it receives bad data, so
what I'd like to find out is whether the driver is sending bad data to
the server. The problem is that if this is the case the bad data would
have been sent slightly earlier, so by the time we get the "connection
was forcibly closed" error we are beyond the error.

OK, I'll see what I can find in the logs. I assume that I can backtrace the [connxx] number, so if [conn1] disconnects on the server, I can go backwards and see what the last data operation was for [conn1] ?

3. Make sure safe mode is on, that can help detect problems sooner.

Wondering what this is and why it helps. Turned it on yesterday but didn't help with this issue.

4. If possible provide me with an exception message and stack trace
that provides more information to act on. I know this is difficult, or
we would already have it! If we can also correlate statements on the
client side with error messages in the server log that would provide
helpful clues as well.

I'll try. Problem is that this seems to happen under load more than in any reproducible scenario. Recreating table and indexes on more than one collection at a time is currently implicated.

Thomas DeMille

unread,

Dec 28, 2010, 9:56:01 AM12/28/10

to mongod...@googlegroups.com

Interesting thought, maybe that is my problem, IF web site is using the index while I drop it, then we get an error and connection is dropped..

Robert, any inside knowledge on this, how does mongo handle indexes being dropped, does it wait for connections using it to cease, then lock it and drop it? Or could above scenario cause an issue?

Robert Stam

unread,

Dec 28, 2010, 10:06:39 AM12/28/10

to mongodb-user

When SafeMode is on every update operation (like Insert) is
immediately followed by a call to GetLastError to check that the
update operation succeeded. The C# driver packages the Update and
GetLastError messages into a single network packet and sends them both
to the server in a single packet.

I'll ask about how dropping an index is handled, but my guess is:
operations using the index that are already in progress are allowed to
complete, the index is then dropped, and subsequent operations use
execution plans without the index until such time as the index has
been recreated. I don't see why this should cause any errors (other
than timeouts if the absence of the index is causing gross
inefficiencies).

Tracing back from "connection was forcibly closed" on the server won't
do any good because that's just a normal close. Just look for other
error messages in the server log and see what you find.

On Dec 28, 9:56 am, Thomas DeMille <tdemi...@gmail.com> wrote:
> Interesting thought, maybe that is my problem, IF web site is using the
> index while I drop it, then we get an error and connection is dropped..
>
> Robert, any inside knowledge on this, how does mongo handle indexes being
> dropped, does it wait for connections using it to cease, then lock it and
> drop it? Or could above scenario cause an issue?
>

> On Tue, Dec 28, 2010 at 9:41 AM, <nat.lu...@gmail.com> wrote:
> > If you just drop all the rows but not dropping the collection, you don't
> > need to recreate index again. Dropping all rows is like DELETE FROM
> > TableName
>
> > I'm not 100% but dropping index might cause the in progress cursors that
> > use that index to fail
>
> > Sent via BlackBerry from SingTel!

> > ------------------------------
> > *From: * Thomas DeMille <tdemi...@gmail.com>
> > *Sender: * mongod...@googlegroups.com
> > *Date: *Tue, 28 Dec 2010 09:22:27 -0500
> > *To: *<mongod...@googlegroups.com>
> > *ReplyTo: * mongod...@googlegroups.com
> > *Subject: *Re: [mongodb-user] Re: "An existing connection was forcibly

> > closed by the remote host" I'm seeing many of these errors since switching
> > to the 10Gen supported c# driver
>
> > I'll try that. In SQL Server, it is sometimes faster to drop the index, do
> > bulk operations, and then recreate....
>
> > if I have a number of indexes created on a collection, and I drop all the
> > rows, and then add them all back in again, would I then call ensureindex?
> > Would that be different than dropping and recreating the index?
>

> >> > > mongodb-user...@googlegroups.com<mongodb-user%2Bunsu...@googlegroups.com>

> >> <mongodb-user%2Bunsubscribe@google groups.com>
> >> > > .
> >> > > For more options, visit this group at
> >> > >http://groups.google.com/group/mongodb-user?hl=en.
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "mongodb-user" group.
> >> To post to this group, send email to mongod...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> mongodb-user...@googlegroups.com<mongodb-user%2Bunsu...@googlegroups.com>

> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/mongodb-user?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongod...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > mongodb-user...@googlegroups.com<mongodb-user%2Bunsu...@googlegroups.com>

> > .
> > For more options, visit this group at
> >http://groups.google.com/group/mongodb-user?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongod...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > mongodb-user...@googlegroups.com<mongodb-user%2Bunsu...@googlegroups.com>

Thomas DeMille

unread,

Dec 28, 2010, 10:46:10 AM12/28/10

to mongod...@googlegroups.com

OK, here is the problem, queries like above and below causing a memory exception in mongo and shutting down mongo.. Notice the large number of $or params.. which result in a c++ uncaught memory allocation error (see log at end), which results in mongo shutting down and closing all my connections. Asp.Net probably takes a while to recover from so many connections being dropped, or perhaps the new c# driver has a longer timeout than the old one which is why I'm noticing this?

What do I need to do to get the Mongo Server Team looking at this?

Mon Dec 27 19:14:52 [conn1] run command BtSearchCache.$cmd { count: "Board_12", query: { g: { $lte: 200000 }, RN: { $ne: 1 }, CL: { $ne: 1 }, $or: [ { c: "2362" }, { c: "2343" }, { c: "2333" }, { c: "2388" }, { c: "2302" }, { c: "2353" }, { c: "1561" }, { c: "2312" }, { c: "2280" }, { c: "1438" }, { c: "2255" }, { c: "2254" }, { c: "1435" }, { c: "1407" }, { c: "2262" }, { c: "1356" }, { c: "2326" }, { c: "2278" }, { c: "2275" }, { c: "2351" }, { c: "2417" }, { c: "1484" }, { c: "2266" }, { c: "41397" }, { c: "2299" }, { c: "2334" }, { c: "1276" }, { c: "41396" }, { c: "2354" }, { c: "2235" }, { c: "2233" }, { c: "1363" }, { c: "2425" }, { c: "2376" }, { c: "2366" }, { c: "2258" }, { c: "1453" }, { c: "1475" }, { c: "2358" }, { c: "1434" }, { c: "1476" }, { c: "2345" }, { c: "1402" }, { c: "2405" }, { c: "2315" }, { c: "2323" }, { c: "2399" }, { c: "2301" }, { c: "1334" }, { c: "1416" }, { c: "2328" }, { c: "2270" }, { c: "2352" }, { c: "1947" }, { c: "2438" }, { c: "2335" }, { c: "2138" }, { c: "2288" }, { c: "1701" }, { c: "41395" }, { c: "41390" }, { c: "1419" }, { c: "1444" }, { c: "2415" }, { c: "1355" }, { c: "1259" }, { c: "1810" }, { c: "1311" }, { c: "2420" }, { c: "1840" }, { c: "2322" }, { c: "2373" }, { c: "2283" }, { c: "2228" }, { c: "1415" }, { c: "1477" }, { c: "2285" }, { c: "1315" }, { c: "2350" }, { c: "2329" }, { c: "2241" }, { c: "2311" }, { c: "2357" }, { c: "2337" }, { c: "1449" }, { c: "2422" }, { c: "2404" }, { c: "2279" }, { c: "1352" }, { c: "2324" }, { c: "2308" }, { c: "2394" }, { c: "2344" }, { c: "2416" }, { c: "2409" }, { c: "1473" }, { c: "2319" }, { c: "2265" }, { c: "2414" }, { c: "2290" }, { c: "2303" }, { c: "2386" }, { c: "2271" }, { c: "2263" }, { c: "2277" }, { c: "2348" }, { c: "2307" }, { c: "2330" }, { c: "1391" }, { c: "2289" }, { c: "1452" }, { c: "2423" }, { c: "2248" }, { c: "2320" }, { c: "2273" }, { c: "1873" }, { c: "2252" }, { c: "2363" }, { c: "1408" }, { c: "1385" }, { c: "2365" }, { c: "2441" }, { c: "2327" }, { c: "2428" }, { c: "1844" }, { c: "2297" }, { c: "2247" }, { c: "41387" }, { c: "2318" }, { c: "2325" }, { c: "1324" }, { c: "2342" }, { c: "2261" }, { c: "2284" }, { c: "2349" }, { c: "2256" }, { c: "2419" }, { c: "2274" }, { c: "2432" }, { c: "2282" }, { c: "2300" }, { c: "41380" }, { c: "2440" }, { c: "2294" }, { c: "2239" }, { c: "2313" }, { c: "2298" }, { c: "41394" }, { c: "41389" }, { c: "2387" }, { c: "41391" }, { c: "41385" }, { c: "41384" }, { c: "2132" }, { c: "2269" }, { c: "1332" }, { c: "2232" }, { c: "2372" }, { c: "1367" }, { c: "2229" }, { c: "2316" }, { c: "2306" }, { c: "2305" }, { c: "2259" }, { c: "1718" }, { c: "2272" }, { c: "2237" }, { c: "2397" }, { c: "1575" }, { c: "6636" }, { c: "1472" }, { c: "2250" }, { c: "2398" }, { c: "1813" }, { c: "2234" }, { c: "2424" }, { c: "41379" }, { c: "2410" }, { c: "2296" }, { c: "2407" }, { c: "7094" }, { c: "2332" }, { c: "1335" }, { c: "2336" }, { c: "2395" }, { c: "1448" }, { c: "2385" }, { c: "2400" }, { c: "1277" }, { c: "1374" }, { c: "2267" }, { c: "2242" }, { c: "2243" }, { c: "2439" }, { c: "2383" }, { c: "2364" }, { c: "2287" }, { c: "2314" }, { c: "2370" }, { c: "2291" }, { c: "2338" }, { c: "2341" }, { c: "2246" }, { c: "2355" }, { c: "2268" }, { c: "1617" }, { c: "1471" }, { c: "2264" }, { c: "2437" }, { c: "2426" }, { c: "1864" }, { c: "2379" }, { c: "2384" }, { c: "2380" }, { c: "2276" }, { c: "2310" }, { c: "1611" }, { c: "2331" }, { c: "2412" }, { c: "2359" }, { c: "1463" }, { c: "1809" }, { c: "2230" }, { c: "2368" }, { c: "2374" }, { c: "2286" }, { c: "41393" }, { c: "41388" }, { c: "41381" }, { c: "41383" }, { c: "41392" }, { c: "41386" }, { c: "2238" }, { c: "2295" }, { c: "2382" }, { c: "1886" }, { c: "2403" }, { c: "1404" }, { c: "1504" }, { c: "2347" }, { c: "2408" }, { c: "2356" }, { c: "1497" }, { c: "1357" }, { c: "1313" }, { c: "1398" }, { c: "2321" }, { c: "1713" }, { c: "1576" }, { c: "2292" }, { c: "2309" }, { c: "2317" }, { c: "2236" }, { c: "2402" }, { c: "2304" }, { c: "2369" }, { c: "2244" }, { c: "2381" }, { c: "2367" }, { c: "2389" }, { c: "2281" }, { c: "2492" }, { c: "2391" }, { c: "2251" }, { c: "2339" }, { c: "2378" }, { c: "2260" }, { c: "2411" }, { c: "2413" }, { c: "2377" }, { c: "1397" }, { c: "2390" }, { c: "2431" }, { c: "1901" }, { c: "2293" }, { c: "41382" }, { c: "2361" }, { c: "2346" }, { c: "2393" }, { c: "2253" }, { c: "2257" }, { c: "2231" } ] } }

Mon Dec 27 19:15:08 [conn2] Uncaught std::exception: bad allocation, terminating

Mon Dec 27 19:15:08 dbexit:

Mon Dec 27 19:15:08 [conn2] shutdown: going to close listening sockets...

Mon Dec 27 19:15:08 [conn2] closing listening socket: 224

Mon Dec 27 19:15:08 [conn2] closing listening socket: 228

Mon Dec 27 19:15:08 [conn2] closing listening socket: 240

Mon Dec 27 19:15:08 [conn2] closing listening socket: 244

Mon Dec 27 19:15:08 [conn2] shutdown: going to flush oplog...

Mon Dec 27 19:15:08 [conn2] shutdown: going to close sockets...

Mon Dec 27 19:15:08 [conn2] shutdown: waiting for fs preallocator...

Mon Dec 27 19:15:08 [conn2] shutdown: closing all files...

Mon Dec 27 19:15:08 [conn4] MessagingPort recv() errno:10053 An established connection was aborted by the software in your host machine. 192.168.100.187:62638

Mon Dec 27 19:15:08 [conn4] SocketException: 9001 socket exception

Mon Dec 27 19:15:08 [conn4] end connection 192.168.100.187:62638

Mon Dec 27 19:15:08 [conn3] MessagingPort recv() errno:10053 An established connection was aborted by the software in your host machine. 192.168.100.187:62627

Mon Dec 27 19:15:08 [conn3] SocketException: 9001 socket exception

Mon Dec 27 19:15:08 [conn3] end connection 192.168.100.187:62627

Mon Dec 27 19:15:08 closeAllFiles() finished

nat....@gmail.com

unread,

Dec 28, 2010, 10:49:41 AM12/28/10

to mongod...@googlegroups.com

Why don't you use $in operator? The query can be much simpler and faster.

Sent via BlackBerry from SingTel!

From: Thomas DeMille <tdem...@gmail.com>

Sender: mongod...@googlegroups.com

Date: Tue, 28 Dec 2010 10:46:10 -0500

To: <mongod...@googlegroups.com>

ReplyTo: mongod...@googlegroups.com

Subject: Re: [mongodb-user] Re: "An existing connection was forcibly closed bythe remote host" I'm seeing many of these errors since switching to the 10Gensupported c# driver

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

Robert Stam

unread,

Dec 28, 2010, 10:50:27 AM12/28/10

to mongodb-user

You can create a JIRA ticket at:

http://jira.mongodb.org/browse/SERVER

nat....@gmail.com

unread,

Dec 28, 2010, 10:55:28 AM12/28/10

to mongod...@googlegroups.com

It probably would be nice if there is a mechanism for server to propagate the exception/error back to the driver.

@Thomas, when you looked at the log file earlier, why did you notice "bad allocation" error?

Sent via BlackBerry from SingTel!

-----Original Message-----
From: Robert Stam <rstam...@gmail.com>
Sender: mongod...@googlegroups.com

Date: Tue, 28 Dec 2010 07:50:27
To: mongodb-user<mongod...@googlegroups.com>
Reply-To: mongod...@googlegroups.com
Subject: [mongodb-user] Re: "An existing connection was forcibly closed bythe
remote host" I'm seeing many of these errors since switching to the
10Gensupported c# driver

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

Thomas DeMille

unread,

Dec 28, 2010, 11:09:27 AM12/28/10

to mongod...@googlegroups.com

OH... The Samus driver didn't support $OR queries, so we routed those to Sql Server instead of Mongo, which is why this appears to be new because of the driver.. So in actuality it's a mongo server $or issue.

To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

Thomas DeMille

unread,

Dec 28, 2010, 11:11:13 AM12/28/10

to mongod...@googlegroups.com

It's a big log file! Looking at it in notepad++ is not exactly fast, 4 million lines is a lot for a text editor :)

Thomas DeMille

unread,

Dec 28, 2010, 11:19:36 AM12/28/10

to mongod...@googlegroups.com

In the case below $in would work, but in some cases (the query is built in code based on user selections) the query below would contains multiple attribute c=123 OR d = 322 for instance.

nat....@gmail.com

unread,

Dec 28, 2010, 11:26:34 AM12/28/10

to mongod...@googlegroups.com

That's fine as long as u can group query on the same attribute with $in operator to reduce number of clauses. you don't need to totally eliminate $or

Sent via BlackBerry from SingTel!

From: Thomas DeMille <tdem...@gmail.com>

Sender: mongod...@googlegroups.com

Date: Tue, 28 Dec 2010 11:19:36 -0500

To: <mongod...@googlegroups.com>

ReplyTo: mongod...@googlegroups.com

Subject: Re: [mongodb-user] Re: "An existing connection was forcibly closedbythe remote host" I'm seeing many of these errors since switching to the10Gensupported c# driver

Thomas DeMille

unread,

Dec 28, 2010, 11:30:46 AM12/28/10

to mongod...@googlegroups.com

Thanks Nat, got me thinking and I see some places where I'm using OR and could be using IN, so that might be helpful.. I'm going to try that and see if it's the large number of attributes or the $OR query that is the issue in my environment.

Tomas Mirezko

unread,

Jun 19, 2011, 1:53:03 PM6/19/11

to mongod...@googlegroups.com

Hello everyone,

I encountered same exception you are posting about, however it doesn't have to do anything with indexing/reindexing and so:

System.IO.IOException was unhandled

Message=Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.

Source=System

StackTrace:

at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)

at MongoDB.Bson.IO.BsonBuffer.WriteTo(Stream stream) in C:\work\10gen\mongodb\mongo-csharp-driver\Bson\IO\BsonBuffer.cs:line 761

at MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoRequestMessage message, SafeMode safeMode) in C:\work\10gen\mongodb\mongo-csharp-driver\Driver\Internal\MongoConnection.cs:line 377

at MongoDB.Driver.MongoCursorEnumerator`1.GetReply(MongoConnection connection, MongoRequestMessage message) in C:\work\10gen\mongodb\mongo-csharp-driver\Driver\Core\MongoCursorEnumerator.cs:line 262

at MongoDB.Driver.MongoCursorEnumerator`1.GetFirst() in C:\work\10gen\mongodb\mongo-csharp-driver\Driver\Core\MongoCursorEnumerator.cs:line 223

at MongoDB.Driver.MongoCursorEnumerator`1.MoveNext() in C:\work\10gen\mongodb\mongo-csharp-driver\Driver\Core\MongoCursorEnumerator.cs:line 126

at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)

at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

at ConsoleApplication1.Program.Main(String[] args) in C:\Users\Tomáš Mizerák\documents\visual studio 2010\Projects\MongoDB_tests\ConsoleApplication1\Program.cs:line 24

at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)

at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)

at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()

at System.Threading.ThreadHelper.ThreadStart_Context(Object state)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

at System.Threading.ThreadHelper.ThreadStart()

InnerException: System.Net.Sockets.SocketException

Message=An existing connection was forcibly closed by the remote host

Source=System

ErrorCode=10054

NativeErrorCode=10054

StackTrace:

at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)

at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)

InnerException:

Driver is official c# 1.1.0.4184 (from github already compiled, zip)

I have minimized steps to reproduce:

1) init db/coll references and read something

var db = MongoDatabase.Create("mongodb://localhost/test");

var coll = db.GetCollection("coll");

coll.FindAll().ToList();

2) shutdown mongo server (ctrl+c) gracefully

3) start mongo server

4) and read more ...

coll.FindAll().ToList(); <-- this is when the IOException happens

I understand, that connection was closed, but why does client driver throws exception, that something is wrong that at the time of the request, server is running? Shouldn't the driver reconnect?

T.

Robert Stam

unread,

Jun 19, 2011, 3:55:59 PM6/19/11

to mongodb-user

In this scenario the C# driver will throw one IOException, and will
reconnect on the next operation attempted after that.

None of the drivers retry failed operations. The idea is that it is up
to the application to decide:

- whether to retry or not
- how many times to retry
- or for how long to keep retrying

On Jun 19, 1:53 pm, Tomas Mirezko <tomas.mize...@gmail.com> wrote:
> Hello everyone,
>
> I encountered same exception you are posting about, however it doesn't have
> to do anything with indexing/reindexing and so:
>
> System.IO.IOException was unhandled
> Message=Unable to write data to the transport connection: An existing
> connection was forcibly closed by the remote host.
> Source=System
> StackTrace:
> at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32
> offset, Int32 size)
> at MongoDB.Bson.IO.BsonBuffer.WriteTo(Stream stream) in
> C:\work\10gen\mongodb\mongo-csharp-driver\Bson\IO\BsonBuffer.cs:line 761
> at
> MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoRequestMessage
> message, SafeMode safeMode) in

> C:\work\10gen\mongodb\mongo-csharp-driver\Driver\Internal\MongoConnection.c s:line

> 377
> at MongoDB.Driver.MongoCursorEnumerator`1.GetReply(MongoConnection
> connection, MongoRequestMessage message) in

> C:\work\10gen\mongodb\mongo-csharp-driver\Driver\Core\MongoCursorEnumerator .cs:line

> 262
> at MongoDB.Driver.MongoCursorEnumerator`1.GetFirst() in

> C:\work\10gen\mongodb\mongo-csharp-driver\Driver\Core\MongoCursorEnumerator .cs:line

> 223
> at MongoDB.Driver.MongoCursorEnumerator`1.MoveNext() in

> C:\work\10gen\mongodb\mongo-csharp-driver\Driver\Core\MongoCursorEnumerator .cs:line

Tomas Mirezko

unread,

Jun 20, 2011, 4:59:45 AM6/20/11

to mongod...@googlegroups.com

Hi Robert,

that makes sense.

I am developing ASP.NET MVC 3 application where controller uses object wrapping MongoCollection (this object exposes specific methods that controller can use, for example PostCollection.FindLatest, optimized queries for mongodb).

What would be the best practice to wrap this failed/reconnection logic?

Would AOP be recommended? (marking method with custom [RetryQueryAttribute], which would invoke interception and tried to repeat the method when IOException was captured (would be great if wrapped to MongoConnectionException in driver itself). This approach wouldn't pollute the code of the query and would isolate retry policy into custom RetryQueryInterceptor class. I am using Ninject.Extensions.Interception + LinFu)

I am asking this, because I just graduated and I don't have much experience in this kind of enterprise design, any directions (or articles) would be helpful.

Thanks a lot,

T.

Robert Stam

unread,

Jun 20, 2011, 9:59:37 AM6/20/11

to mongodb-user

I'm not familiar with some of the tools you are using, but what you
are suggesting sounds reasonable. It is a challenge to try to not
duplicate your retry logic in many places, although one thing to keep
in mind is that you might sometimes need different retry logic
depending on what operation failed.

Reply all

Reply to author

Forward