Connection pool issue with c# driver

[Alex Brown]

I initialize my MongoDatabase instance like this:

        public static MongoDatabase Initialize(string connectionString)

        {

            var url = new MongoUrl(connectionString);

            var client = new MongoClient(connectionString);

            var server = client.GetServer();

            var database = server.GetDatabase(url.DatabaseName);

            return database;

        }

Then, in my ioc setup (Autofac)

            builder.Register(c => InitializeDatabase.Initialize()).SingleInstance();

So, I create ONE MongoDatabase per app instance

I then have a "CollectionResolver " class, which takes a MongoDatabase:

    public class CollectionResolver : ICollectionResolver

    {

        private readonly MongoDatabase _mongoDatabase;

        public CollectionResolver(MongoDatabase mongoDatabase)

        {

            _mongoDatabase = mongoDatabase;

        }

        public MongoCollection<T> Resolve<T>()

        {

            return _mongoDatabase.GetCollection<T>(typeof(T).Name.ToLower());

        }

    }

again, this is a singleton 

However, when I load test my app, with several thousand users, using New Relic monitoring, I find i have many hundred open connections to mongo...

Any idea why?

[craiggwilson]

That's certainly troubling. I haven't used New Relic before, so I'd like to know some more specifics on how exactly it has ascertained the number of open connections to mongo.

Some further questions:

What version of the driver?

What type of setup do you have? A 3 member replica set, etc...

What is your connection string? Please replace any sensitive informantion.

[Alex Brown]

Hey Craig,

Essentially we're seeing huge numbers of TCP connections to mongodb when load testing our app (circa 10k concurrent requests)

Eventually, that's causing the mongodb server to stop responding

We're on a 10 node cluster for the web servers on EC2

 

Answers below:

What version of the driver?

 1.8.3.9

What type of setup do you have? A 3 member replica set, etc...

3 member replica set - primary, secondary and arbiter

What is your connection string? Please replace any sensitive informantion.

mongodb://<ip_address>/<database>?replicaSet=rs0

[craiggwilson]

The default maximum connection pool size for the .NET driver is 100 connections per server. So, applying some math, that's ~1000 connections for each mongod.  Each mongod should be able to handle up to 20k connections, so 1k isn't all that much.

Are you seeing more than 1k per mongod?  

Are the front-end servers pushing more than ~200 connections each in total (100 per mongod)?

What read preference are you using?

[Alex Brown]

Yeah, that's what I thought

We mainly see issues around the 1000 request mark (When hitting the server with 1000 requests at the same time)

The query is a very simple "get by id"

Are you seeing more than 1k per mongod?  
Are the front-end servers pushing more than ~200 connections each in total (100 per mongod)?

How can i check for both of these?

 

What read preference are you using?

Default

[Alex Brown]

I got to the bottom of this...

I had ensureIndex being called in the constructor of an object that was instantiated for every request...

With wireshark, i could see an insert op being executed (on indexes colletion) for every request, so was able to track this down!

Sorted now, thanks!