One thread per connection

[Grégoire Seux]

Hello !

 

I have noticed that mongo uses a lot of threads and seen in the documentation that :

the server will use one thread per TCP connection

 

Isn’t this quite dangerous for the server ? In high load context (for instance a lot of concurrent access), the server will open a lot of threads and performance will drop.

Your advice is to manager a pool of connection application side but there is also a recommendation to set up a mongoS on each application server.

Obviously, this is not possible to manage to global limitation of concurrent connection across multiple application server. How can it scale ?

 

Would a thread pool be easy  to implement to avoid to create thousands of threads ? Do you plan to fix this ?

 

--

Grégoire

[Scott Hernandez]

There is an open issue to implement multiplex'n of connections and
async network support but there are some expectations in the
driver/wire-protocol which require the thread per connections on the
server for now.

Clients do implement pooling and that is where things work as you
(seem) to want. There is no one (new) connection per operation if that
is what you were thinking.

If you want to talk about details and implementation it is best to
post on the mongodb-dev group:
http://groups.google.com/group/mongodb-dev/

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-dev" group.
> To post to this group, send email to mongo...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-dev...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-dev?hl=en.

[Scott Hernandez]

On Mon, Jun 4, 2012 at 10:05 AM, Scott Hernandez
<scotthe...@gmail.com> wrote:
> There is an open issue to implement multiplex'n of connections and
> async network support but there are some expectations in the
> driver/wire-protocol which require the thread per connections on the
> server for now.
>
> Clients do implement pooling and that is where things work as you
> (seem) to want. There is no one (new) connection per operation if that
> is what you were thinking.

Um, sorry... misread the list, my bad.

[Grégoire Seux]

Thanks for your answer Scott,
I know that clients are pooling connections but this is not enough if you use lots of different clients (hundreds of mongoS for instance).
Do you refer to https://jira.mongodb.org/browse/SERVER-1714 ?
I understand that multiplexing will reduce the probability of increasing the number of opened connection but this is only a defensive fix. This would still be possible that a lof a different clients (thousands) will open connections (therefore a lof of threads), am I wrong ?

Grégoire

[Scott Hernandez]

Yes, and https://jira.mongodb.org/browse/SERVER-929.

One of the underlying problems (now, that it can't be solved with more
changes) is that we don't respond to writes at the wire protocol level
so in order to get the error (getLastError) it is stored with the
thread (socket) and limiting.

The general issue I see is not really the threads issues (although the
stack and OS overhead will get large) but more about the number of
connections. If we can more efficiently use the existing connections
then creating more won't be too big a problem, but both are issues and
we are looking for solutions to both, as you can see in the issues
filed.