Manage connections between client and mongos by using asio

[wolfk...@gmail.com]

Hi official guys.

Currently, connections between client and mongos/mongod is in the model of one-thread-per-connection.

We may have 40k connections connected to one mongos, and 40k threads, each serve a connection. it is unacceptable and hard to say if system trash is related to too many connections.

Is there a plan, or an issue, tracking this problem ?

thanks!

[wolfk...@gmail.com]

Sorry, I mean an issue or plan to use asio to manage client->server connections

在 2017年11月16日星期四 UTC+8下午3:27:59，wolfk...@gmail.com写道：

[Andrew Morrow]

Hi -

This feature is under active development right now.

We have already transitioned the server to use ASIO based networking in the upcoming 3.6 release. However, this is still in a synchronous, thread-per-connection model. The 3.6 release will also include an experimental and unsupported version of an asynchronous ASIO based networking layer which will multiplex client connections over an adaptive pool of worker threads. In this model, if you have 40k connections but only 100 of those connections are concurrently active, the server will only have approximately 100 worker threads. We will be working to make the adaptive model fully supported, and, if possible, the default, during the next release of MongoDB after 3.6. 

Thanks,

Andrew

--
You received this message because you are subscribed to the Google Groups "mongodb-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev+unsubscribe@googlegroups.com.
To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-dev/234fc774-e41f-4d0c-9887-06b891aa41f7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[wolfk...@gmail.com]

Awesome!

If so, I just need to backport 3.6's code into my company's 3.2 branch, my company currently is using 3.2

I was wondering to implement this myself.

Btw, say something unrelated, mongo's community(more precisely, mongo company) has lower acceptance to community's code contribution, compared to mariadb's community.

在 2017年11月16日星期四 UTC+8下午10:49:21，acm写道：

Hi -

This feature is under active development right now.

We have already transitioned the server to use ASIO based networking in the upcoming 3.6 release. However, this is still in a synchronous, thread-per-connection model. The 3.6 release will also include an experimental and unsupported version of an asynchronous ASIO based networking layer which will multiplex client connections over an adaptive pool of worker threads. In this model, if you have 40k connections but only 100 of those connections are concurrently active, the server will only have approximately 100 worker threads. We will be working to make the adaptive model fully supported, and, if possible, the default, during the next release of MongoDB after 3.6. 

Thanks,

Andrew

On Thu, Nov 16, 2017 at 2:57 AM, <wolfk...@gmail.com> wrote:

Sorry, I mean an issue or plan to use asio to manage client->server connections

在 2017年11月16日星期四 UTC+8下午3:27:59，wolfk...@gmail.com写道：

Hi official guys.

Currently, connections between client and mongos/mongod is in the model of one-thread-per-connection.

We may have 40k connections connected to one mongos, and 40k threads, each serve a connection. it is unacceptable and hard to say if system trash is related to too many connections.

Is there a plan, or an issue, tracking this problem ?

thanks!

--
You received this message because you are subscribed to the Google Groups "mongodb-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev...@googlegroups.com.

To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.

[Andrew Morrow]

On Thu, Nov 16, 2017 at 11:11 AM, <wolfk...@gmail.com> wrote:

Awesome!

If so, I just need to backport 3.6's code into my company's 3.2 branch, my company currently is using 3.2

Unfortunately I think you will find doing such a backport to be somewhere between extremely difficult and completely impossible. The ingress networking stack has been entirely restructured in 3.4 and 3.6.

 

I was wondering to implement this myself.

Btw, say something unrelated, mongo's community(more precisely, mongo company) has lower acceptance to community's code contribution, compared to mariadb's community.

I do not follow MariaDB's community closely enough to know whether that statement is true or not, but I'm willing to assume that it is.

From our perspective, we don't see a large volume of pull requests for the server repository that meet our requirements in terms of code style, quality, testing, and alignment with our overall development roadmap. Often, the pull requests are correct in a narrow context, but would have serious adverse consequences in a distributed environment, or would preclude the proper implementation of features already in flight. As a result, we end up declining many of the pull requests that we do get, and I think that often discourages external developers from further involvement. In other cases, we have worked extensively with external contributors to bring their work to a point where it almost could be accepted, only to have them decline to continue to invest in moving the pull request forward.

This situation is unfortunate, and if you have suggestions on how we might change that, I would be very interested to hear them.

Finally, I think there are other MongoDB projects that do have much higher community involvement than the server, particularly the language drivers and tools.

Thanks,

Andrew

 

在 2017年11月16日星期四 UTC+8下午10:49:21，acm写道：

Hi -

This feature is under active development right now.

We have already transitioned the server to use ASIO based networking in the upcoming 3.6 release. However, this is still in a synchronous, thread-per-connection model. The 3.6 release will also include an experimental and unsupported version of an asynchronous ASIO based networking layer which will multiplex client connections over an adaptive pool of worker threads. In this model, if you have 40k connections but only 100 of those connections are concurrently active, the server will only have approximately 100 worker threads. We will be working to make the adaptive model fully supported, and, if possible, the default, during the next release of MongoDB after 3.6. 

Thanks,

Andrew

On Thu, Nov 16, 2017 at 2:57 AM, <wolfk...@gmail.com> wrote:

Sorry, I mean an issue or plan to use asio to manage client->server connections

在 2017年11月16日星期四 UTC+8下午3:27:59，wolfk...@gmail.com写道：

Hi official guys.

Currently, connections between client and mongos/mongod is in the model of one-thread-per-connection.

We may have 40k connections connected to one mongos, and 40k threads, each serve a connection. it is unacceptable and hard to say if system trash is related to too many connections.

Is there a plan, or an issue, tracking this problem ?

thanks!

--
You received this message because you are subscribed to the Google Groups "mongodb-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev...@googlegroups.com.
To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-dev/234fc774-e41f-4d0c-9887-06b891aa41f7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mongodb-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev+unsubscribe@googlegroups.com.

To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-dev/5feabfb0-0dba-4f65-ac97-f5a1d11c8b8b%40googlegroups.com.

[wolfk...@gmail.com]

hi, acm:

Long time since the last discussion.

You mentioned:

"""

In this model, if you have 40k connections but only 100 of those connections are concurrently active, the server will only have approximately 100 worker threads

"""

If we have 40k active connections, you still have to create 40k threads, this is because this plan does not completely split network-io works from cpu works. Only the read-from-client/write-to-client layer is splited.

The whole executing stack still mixes io and cpu job, leaving it to kernal to schedule.

Is it possible to use a complete reactor-model to solve this problem? If so, I think less than 10 threads is needed.

在 2017年11月16日星期四 UTC+8下午10:49:21，acm写道：

Hi -

This feature is under active development right now.

We have already transitioned the server to use ASIO based networking in the upcoming 3.6 release. However, this is still in a synchronous, thread-per-connection model. The 3.6 release will also include an experimental and unsupported version of an asynchronous ASIO based networking layer which will multiplex client connections over an adaptive pool of worker threads. In this model, if you have 40k connections but only 100 of those connections are concurrently active, the server will only have approximately 100 worker threads. We will be working to make the adaptive model fully supported, and, if possible, the default, during the next release of MongoDB after 3.6. 

Thanks,

Andrew

On Thu, Nov 16, 2017 at 2:57 AM, <wolfk...@gmail.com> wrote:

Sorry, I mean an issue or plan to use asio to manage client->server connections

在 2017年11月16日星期四 UTC+8下午3:27:59，wolfk...@gmail.com写道：

Hi official guys.

Currently, connections between client and mongos/mongod is in the model of one-thread-per-connection.

We may have 40k connections connected to one mongos, and 40k threads, each serve a connection. it is unacceptable and hard to say if system trash is related to too many connections.

Is there a plan, or an issue, tracking this problem ?

thanks!

--
You received this message because you are subscribed to the Google Groups "mongodb-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev...@googlegroups.com.

To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.