How many consul client agent can be supported by a data center ?

[hugolee...@gmail.com]

Hi  

Recently I do a test for consul, which is how many consul client agent can be supported by a data center.

The test environment is total 5 pc. Three consul agent make up server and each agent use one pc. The last 2 pc make up client,each pc creates 300 consul agent as client mode by using different ports ,and it also watch a service.

In the test,I find many errors in the output such as "consul.watch: Watch (type: service) errored: Unexpected response code: 500 (rpc error: failed to get conn: dial tcp 10.16.0.33:8300: i/o timeout), retry in 5s".

I doubt that there is something wrong in my test.And I want to know how many consul client agent can be supported by a data center?

Thank you very much. 

[Armon Dadgar]

Hey,

There are several production clusters with ~5K clients running in a single datacenter.

There are many hundreds of deployments that are larger than 300 agents in a datacenter,

so my guess is there is a network configuration or firewall issue at play.

Hope that helps!

Best Regards,

Armon Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/a2da8ad5-02c1-4c72-bd81-7290052b6b20%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ja...@fpcomplete.com]

On Friday, October 23, 2015 at 6:10:50 AM UTC-4, hugolee...@gmail.com wrote:

Hi  

Recently I do a test for consul, which is how many consul client agent can be supported by a data center.

The test environment is total 5 pc. Three consul agent make up server and each agent use one pc. The last 2 pc make up client,each pc creates 300 consul agent as client mode by using different ports ,and it also watch a service.

The vast majority of us deploying consul are not running hundreds of client agents per host. Maybe some people are deploying more than on agent to a single host, and maybe some of us are deploying more than one client to a single host, but there is no way that your test matches typical, real-world use.

 

I doubt that there is something wrong in my test.And I want to know how many consul client agent can be supported by a data center?

Are you planning to run a system with hundreds of agents on a single node? Your test should match, as closely as possible, to what your real-world use will look like, and it helps to understand the limits of your mockup. The number of consul clients a consul cluster can handle depends on a lot of factors, it is not a black-and-white, discrete number. The number, type, and size of the leaders, the type of network link available to the cluster, other network traffic, the number of KV/service changes to the directory, etc, etc, etc.. these will all affect the raw performance. If you run into a limit, understand it is a limit of YOUR system / mock, and you have to look more closely to determine why you hit that limit before you can point at consul and say you have found consul's limit.

Goodluck!

[hugolee...@gmail.com]

Thank you very much! And I want to ask another question:" What's the max QPS of K/V store by consul?"

在 2015年10月24日星期六 UTC+8上午7:12:22，Armon Dadgar写道：

[hugolee...@gmail.com]

Thank you very much.

在 2015年10月24日星期六 UTC+8下午8:56:31，ja...@fpcomplete.com写道：

[Armon Dadgar]

The maximum QPS will very depending on many factors like hardware, networks, key sizes, access patterns, etc. Consul was previously benchmarked (see the repo for the bench folder) doing 10K QPS on pure reads with small keys. Consul 0.6 will be improving read performance substantially as well. That all said, best way to find out is simulate your use case and benchmark.

Best Regards,

Armon Dadgar

Sent from my iPhone

To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/8e60ea09-9a3d-4169-a1f7-5b336ea713da%40googlegroups.com.