Low performance of vault configured with HA backed etcd

[Tomato_]

In test env, we have configured two Vault server with HA backed etcd, and configured it with SSL. Stress testing shows that etcd storage backend can only handle around 200 write requests per second, and the errors tends to increase as the stress testing moves on, the active vault will also turn to standby mode as errors increase.

Here is the stress testing log

1.  If anyone has ever configured vault with etcd in HA mode ? how does your vault performance like ?

2.  I wonder the reason why vault does not perform well when configured with etcd while it can handle almost ten times more requests when configured with Consul ?

Thank you all !

[Matt Button]

Hi Xie,

Please could you give some more details about your benchmarking? Is it running on your local machine, or on some servers? Do you have a script that's hitting specific endpoints?

Matt

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/8767ccbd-180a-467d-9266-a1583054633a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tomato_]

Hi Matt !

    Thanks for your reply!

    Etcd are on remote servers, I did not control which specific endpoints to be hit, I did two stress tests configuring vault with one etcd backend and with five etcd backends, and the results turned to have no great differences.  the picture I gave before is the result when vault was configured with only one etcd backend.

    My test tool is jmeter, here is the configuration and repeated post requests used in stress test.

    Here is my Vault config file

backend "etcd" {

    address = "https://100.88.81.30:4001,https://100.88.80.30:4001,https://100.88.80.11:4001,https://100.88.82.11:4001,https://100.88.78.11:4001"

    path = "kms/"

    username = "kms"

    password = "Vdasdf*^hs"

    tls_ca_file = "/home/test/etcd/ca.pem"

    tls_cert_file = "/home/test/etcd/etcd-5p.pem"

    tls_key_file = "/home/test/etcd/etcd-5p-key.pem"

    advertise_addr ="https://100.88.109.42:8080"

}

listener "tcp" {

  address = "0.0.0.0:8080"

  tls_disable = 0

  tls_cert_file = "/home/test/vault/kms.intra.test.com.pem"

  tls_key_file = "/home/test/vault/kms.intra.test.com-key.pem"

}

default_lease_ttl="87600h"

max_lease_ttl="87600h"

在 2016年8月11日星期四 UTC+8下午11:23:33，Matt Button写道：

Hi Xie,

Please could you give some more details about your benchmarking? Is it running on your local machine, or on some servers? Do you have a script that's hitting specific endpoints?

Matt

On Thu, Aug 11, 2016 at 1:55 PM, Tomato_ <xqing....@gmail.com> wrote:

In test env, we have configured two Vault server with HA backed etcd, and configured it with SSL. Stress testing shows that etcd storage backend can only handle around 200 write requests per second, and the errors tends to increase as the stress testing moves on, the active vault will also turn to standby mode as errors increase.

Here is the stress testing log

1.  If anyone has ever configured vault with etcd in HA mode ? how does your vault performance like ?

2.  I wonder the reason why vault does not perform well when configured with etcd while it can handle almost ten times more requests when configured with Consul ?

Thank you all !

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

[Matt Button]

Hey Xie,

When you say they're running on remote servers, are your vault/etcd/jmeter nodes all running in the same DC, or is vault running locally and etcd running somewhere else?

To be honest I'm not quite sure about this issue, and I don't really know anything about etcd. Given that you're only writing to a single key, I wonder if there's some contention when etcd receives lots of concurrent updates to the same key? It might be worth seeing if varying the secret name has any impact on the results. ¯\_(ツ)_/¯

Matt

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/0a40bd8e-7708-4102-a4f8-04e6d60d350f%40googlegroups.com.