What does Consul kv limit mean for Vault?

[suresh.c...@mulesoft.com]

I understand that the limit on a key's value in Consul is 512KB.

Could someone please provide me an understanding of this in context of Vault?

We recently hit this issue(Value exceeds 524288 byte limit) with about 1500 vault mounts (secret backend type=generic). And here are more details on how we organized data in Vault.

Each mount has a few hundred paths and each path contains secrets. 

For example, 

Mounts: a/b/{m_ID} 

Paths:                /x/{p_ID}

                          /y/{p_ID}

                          /z/{p_ID}

where m_ID = 1 - 1000

and p_ID = 1 -100

and each p_ID contains json data

[Jeff Mitchell]

Hi Suresh,

It's a hard limit that we try to keep in mind and work around (it or a similar value are also a limit for multiple of the other backends that rely on consensus protocols).

We already compress the mount table storage to allow more mounts to be mounted, but we don't currently have any kind of way of splitting the mount table up. We may add such logic in a future release, but we've only heard of this issue once before, and it was solved for that user via the compression we added.

The other reason we haven't tackled this is that other than logically, both from a data storage perspective and ACL perspective, _for the KV mount type_, there is very little difference in Vault between having a single mount with 1500 top-level paths and having 1500 mounts each with a single top-level path.  The only real difference I can think of would be if you were seal wrapping, since that is specified at a mount level.

Best,

Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/7c7f4c6b-0fb5-4ae5-aa9e-816dbe3c9328%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[suresh.c...@mulesoft.com]

Thanks Jeff for the quick response! It is really helpful.

So does that mean if we're planning to store a lot of secret data in Vault, be it in one mount or multiple top-level paths, Consul is not an ideal backend choice? 

If yes, is there a way we can make it work with Consul?

My second option might be to go for a different backend type (like DynamoDB). Should I be aware of any such limits for my use case?

Thanks,

Suresh

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

[Jeff Mitchell]

Hi Suresh,

On Thu, Mar 1, 2018 at 4:36 PM, <suresh.c...@mulesoft.com> wrote:

So does that mean if we're planning to store a lot of secret data in Vault, be it in one mount or multiple top-level paths, Consul is not an ideal backend choice? 

If yes, is there a way we can make it work with Consul?

It should be totally fine to store it in Consul -- the 512kb limit is per key. So if you did it in one (or, fewer than 1500) mounts you should have no problem with the mount table being too big.

Obviously this is subject to limits around Consul itself; your instance sizing would have to be big enough to accommodate its working set. The consul list would be the best place to ask for some guidance around that, if you have an estimate of number of keys and overall size of data.

My second option might be to go for a different backend type (like DynamoDB). Should I be aware of any such limits for my use case?

Basically, HC supports only inmem/file/Consul, so lots of people use DynamoDB but if you run into problems we will mostly not be able to help you.

The other bit is if you want to use replication, it's not supported on anything other than Consul.

Best,

Jeff 

[Suresh]

Ohk Thanks Jeff.

So we ran an experiment with a couple of mounts but lots of data in one mount. And we have syslog audit enabled.

The issue we run into after we have inserted about 10000 secrets is: [ERROR] audit: backend failed to log response: backend=syslog/ error=write unixgram ->/var/run/log: write: message too long

This has been discussed here: https://github.com/hashicorp/vault/issues/3617

Is there a solution or workaround to this? We did try enabling TCP for syslog and increasing the MaxMessageSize without any luck.

Thanks,

Suresh

[Jeff Mitchell]

Hi Suresh,

Unfortunately, as on that ticket, there's not much I can do about it if you can't force your syslog server to accept an increased size message. I doubt it has to do with the number of secrets, my guess is that you have a single secret with a very large number of keys.

You might want to see if you can use the "socket" audit type with your syslog daemon as it shouldn't have the same specification restriction.

Best,

Jeff

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/d5d0369d-5b8f-441c-91ed-9514053b949c%40googlegroups.com.