hello,
I have implemented a poc on an integration of hashicorp vault with quarkus.
it is definitely not ready for a pull request, but this is enough to engage in a conversation if there is some interest.
from a user perspective, its use is as simple as specifying:
quarkus.datasource.vault-password.role=myapprole
quarkus.datasource.vault-password.secret-path=foo
quarkus.datasource.vault-password.secret-key=password
quarkus.datasource.vault-password.url=http://localhost:30200
it is expected that a vault has been provisionned with:
vault kv put secret/foo password=bar
most of the code is in:
https://github.com/vsevel/quarkus/blob/master/extensions/agroal/runtime/src/main/java/io/quarkus/agroal/runtime/vault/VaultPassword.java
a complete test procedure is provided in:
https://github.com/vsevel/quarkus/blob/master/extensions/agroal/deployment/src/test/vault/README.md
In VaultPassword.java's javadoc I provide several points of discussion regarding design and limitations.
it would be beneficial to get the agroal team opinion.
please reach out if there is some interest to go further.
regards,
vincent
Sorry for having dropped the ball.
Sanne, Guillaume, Scott or me should definitely review and give your some feedback.
The one thing that comes to mind is that the username and password should be a extension build step that Agroal or other password hungry extensions should be able to listen to. That way they can optionally get passwords from other buddies.
I don't know this space well enough and whether the notion of vault password role or maybe path is common to the various solutions besides Hashicorp's?
Also is the same vault usually shared for all passwords (db and otherwise). I'd expect the following config approach
quarkus.hashicorp-vault.role=myapprole #assuming the role is unique for the whole app?
quarkus.hashicorp-vault.url=http://localhost:30200
quarkus.datasource.vault-path=foo
quarkus.datasource.vault-key=password # can we default it or force it at password to limit config overhead?
Scott do you think that fares with the way we expose security in Quarkus?
PS: I'm off the internet so can't look at your doc.
--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
Visit this group at https://groups.google.com/group/quarkus-dev.
For more options, visit https://groups.google.com/d/optout.
Hello,
one initial comment is that vault handles many use cases, beside just storing secrets.
for instance in 1.1 they just released the transit secret engine that can be used as an encryption/decryption service.
and secrets are not limited to passwords. even though I would assume that this represents a fair portion of usages, and that is why I targeted that use case in my poc.
from a design perspective, that means that vault might deserve to be its own extension, usable independently from agroal.
for instance it would be nice to have some sort of microprofile config backed by vault, so that I do not have to store my confidential params in clear text. config map vs secret in k8s analogy.
this extension would be then used either directly by the application, or indirectly by other extensions, such as agroal.
the agroal use case is really interesting beside just storing the password in a secure fashion: there is a feature in vault that consists in renewing the db account username/password in a regular manner (eg every hour) => https://www.vaultproject.io/api/secret/databases/index.html
the challenge is that the connection pool has to play along: every now and then it will have to connect to vault, get the new account username/password, expire its own connections with the old db account, and repopulate the pool with new connections.
the only drawback to this approach is that you have to provide vault with a db admin account that can create and remove db applicative accounts, and it is a bit cumbersome to setup. but this goes a long way in securing access to sensitive databases.
even without dynamic db accounts in place, if you want to handle password change by an administrator without restarting the application, you will have to call vault on a regular basis (potentially renewing the client’s token), which means that you cannot just get the password at startup and keep trying to create connections with the same password for the entire lifetime of the application.
note that this is not particularly well handled by spring cloud either: https://github.com/spring-cloud/spring-cloud-vault/issues/85#issuecomment-283417473
the bottom line is that agroal needs to evolve to handle well those use cases.
of course, a very naive approach simply connects to vault at agroal startup, get the password and assume it will not change overtime. add support for the k8s auth method (https://www.vaultproject.io/docs/auth/kubernetes.html) and you basically have what I developed in my poc. I guess it is better than nothing, but we can do much better.
>> I don't know this space well enough and whether the notion of vault password role or maybe path is common to the various solutions besides Hashicorp's?
to be honest I do not know much about alternatives (and I am not sure there are alternatives as advanced as vault). I looked at bitnami sealed secrets, but they take a very narrow approach limited to securing k8s secrets.
in vault’s kv secret engine you have the concept of policy, which is a list of capabilities (eg read, list, write, …) applicable to a path (vault is a hierarchical structure; each leaf is a map of attribute/secret pairs) eg /secret/myapp/db
the role in vault is the glue between one or more users (eg service accounts in k8s) and one or more policies.
the role and path are very specific to vault.
>> Also is the same vault usually shared for all passwords (db and otherwise).
I would assume so.
>> quarkus.hashicorp-vault.role=myapprole #assuming the role is unique for the whole app?
it could be made a conscious choice for simplicity sake, but this is not necessarily the case: in my application I could use a role ‘ad-reader’ and another ‘foo-db-owner’, or just one role ‘foo-app’.
only the url is really a marker for the vault’s identity. the role, path and attributes are more contextual.
we would have to be able to configure vault for the other use cases as well, such as injecting secrets directly into the app.
>> quarkus.datasource.vault-key=password # can we default it or force it at password to limit config overhead?
defaulting it to ‘password’ is reasonable.
once we have an initial design, we could also engage with the hashicorp folks; may be they would be willing to valide some of the assumptions.
I was going to start poc-ing some of the changes we could do in agroal to make it more dynamic, just to fuel the discussion.
and I can dedicate some time on the whole issue. but it would be better to get some guidance (plus some help on the some quarkus specifics) from you guys, to avoid going into the wrong direction.
To unsubscribe from this group and stop receiving emails from it, send an email to quark...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/00028a93-74d8-4f53-8223-6eab505ef2f1%40googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to quark...@googlegroups.com.
Hello
just as a reminder, a typical vault's use case looks like this:
- the client logs in using one of the auth methods (typically in k8s you would send the service account's jwt token, plus the role; https://www.vaultproject.io/api/auth/kubernetes/index.html#login)
- as a result the client gets a client token. this token has an expiry time, whether it can be renewed or not, and might have a max TTL
- using the client token, the client can read secrets from the vault; eg https://www.vaultproject.io/api/secret/kv/kv-v1.html#read-secret
- when the token is set to expire, the client should renew it (if he intends reading secrets again); eg https://www.vaultproject.io/api/auth/token/index.html#renew-a-token-self-
if we try to map this to a pool's behavior, it means that credentials should be refreshed on a regular basis, so that:
- we can update the password if it has changed
- we can renew the client token
- if we reach the max TTL for the client token, then we should re-login to get a new client token
if the db admin changes the password and updates vault with it, this will not break existing connections. but establishing new connections should be done with the new password in a reasonable timeframe.
for dynamic secrets it is slightly different: we do not read the password from the kv secret engine, instead we generate new credentials using the db secret engine:
here is the typical use case
- the application starts and asks vault to generate new credentials: https://www.vaultproject.io/api/secret/databases/index.html#generate-credentials
- vault creates the user in db with an initial duration validity (ie: the lease, eg 1h) and max TTL (eg 1 day), and returns credentials to application
- application connects to db
- before the end of the lease (eg 1h) the application asks vault to extend the duration validity for that user (ie: 1 more hour)
- application keeps using the user, existing connections are not broken
- after many lease renewal, when we reach the max TTL for that user, vault will note renew the lease, and the credentials will expire
- application needs to ask vault to regenerate a new user/password to continue working; I believe this is that particular use case that is not well handled by spring
using the cli, here is how to generate new credentials:
>> vault read database/creds/mydbrole
Key Value
--- -----
lease_id database/creds/mydbrole/8gnSjKWZzshixMi9qwtwjGaC
lease_duration 1h
lease_renewable true
password A1a-2b71OosLY88yhwEM
username v-kubernet-mydb-u-p0BOKkMZWlwQ44ELXkQo-1548692214
and here is how to renew:
>> vault lease revoke database/creds/mydbrole/8gnSjKWZzshixMi9qwtwjGaC
now back to agroal.
not caching the credentials is definitely going to allow us to make progress.
I am going to refactor my code so that I can take advantage of this feature, plus I will make it in such a way that we can separate it from the agroal extension if we wish to at some point.
I am not convinced that playing with the connection max time, or relying on a custom connection validator will be sufficient because the refresh is at the credentials level, not the connection level.
this means I will have to implement my own refresh mechanism at this point (eg spawn a thread). we will always have the opportunity later to get the timer from somewhere else (either a quarkus timer, an agroal task that uses the housekeepingExecutor, or whatever else that can run either periodically-to handle the password changed by the admin- or at specific times when it is time to renew leases).
it would be great if you could ping me when you have made progress on the « non cached credentials » improvement in agroal.
Thanks,
Vincent
Hello Emmanuel,
Thanks for bringing me into this discussion.
Looking into Vincent POC, and some changes recently made to Agroal, I certainly thing Vault support is achievable without too much hassle.
The main hurdle currently, that is rightly mentioned in the POC, Agroal caches the credentials and that will obviously have to change. I will do it ASAP.
Although re-autentication is something not supported directly, recently a 'max lifetime' to the connections was added, that with the correct sizing of the pool can achieve pretty much the same effect.
Even with prior versions is possible to flush the pool programatically. A custom connection validator is also another option.
Another thing that has evolved is the way authentication extensions are handled in Quarkus. Adding Vault would make a good showcase for that.
There is also some open questions though. The biggest one seems to be where this will live: as a module in Agroal or an Extension in Quarkus.
Regards,
LUIS BARREIRO
MIDDLEWARE PERFORMANCE TEAM
> an email to quark...@googlegroups.com.
>> What is your typical deployment of the vault server in your environment in terms of storage and listeners?
listener = tcp (I do not know about any other one)
https://www.vaultproject.io/docs/configuration/listener/index.html
storage = consul
>> Are you just using the open source version of vault?
yes. the official one available through dockerhub
--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
Visit this group at https://groups.google.com/group/quarkus-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/3bf8b778-72e6-4711-9bb4-362b9b6fc7a5%40googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
Visit this group at https://groups.google.com/group/quarkus-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/f9748b38-5a28-41e6-b592-56550a47d625%40googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
Visit this group at https://groups.google.com/group/quarkus-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/aadddfba-4580-41a2-8525-df35773b8b5a%40googlegroups.com.
io.quarkus.vault.service.url
instead of
io.quarkus.vault.VaultProxy/mp-rest/url
given that it is used by VaultService
The other thing which I guess is important to decide upon now is how it will be used by other extensions/or user code needing an access to the vault service.
For example, I would not like to start updating the smallrye-jwt extension to have VaultService injected - it may not be required by some users.
As discussed earlier, I'd only like to update the smallrye-jwt configuration to point to the Vault and get the public key fetched transparently.
I don't know at the moment how technically it should be done, but I guess it is either a matter of shipping a custom smallrye-config ConfigSource (the easiest option ?)
or use the custom URL handler (that would mean the composite URL only would be used instead of for example having 4 properties for the userpass.etc. possibly the hardest option)
The user level annotation (the one you suggested earlier) can be added later if the auto-resolution is sorted out somehow...
What do you think ?
Sergey
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
Visit this group at https://groups.google.com/group/quarkus-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/1b1aba83-80fa-4d0f-94c8-1f58f588ab86%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
Visit this group at https://groups.google.com/group/quarkus-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/615ed76c-55b7-4334-84d6-ee307b679661%40googlegroups.com.
+1Sergey
On Tue, Jul 16, 2019 at 7:35 PM V. Sevel <vvs...@gmail.com> wrote:
Hello Sergey,
You bring up many valid and interesting points.
However, I believe that at this point the quarkus core team should step up and define their vision, so that we do not go off-road and spend our wheels on something that does not fit the broader vision of the product, regarding security and secret mgmt.
People are free to get inspiration from what I have done, or do something else entirely different, as I have probably focused on a narrow use case.
So I will let the subject rest until it gets more attention. And if solicited, I will gladly offer some help.
Regards,
Vincent
--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quark...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/0b9f65e8-d5c8-4ae6-a212-1b93b0573c36%40googlegroups.com.