hashicorp vault support

[V. Sevel]

hello,
I have implemented a poc on an integration of hashicorp vault with quarkus.
it is definitely not ready for a pull request, but this is enough to engage in a conversation if there is some interest.

from a user perspective, its use is as simple as specifying:

quarkus.datasource.vault-password.role=myapprole
quarkus.datasource.vault-password.secret-path=foo
quarkus.datasource.vault-password.secret-key=password
quarkus.datasource.vault-password.url=http://localhost:30200

it is expected that a vault has been provisionned with:
vault kv put secret/foo password=bar

most of the code is in:
https://github.com/vsevel/quarkus/blob/master/extensions/agroal/runtime/src/main/java/io/quarkus/agroal/runtime/vault/VaultPassword.java

a complete test procedure is provided in:
https://github.com/vsevel/quarkus/blob/master/extensions/agroal/deployment/src/test/vault/README.md

In VaultPassword.java's javadoc I provide several points of discussion regarding design and limitations.

it would be beneficial to get the agroal team opinion.

please reach out if there is some interest to go further.

regards,
vincent

[Emmanuel Bernard]

Sorry for having dropped the ball.

Sanne, Guillaume, Scott or me should definitely review and give your some feedback.

The one thing that comes to mind is that the username and password should be a extension build step that Agroal or other password hungry extensions should be able to listen to. That way they can optionally get passwords from other buddies.

I don't know this space well enough and whether the notion of vault password role or maybe path is common to the various solutions besides Hashicorp's?
Also is the same vault usually shared for all passwords (db and otherwise). I'd expect the following config approach

quarkus.hashicorp-vault.role=myapprole #assuming the role is unique for the whole app?
quarkus.hashicorp-vault.url=http://localhost:30200

quarkus.datasource.vault-path=foo
quarkus.datasource.vault-key=password # can we default it or force it at password to limit config overhead?

Scott do you think that fares with the way we expose security in Quarkus?

PS: I'm off the internet so can't look at your doc.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
Visit this group at https://groups.google.com/group/quarkus-dev.
For more options, visit https://groups.google.com/d/optout.

[V. Sevel]

Hello,

one initial comment is that vault handles many use cases, beside just storing secrets.

for instance in 1.1 they just released the transit secret engine that can be used as an encryption/decryption service.

and secrets are not limited to passwords. even though I would assume that this represents a fair portion of usages, and that is why I targeted that use case in my poc.

from a design perspective, that means that vault might deserve to be its own extension, usable independently from agroal.

for instance it would be nice to have some sort of microprofile config backed by vault, so that I do not have to store my confidential params in clear text. config map vs secret in k8s analogy.

this extension would be then used either directly by the application, or indirectly by other extensions, such as agroal.

the agroal use case is really interesting beside just storing the password in a secure fashion: there is a feature in vault that consists in renewing the db account username/password in a regular manner (eg every hour) => https://www.vaultproject.io/api/secret/databases/index.html

the challenge is that the connection pool has to play along: every now and then it will have to connect to vault, get the new account username/password, expire its own connections with the old db account, and repopulate the pool with new connections.

the only drawback to this approach is that you have to provide vault with a db admin account that can create and remove db applicative accounts, and it is a bit cumbersome to setup.  but this goes a long way in securing access to sensitive databases. 

even without dynamic db accounts in place, if you want to handle password change by an administrator without restarting the application, you will have to call vault on a regular basis (potentially renewing the client’s token), which means that you cannot just get the password at startup and keep trying to create connections with the same password for the entire lifetime of the application.

note that this is not particularly well handled by spring cloud either: https://github.com/spring-cloud/spring-cloud-vault/issues/85#issuecomment-283417473 

the bottom line is that agroal needs to evolve to handle well those use cases.

of course, a very naive approach simply connects to vault at agroal startup, get the password and assume it will not change overtime. add support for the k8s auth method (https://www.vaultproject.io/docs/auth/kubernetes.html) and you basically have what I developed in my poc. I guess it is better than nothing, but we can do much better.

>> I don't know this space well enough and whether the notion of vault password role or maybe path is common to the various solutions besides Hashicorp's?

to be honest I do not know much about alternatives (and I am not sure there are alternatives as advanced as vault). I looked at bitnami sealed secrets, but they take a very narrow approach limited to securing k8s secrets.

in vault’s kv secret engine you have the concept of policy, which is a list of capabilities (eg read, list, write, …) applicable to a path (vault is a hierarchical structure; each leaf is a map of attribute/secret pairs) eg /secret/myapp/db

the role in vault is the glue between one or more users (eg service accounts in k8s) and one or more policies. 

the role and path are very specific to vault.

>> Also is the same vault usually shared for all passwords (db and otherwise).

I would assume so.

>> quarkus.hashicorp-vault.role=myapprole #assuming the role is unique for the whole app?

it could be made a conscious choice for simplicity sake, but this is not necessarily the case: in my application I could use a role ‘ad-reader’ and another ‘foo-db-owner’, or just one role ‘foo-app’. 

only the url is really a marker for the vault’s identity. the role, path and attributes are more contextual.

we would have to be able to configure vault for the other use cases as well, such as injecting secrets directly into the app.

>> quarkus.datasource.vault-key=password # can we default it or force it at password to limit config overhead?

defaulting it to ‘password’ is reasonable.

once we have an initial design, we could also engage with the hashicorp folks; may be they would be willing to valide some of the assumptions.

I was going to start poc-ing some of the changes we could do in agroal to make it more dynamic, just to fuel the discussion.

and I can dedicate some time on the whole issue. but it would be better to get some guidance (plus some help on the some quarkus specifics) from you guys, to avoid going into the wrong direction.

To unsubscribe from this group and stop receiving emails from it, send an email to quark...@googlegroups.com.

[Emmanuel Bernard]

Hey Luis,

Could you look at the idea Vincent proposes around having the connection pool recycle connections with new credentials ?

Emmanuel 

To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

Visit this group at https://groups.google.com/group/quarkus-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/00028a93-74d8-4f53-8223-6eab505ef2f1%40googlegroups.com.

[Scott Stark]

I'm still putting together a security design revamp document to begin discussing with the various teams; Elytron, Keycloak, Quarkus, MicroProfile, Smallrye, ...

We need to revamp and make consistent the handling of security credentials and related information and right now it is not what I would call production quality. My concern with a vault and security service like Hashicorp is that probably want similar services available in our cloud offerings that build on Keycloak and other existing security layers at least as an abstraction for the comment security extension layers. We really need better handling of secrets across extensions as well.

To unsubscribe from this group and stop receiving emails from it, send an email to quark...@googlegroups.com.

[V. Sevel]

Hello

just as a reminder, a typical vault's use case looks like this:

 - the client logs in using one of the auth methods (typically in k8s you would send the service account's jwt token, plus the role; https://www.vaultproject.io/api/auth/kubernetes/index.html#login) 

 - as a result the client gets a client token. this token has an expiry time, whether it can be renewed or not, and might have a max TTL

 - using the client token, the client can read secrets from the vault; eg https://www.vaultproject.io/api/secret/kv/kv-v1.html#read-secret

 - when the token is set to expire, the client should renew it (if he intends reading secrets again); eg https://www.vaultproject.io/api/auth/token/index.html#renew-a-token-self-

if we try to map this to a pool's behavior, it means that credentials should be refreshed on a regular basis, so that:

 - we can update the password if it has changed

 - we can renew the client token

 - if we reach the max TTL for the client token, then we should re-login to get a new client token

if the db admin changes the password and updates vault with it, this will not break existing connections. but establishing new connections should be done with the new password in a reasonable timeframe.

for dynamic secrets it is slightly different: we do not read the password from the kv secret engine, instead we generate new credentials using the db secret engine: 

here is the typical use case

 - the application starts and asks vault to generate new credentials: https://www.vaultproject.io/api/secret/databases/index.html#generate-credentials

 - vault creates the user in db with an initial duration validity (ie: the lease, eg 1h) and max TTL (eg 1 day), and returns credentials to application

 - application connects to db

 - before the end of the lease (eg 1h) the application asks vault to extend the duration validity for that user (ie: 1 more hour)

 - application keeps using the user, existing connections are not broken

 - after many lease renewal, when we reach the max TTL for that user, vault will note renew the lease, and the credentials will expire

 - application needs to ask vault to regenerate a new user/password to continue working; I believe this is that particular use case that is not well handled by spring

using the cli, here is how to generate new credentials:

>> vault read database/creds/mydbrole

Key                Value

---                -----

lease_id           database/creds/mydbrole/8gnSjKWZzshixMi9qwtwjGaC

lease_duration     1h

lease_renewable    true

password           A1a-2b71OosLY88yhwEM

username           v-kubernet-mydb-u-p0BOKkMZWlwQ44ELXkQo-1548692214

and here is how to renew:

>> vault lease revoke database/creds/mydbrole/8gnSjKWZzshixMi9qwtwjGaC

now back to agroal.

not caching the credentials is definitely going to allow us to make progress.

I am going to refactor my code so that I can take advantage of this feature, plus I will make it in such a way that we can separate it from the agroal extension if we wish to at some point.

I am not convinced that playing with the connection max time, or relying on a custom connection validator will be sufficient because the refresh is at the credentials level, not the connection level.

this means I will have to implement my own refresh mechanism at this point (eg spawn a thread). we will always have the opportunity later to get the timer from somewhere else (either a quarkus timer, an agroal task that uses the housekeepingExecutor, or whatever else that can run either periodically-to handle the password changed by the admin- or at specific times when it is time to renew leases).

it would be great if you could ping me when you have made progress on the « non cached credentials » improvement in agroal. 

Thanks,

Vincent

Hello Emmanuel,

Thanks for bringing me into this discussion.

Looking into Vincent POC, and some changes recently made to Agroal, I certainly thing Vault support is achievable without too much hassle.

The main hurdle currently, that is rightly mentioned in the POC, Agroal caches the credentials and that will obviously have to change. I will do it ASAP.

Although re-autentication is something not supported directly, recently a 'max lifetime' to the connections was added, that with the correct sizing of the pool can achieve pretty much the same effect. 

Even with prior versions is possible to flush the pool programatically. A custom connection validator is also another option.

Another thing that has evolved is the way authentication extensions are handled in Quarkus. Adding Vault would make a good showcase for that.

There is also some open questions though. The biggest one seems to be where this will live: as a module in Agroal or an Extension in Quarkus.

Regards,

LUIS BARREIRO

MIDDLEWARE PERFORMANCE TEAM

[Emmanuel Bernard]

On 8 May 2019, at 21:39, Scott Stark wrote:

> I'm still putting together a security design revamp document to begin
> discussing with the various teams; Elytron, Keycloak, Quarkus,
> MicroProfile, Smallrye, ...
>
> We need to revamp and make consistent the handling of security
> credentials
> and related information and right now it is not what I would call
> production quality. My concern with a vault and security service like
> Hashicorp is that probably want similar services available in our
> cloud
> offerings that build on Keycloak and other existing security layers at
> least as an abstraction for the comment security extension layers. We
> really need better handling of secrets across extensions as well.

Well, Kube and OpenShift do handle secrets differently as its handled by
the orchestrator and injected via env variable or mounting in the
container. So form a Quarkus app, it's just a runtime property.
I'm not quite sure how Kube would handle secret rotation but I suspect
they would go with rolling out a new instance of the pods so that it
reads the new creds at start up. But again I'm not an expert in that
area.

>> email to quark...@googlegroups.com <javascript:>.

>> Visit this group at https://groups.google.com/group/quarkus-dev.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>

> --
> You received this message because you are subscribed to the Google
> Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to quarkus-dev...@googlegroups.com.

> Visit this group at https://groups.google.com/group/quarkus-dev.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/quarkus-dev/80582cc1-9af7-4bd6-85dd-877de9f6117d%40googlegroups.com.

[V. Sevel]

Hello,

I have finished working on a prod level PR for that feature, addressing some of the feedbacks I got out of this discussion.

https://github.com/quarkusio/quarkus/pull/2897

I am hoping you will find value in it.

> an email to quark...@googlegroups.com.

[Scott Stark]

Thanks, I'm going to take a look at it.

What is your typical deployment of the vault server in your environment in terms of storage and listeners?

Are you just using the open source version of vault?

[V. Sevel]

Hi Scott,

>> What is your typical deployment of the vault server in your environment in terms of storage and listeners?

listener = tcp (I do not know about any other one)
https://www.vaultproject.io/docs/configuration/listener/index.html

storage = consul

>> Are you just using the open source version of vault?

yes. the official one available through dockerhub

[Sergey Beryozkin]

Would it be possible to use it from the other Quarkus extensions which may require the secure key storage, for example, smallrte-jwt or may be keycloak extensions ?

Thanks, Sergey

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

Visit this group at https://groups.google.com/group/quarkus-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/3bf8b778-72e6-4711-9bb4-362b9b6fc7a5%40googlegroups.com.

[V. Sevel]

Hello,

I have started POCing a vault extension, that could be used eventually by anybody in general, and agroal in particular.

it is at: https://github.com/vsevel/quarkus/tree/vaultext/extensions/vault

to get started I have tried creating a rest client (VaultProxy) to vault that could be injected directly, and a higher abstraction VaultService that would handle authentication, renewal, ...

VaultService has a runtime config with information such as user and password in userpass auth, k8s secret path, ...

I have struggled getting the cdi part to work.

I have been hit by something similar to https://github.com/quarkusio/quarkus/issues/2579 (classes loaded by the sun.misc.Launcher$AppClassLoader and the RuntimeClassLoader), but could not get the recommended fix to change anything in my case.

I have tried to get some inspiration from the AgroalProcessor and the FlywayProcessor, with no luck.

I am probably missing something obvious, but I have tried many things and could not get it to work.

I have left a bunch of FIXME in the code with the associated error when I uncomment the code.

In summary I cannot get success in injecting or finding from the Arc container either the VaultProxy or the AbstractVaultServiceProducer.

anybody would be kind enough to review the code and suggest some fix?

thanks,

vincent

[Sergey Beryozkin]

Hi Vincent

IMHO it would be an awesome feature to have once various other components would just point to the vault to get the key out, which can be rotated out of band etc, it would be good.

I've been thinking, would it be simpler for this extension to register some URL handler or something else (JNDI like etc), so that for example one say in the configuration:

mp.jwt.verify.publickey.location=vault://mykey?pswd=123?someotherparam=567

?

(I suppose having an option to inject a dedicated client may work for some places too but I'd be happy to just configure similar to above, camel component like style)

Cheers, Sergey

To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

Visit this group at https://groups.google.com/group/quarkus-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/f9748b38-5a28-41e6-b592-56550a47d625%40googlegroups.com.

[V. Sevel]

Hello Sergey,

I agree something like this would be nice, and make it easy to integrate with vault for simple use cases where all you want is retrieve a secret string.

I am thinking this is something that should be provided by the platform (or at least guidance on how to do it). 

we should be able to use this indirection from the config, in a similar way that wildfly did with its own vault with something like this: ${VAULT::VAULT_BLOCK::ATTRIBUTE_NAME::MASKED_STRING} in the standalone.xml configuration.

mp.jwt.verify.publickey.location=${vault:path=secret/mysecrets,key=mysecret}

and also from injection points, such as in:

@Inject

@Vault(path="secret/mysecrets", key="mysecret")

String mySecret;

and the indirection should not limit itself to fetching the value once at startup, otherwise we would not be able to handle lease renewals/recreation.

Vincent

[Sergey Beryozkin]

Hi Vincent

This looks good, I guess there could be a simple option, such as a once only injection of a secret or public key value into a configuration property (probably via the custom Configuration Source) or into a field via a @Vault qualifier (as per your example above) or a slightly more involved one where a user code would use VaultProxy (the one you talked about in a previous email) if more than fetching a key/secret only once is needed...

Cheers, Sergey

To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

Visit this group at https://groups.google.com/group/quarkus-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/aadddfba-4580-41a2-8525-df35773b8b5a%40googlegroups.com.

[Sergey Beryozkin]

The only other suggestion I have is may be it is also worth having a common Key Management API (Quarkus Vault would be the 1st implementation of it) so that the alternative mechanisms could be plugged in without the user code being affected. Having a god for all the cases common API does not always work, so it won't preclude the Vault users going with the VaultProxy if they prefer, but might give some users a piece of mind if they decide to switch to something else :-)

It does not need to be done now but might be worth giving a thought once the Vault PR is ready otherwise.

Cheers, Sergey

[V. Sevel]

Hello,

I ended up reworking the PR.

details can be found here: https://github.com/quarkusio/quarkus/pull/2897#issuecomment-510610428

regards,

Vincent

Le jeudi 20 juin 2019 06:11:44 UTC+2, Scott Stark a écrit :

[Sergey Beryozkin]

Hi Vincent

IMHO it can become part of the Quarkus security innovation story :-)

I'd propose to use:

io.quarkus.vault.service.url

instead of

io.quarkus.vault.VaultProxy/mp-rest/url

given that it is used by VaultService

The other thing which I guess is important to decide upon now is how it will be used by other extensions/or user code needing an access to the vault service.

For example, I would not like to start updating the smallrye-jwt extension to have VaultService injected - it may not be required by some users.

As discussed earlier, I'd only like to update the smallrye-jwt configuration to point to the Vault and get the public key fetched transparently.

I don't know at the moment how technically it should be done, but I guess it is either a matter of shipping a custom smallrye-config ConfigSource (the easiest option ?)

or use the custom URL handler (that would mean the composite URL only would be used instead of for example having 4 properties for the userpass.etc. possibly the hardest option) 

The user level annotation (the one you suggested earlier) can be added later if the auto-resolution is sorted out somehow...

What do you think ?

Sergey 

To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

Visit this group at https://groups.google.com/group/quarkus-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/1b1aba83-80fa-4d0f-94c8-1f58f588ab86%40googlegroups.com.

[Sergey Beryozkin]

I was thinking a bit more about it. There are few extensions in Quarkus where either public or private key/secret material is needed:

- smallrye-jwt (public key, private key is a future possibility)

- keycloak (public key, and possibly a credentials secret ? )

- the WIP oauth2 extension (hope the team will approve this one soon :-), introspection endpoint client secret plus an optional server cert)

- probably other extensions

So in the extreme case, and when it is all running somewhere in the cloud, there would 5+ points in the configuration where the passwords/keys/certs would have to be set, and it seems that Vault would be the perfect central place for storing all sort of key/password materials. Or am I misunderstanding the role the Vault can play ?

Assuming Vault can be such a place, the question is at what point to support the other extensions' transparent link with the Vault, as part of this PR or later (a POC test with the test extension should do). It can be done later but in that case it would have to be built around whatever Vault proxy/service configuration this PR settles upon, which may work well, but it will most likely mean the ConfigSource option would be the only option, so there should be at least some confidence now that the custom ConfigSource option will eventually deliver, assuming the extensions to the Vault link is of interest to Quarkus

Cheers, Sergey   

[V. Sevel]

Hello Sergey,
You bring up many valid and interesting points.
However, I believe that at this point the quarkus core team should step up and define their vision, so that we do not go off-road and spend our wheels on something that does not fit the broader vision of the product, regarding security and secret mgmt.
People are free to get inspiration from what I have done, or do something else entirely different, as I have probably focused on a narrow use case.
So I will let the subject rest until it gets more attention. And if solicited, I will gladly offer some help.
Regards,
Vincent

[Sergey Beryozkin]

+1

Sergey

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
Visit this group at https://groups.google.com/group/quarkus-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/615ed76c-55b7-4334-84d6-ee307b679661%40googlegroups.com.

[V. Sevel]

hello,

I completed the development of a vault config source that allows properties stored in vault to be seen through the MP config.

this allows:

 - the application to access it normally either programmatically or through injection

 - properties to be consumed by quarkus itself or any other extension

 - to lift the requirement to have a dependency on the vault extension for this basic use case

full details in:

https://github.com/quarkusio/quarkus/pull/2897#issuecomment-515522473

vincent

Le mercredi 17 juillet 2019 00:01:06 UTC+3, Sergey Beryozkin a écrit :

+1

Sergey

On Tue, Jul 16, 2019 at 7:35 PM V. Sevel <vvs...@gmail.com> wrote:

Hello Sergey,
You bring up many valid and interesting points.
However, I believe that at this point the quarkus core team should step up and define their vision, so that we do not go off-road and spend our wheels on something that does not fit the broader vision of the product, regarding security and secret mgmt.
People are free to get inspiration from what I have done, or do something else entirely different, as I have probably focused on a narrow use case.
So I will let the subject rest until it gets more attention. And if solicited, I will gladly offer some help.
Regards,
Vincent

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to quark...@googlegroups.com.

[Sergey Beryozkin]

Hi Vincent

I'll try to find some time shortly to experiment with your extension and will comment on your PR. No doubt this extension will be of help. The indirect use of the vault in other extensions (example, in smallrye-jwt) can be added later, I can do it myself, but I'd like to make sure it will be possible to do it on top of the original PR.

Thanks, Sergey

To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/0b9f65e8-d5c8-4ae6-a212-1b93b0573c36%40googlegroups.com.