Configuration: Should we generally enable reading secrets from files?

[Bjoern Rabenstein]

Hi Prometheans,

Container orchestration platforms like Kubernetes offer secrets
management. K8s provides those secrets directly to the Kubelet, or via
environment variables, or as files in a volume that containers can
mount, see
https://kubernetes.io/docs/concepts/configuration/secret/#overview-of-secrets
for details.

Good arguments have been made why secrets in environment variables are
problematic. In the Prometheus ecosystem, we have mostly converged on
using files in the scenario described here. That works just fine for
the password of HTTP basic auth, the bearer token, TLS certificates,
and probably more. However, there are a bunch of secrets in config
files (in particular for Prometheus itself and for the Alertmanager)
that _must_ be provided in the config file itself. (Search for
`<secret>` in the documentation of a config file to find all secrets.)
If you want to leverage the K8s secrets management for those, you have
to jump through hoops, i.e. set up an init container that creates a
config on the fly before starting the actual Prometheus or
Alertmanager binary.

My inner minister for consistency tells me we should either allow all
secrets to be provided in a file or none. My inner minister for user
experience tells me we can hardly make users jump through those hoops
for the secrets where we currently allow files.

So what do you think about generally providing a `xxx_file: <string>` config
option where we currently just allow `xxx: <secret>`? There are a lot
of those, but maybe it's the way to go?

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

[Frederic Branczyk]

I think all secrets must be readable from files.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20210218144952.GF2747%40jahnn.

[David Leadbeater]

Yes, please.

One implementation that might be nice is if rather than every secret
having to have a matching "secret_file" key alongside it, is if the
secret could be specified as either:

secret: "some-value"

or something like:

secret:
type: file
file: "/etc/my-secrets/..."

Then every bit of code that deals with secrets wouldn't have to deal
with the two fields, which takes priority and all secrets become
consistent if this can be implemented in one place. (I think this
should be possible although not particularly pretty to implement with
go-yaml v2, it would be easier with yaml v3, but the upgrade to that
is kind of blocked on some v3 issues).

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAOs1UmzrpXgFEbh-TG2N%3D%2B8d5teUqrrHp43hjScX9o%3DZdsaA4g%40mail.gmail.com.

[Julien Pivotto]

I am not against this, but I would like two explicit fields each time, like we do for bearer tokens. I am in particular against any kind of yaml trickery to make them fill in one field. 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAP9KPhBwZeU2cgA2riMwjNCHmOgm_hiq5vZVuP2zT7rrMLHbpw%40mail.gmail.com.

[Julien Pivotto]

I would also note that generally we have agreed that it will be possible
to split the prometheus config in multiple file, currently pending a
design document.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20210218144952.GF2747%40jahnn.

--

Julien Pivotto
@roidelapluie

[Björn Rabenstein]

On Thu, Feb 18, 2021 at 10:10 PM Julien Pivotto
<roidel...@prometheus.io> wrote:
>
> I would also note that generally we have agreed that it will be possible
> to split the prometheus config in multiple file, currently pending a
> design document.

I think this would only work nicely if your file with the secret
contains a whole Yaml snippet, not just the password or token or
certificate. However, I assume most organizations use their secrets
management system in a way that it only contains the secret, not Yaml
around it. (That's probably also the way other systems expect it.)

I have filed issues in prometheus/prometheus and
prometheus/alertmanager for this:
https://github.com/prometheus/prometheus/issues/8551
https://github.com/prometheus/alertmanager/issues/2498