Basic auth from environment variable

[Christian Oelsner]

Hello,

I am running prometheus in Kubernetes using prometheus helm chart.

One of my targets require basic authentication in the for of an api key for "username" and an api secret for "password.

I have created them as a kubernetes secret and in the helm values in the env: section i have

- name: BASIC_USER

valueFrom:

secretKeyRef:

name: basic-auth

key: username

- name: BASIC_PW

valueFrom:

secretKeyRef:

name: basic-auth

key: password

When exec in to the pod and doing a printenv i can see that the vars are present, echo'ing them out also gives the expected values.

Now comes my challenge, my scrape job requires authentication, and i am trying to reference the variables like so:

basic_auth:

username: $BASIC_USER

password: $BASIC_PW

But for the life of me, i cant seem to get prometheus to pick it up in the scrape job.

Am i missing the obvious?

If it matters, i am scraping confluent cloud metrics.

Best regards

Christian Oelsner

[Brian Candler]

> But for the life of me, i cant seem to get prometheus to pick it up in the scrape job.

> Am i missing the obvious?

Environment-variable substitution is not performed in prometheus' config file.

You could expose the secret containing the password as a file, rather than an environment variable:

https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod

basic_auth:
  username: some_static_username
  password_file: /path/to/file

The username, though, still has to be included inline.

Since you're deploying via helm, there might be some way the helm chart lets you insert these values into the prometheus configuration.

[Christian Oelsner]

Hello Brian,

thank you for your input. I will give a try and see where that takes me :) 

Best regards

Christian Oelsner

[Christian Oelsner]

Hi again Brian,

That worked nicely for the password :)

Now i just need to figure out a way to something similar with username, as i know that my organization architects will be poiting fingers at me if i commit something to git with a username in it :)

Your help is as usual much appreciated :)

Best

Christian Oelsner

[Christian Oelsner]

Oh,

You mentioned that there might be a way to sort in the helm chart....

I am using the "official" helm chart, and while there might be a way to accomplish it, i have yet to find it.

I will keep digging aroung though :)

/Oelsner

[Brian Candler]

You must be getting that config snippet into prometheus via the helm chart somehow.  If you are just providing the snippet verbatim, then you could do some preprocessing of it. If there's no hook to do this at deployment time via the helm chart, then you could do it at runtime, e.g. via an init container.

Note that prometheus very recently (v2.43.0) added the ability to read additional scrape config files referenced from the main config file:

# Scrape config files specifies a list of globs. Scrape configs are read from # all matching files and appended to the list of scrape configs. scrape_config_files: [ - <filepath_glob> ... ]