Too many scrape configs on prometheus config file

[Ishvar B]

Hi,

I have a prometheus set up which has too many (roughly 50 static configs) and this number would increase as we have lot of vms still be scraped for prometheus. The file is becoming huge , especially when some applications require basic auth. is there any best practice to maintain the config file as with such huge file some times changes would cause accidental errors.

Thanks

Eswar

[Ben Kochie]

Typically people use a configuration management software. For example Ansible is pretty nice.

https://github.com/cloudalchemy/ansible-prometheus

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/65a0249c-c27a-479d-9df0-b68e5102b883%40googlegroups.com.

[Stuart Clark]

The general suggestion is to use a configuration management system such as Ansible to create the configuration file. Equally you can use the different service discovery mechanisms to simplify your configuration too, including the file_sd to pull targets into separate files (which themselves might be build via configuration management tools or scripts that interact with database of record type systems).

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Julien Pivotto]

On 31 Mar 16:54, Stuart Clark wrote:
> The general suggestion is to use a configuration management system such as Ansible to create the configuration file. Equally you can use the different service discovery mechanisms to simplify your configuration too, including the file_sd to pull targets into separate files (which themselves might be build via configuration management tools or scripts that interact with database of record type systems).

file_sd id probably the way to go. When you add new files, you do not
need to reload the config. When you add a new scrape static config, you
have to.

>
> On 31 March 2020 15:59:52 BST, Ishvar B <ishu...@gmail.com> wrote:
> >Hi,
> >
> >I have a prometheus set up which has too many (roughly 50 static
> >configs)
> >and this number would increase as we have lot of vms still be scraped
> >for
> >prometheus. The file is becoming huge , especially when some
> >applications
> >require basic auth. is there any best practice to maintain the config
> >file
> >as with such huge file some times changes would cause accidental
> >errors.
> >
> >Thanks
> >Eswar
> >
> >--

> >You received this message because you are subscribed to the Google
> >Groups "Prometheus Users" group.
> >To unsubscribe from this group and stop receiving emails from it, send
> >an email to prometheus-use...@googlegroups.com.
> >To view this discussion on the web visit
> >https://groups.google.com/d/msgid/prometheus-users/65a0249c-c27a-479d-9df0-b68e5102b883%40googlegroups.com.
>

> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/80F29B6C-5736-4155-8B2A-5988AA2DF855%40Jahingo.com.

--
(o- Julien Pivotto
//\ Open-Source Consultant
V_/_ Inuits - https://www.inuits.eu

[Ishvar B]

Hi,
Thanks.
I have used file_sd for my black box exporter but how to use this for prometheus config yml.can you point to some resource?

Thank
Eswar

[Brian Candler]

Just replace your static_sd section with a file_sd section.

Before:

    job_name: node

    static_configs:

      - targets:

        - foo:9100

        - bar:9100

        - baz:9100

After:

    job_name: node

    file_sd_configs:

      - files:

        - /etc/prometheus/targets.d/node_targets.yml

and the file /etc/prometheus/targets.d/node_targets.yml contains:

- targets:

  - foo:9100

  - bar:9100

  - baz:9100

[Ishvar B]

Hi Brain,

Thanks for you reply. I was able to figure out yesterday. Is there a way to give basic auth details in the yaml file. I have been trying to add basic auth and prometheus is not picking them up. sample config.

- targets: ['**.**.**']

  scheme: "https" 

  basic_auth: 

    username: ""

    password: ""

  labels:

    job: '**'

Thanks

Eswar

[Brian Candler]

The only settings allowed in a file_sd config are "targets" and "labels".

Basic Auth goes in the prometheus.yml file, not the targets file.  If you have groups of targets with different basic auth credentials, this means they'll have to go in separate jobs in prometheus.

Some things you can override by setting special labels, but I don't think you set auth credentials this way.

- targets:

    - foo:9100

    - bar:9100

  labels:

    __metrics_path__: /xxx

    __scheme__: http

- targets:

    - baz:9100

  labels:

    __metrics_path__: /yyy

    __scheme__: https