Combining metric endpoints

[fub...@gmail.com]

Hi!

I have some sort of basic question on how to monitor/scrape metrics from various places. My setup is as follows:

* one server with mailserver, reverse proxy and various docker services

* one server for prometheus (different location, different hoster)

I want to monitor all services on my multi purpose server (with different exporters). I have the node_exporter, the blackbox_exporter and a bunch of own custom exporters from my docker services. All this exporters expose local http endpoints to provide the data. But I don't want to expose this to the internet over http without a authentication mechanism. So I put a reverse proxy in front of it and add basic auth (and tls). This is a snipped from my nginx config:

---

        server_name prometheus.datenhalter.de;

        use_ssl;

        [...]

location /docker {

proxy_pass http://127.0.0.1:9010/metrics;

proxy_redirect off;

proxy_set_header Host $host;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

location /metrics {

proxy_pass http://127.0.0.1:9100/metrics;

proxy_redirect off;

proxy_set_header Host $host;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

        [...]

---

On prometheus, I add it like this:

scrape_configs:

  - job_name: 'prometheus'

    static_configs:

      - targets: ['localhost:9090']

  - job_name: vpsfu-node

    scheme: https

    basic_auth:

      username: ***

      password: ***

    tls_config:

      insecure_skip_verify: true

    static_configs:

      - targets: ['prometheus.datenhalter.de']

    

  - job_name: vpsfu-docker

    scheme: https

    metrics_path: /docker

    basic_auth:

      username: ***

      password: ***

    tls_config:

      insecure_skip_verify: true

    static_configs:

      - targets: ['prometheus.datenhalter.de']

[...]

This works, but my guts says me that this is not the right way to do it.

What is the proper way of doing this?

Is there a way I can combine all metrics endpoints into a single one? Or I am completely wrong and this is the way to go?

  Aaron

[Brian Brazil]

What you're doing is the way to go about this, presuming it's not sane to run the Prometheus right beside what it's monitoring.

 

Is there a way I can combine all metrics endpoints into a single one? Or I am completely wrong and this is the way to go?

That's considered an anti-pattern in Prometheus, see https://www.robustperception.io/one-agent-to-rule-them-all/ 

--

Brian Brazil

www.robustperception.io

[fub...@gmail.com]

Thank you!

That clears up a lot for me.

  Aaron