Multiple exporters per machine

[Isaac Graf]

I am new to prometheus and we are looking to to install it on an existing complex production system.

It looks like we will need multiple exporters per machine- e.g. node_exporter, statsd exporter, custom metric exporter, and some machine types will need other exporters (such as redis exporter).

It seems non-optimal to have prometheus scrape  4 different ports on thousands of machines.

Is there a better way to approach this? 

Thanks very much.

[Stuart Clark]

The design of Prometheus is for applications to ideally expose metrics
about their operation. Where applications don't have native Prometheus
or OpenMetrics support exporters are used. These are designed to be
specific to the application being monitored (e.g. MySQL or the Linux
kernel).

As a result you may end up with several exporters running on a single
machine, depending on what applications you have installed that don't
have native metrics support.

In addition the separation over multiple exporters ensures you don't
have a single point of failure causing total metrics loss (at most you'd
just lose the metrics that a specific exporter was handling).

--
Stuart Clark

[Ben Kochie]

In addition to what Stuart Clar said, I want to talk about one specific question:

> It seems non-optimal to have prometheus scrape  4 different ports on thousands of machines

The Prometheus design is highly parallel. It uses lightweight goroutines to manage the scrapes of targets. So it's actually more optimal to have many small targets, vs few large ones. Having many exporters allows Prometheus to scale up the scrape functions and ingest data smoothly. While this seems like it would be a down side, that's mostly a problem for other monitoring systems. For Prometheus, this is working as designed.

Of course, there are limits. Thousands of targets per job is fine, but millions would be too much for a single Prometheus server.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fd3691bc-6015-4e56-b894-d13cccf35ad4o%40googlegroups.com.

[Brian Candler]

On Monday, 20 July 2020 14:17:13 UTC+1, Isaac Graf wrote:

It seems non-optimal to have prometheus scrape  4 different ports on thousands of machines.

Non-optimal in what way? The total number of metrics is the same.  Are you just worried about opening 4 ports in your firewall?

If so, have a look at exporter_exporter.  This is a simple reverse proxy which lets you expose one port, and optionally protect it with TLS and/or bearer token and/or source address.  You can then bind the individual exporters to 127.0.0.1 so they're not accessible from the outside world, and prometheus can scrape the same port multiple times with different ?module=XXX.

In the simplest configuration you'd still have multiple scrape jobs, because you need to set a different params: module: for each one.  However you can combine them into one scrape job by using some rewriting.  Basically you need to set a "module" label on each target, and then copy it to the __param_module label.

- targets:

    - foo

    - bar

  labels:

    module: mod1

- targets:

    - foo

    - bar

  labels:

    module: mod2

and:

    relabel_configs:

      - source_labels: [module]

        target_label: __param_module

      # Optional but recommended: see https://www.robustperception.io/controlling-the-instance-label

      # This keeps the exporter port number out of your targets file and out of the 'instance' label

      - source_labels: [__address__]

        target_label: instance

      - source_labels: [__address__]

        target_label: __address__

        replacement: '$1:9999'

This doesn't affect the number of scrapes though, it's just a question of how you organize your prometheus jobs.