Federate data with prometheus

[Jacco Steur]

All,

I have a common problem. I have a couple of clusters that I want to monitor with a single prometheus instance that is outside of any of these clusters. A high availabe prometheus server.

To do this, I want to deploy the prometheus operator in my cluster and configure a /federate serviceMonitor endpoint. I can configure that using the following template:

values.yaml

additionalServiceMonitors: []

    - name: "federated-prometheus"

      additionalLabels: {}

      jobLabel: ""

      targetLabels: ""

      selector: {}

      namespaceSelector:     

        any: false

        matchNames: []

      endpoints: []

        - port: ""

        - targetPort: ""

          bearerTokenFile: ""

          interval: 30s

          path: /federation

          scheme: http

          tlsConfig:

            caFile: ""

            certFile: ""

            insecureSkipVerify: false

            keyFile: ""

            serverName: ""

In my HA prometheus instance I can then scrape the /federate uri's of my various kubernetes clusters.

This configurations looks a bit like this: (From the manual)

scrape_configs:

  - job_name: 'federate'

    scrape_interval: 15s

    honor_labels: true

    metrics_path: '/federate'

    params:

      'match[]':

        - '{job="prometheus"}'

        - '{__name__=~"job:.*"}'

    static_configs:

      - targets:

        - 'source-prometheus-1:9090'

        - 'source-prometheus-2:9090'

        - 'source-prometheus-3:9090'

Should this be enough to get this to work? I tried a lot of variations so I start to doubt if I am on the right track.

Thanks in advance.

Jacco.

[Christian Hoffmann]

Hi,

On 5/28/20 5:45 PM, Jacco Steur wrote:
> In my HA prometheus instance I can then scrape the /federate uri's of my
> various kubernetes clusters.
> This configurations looks a bit like this: (From the manual)
>
> scrape_configs:
>   - job_name: 'federate'
>     scrape_interval: 15s
>     honor_labels: true
>     metrics_path: '/federate'
>     params:
>       'match[]':
>         - '{job="prometheus"}'
>         - '{__name__=~"job:.*"}'
>     static_configs:
>       - targets:
>         - 'source-prometheus-1:9090'
>         - 'source-prometheus-2:9090'
>         - 'source-prometheus-3:9090'
>
> Should this be enough to get this to work? I tried a lot of variations
> so I start to doubt if I am on the right track.

I cannot comment on the serviceMonitor part, but the scrape_configs part
looks sane. What issue are you hitting?

Does Prometheus start? Are there any log messages? Do the targets appear
on the Web UI? Are they "up"?

Besides that, it is usually recommended to only federate
aggregated/selected data. Your match entries look like this is the case,
so this seems fine as well.

Kind regards,
Christian