Deploy Prometheus on a Docker Swarm Cluster specifically on a worker node

[thomas.b...@gmail.com]

Hello,

I want to know if we have to install Prometheus on a Docker Swarm Manager o if it is possible to install it to a Worker one to retrieve Docker Swarm Cluster metrics.

I actually encountered the following issue:

monitoring_prometheus.1.ntwl9vdheus9@gb40000p822c3    | level=error ts=2021-02-04T10:09:38.463Z caller=refresh.go:98 component="discovery manager scrape" discovery=dockerswarm msg="Unable to refresh target groups" err="error while listing swarm services: Error response from daemon: This node is not a swarm manager. Worker nodes can't be used to view or modify cluster state. Please run this command on a manager node or promote the current node to a manager."

Which seems to be an issue that the Worker node cannot scap the metrics to the Manager (make sense).

Install Docker services on a Docker Swarm Manager is not in the Swarm best-practises so is there any possibility with an example or some documentation to use maybe the Docker HTTP other /var/run/docker.sock?

This is my prometheus.yml configuration:

global:

  scrape_interval: 30s

  scrape_timeout: 30s

  evaluation_interval: 5m

  external_labels:

    monitor: testing-monitoring

scrape_configs:

  # Prometheus

  - job_name: prometheus

    static_configs:

    - targets:

      - prometheus:9090

  # Docker Swarm job

  - job_name: 'testing-dockerswarm'

    dockerswarm_sd_configs:

      - host: unix:///var/run/docker.sock

        role: tasks

    relabel_configs:

      - source_labels: [__address__]

        separator: ':'

        regex: '(.*):(8089)'

        target_label: __address__

        replacement: '${1}:8080'

      - source_labels: [__address__]

        regex: '(.+)\:(9100|8080|9307|9308)'

        action: keep

      - source_labels: [__meta_dockerswarm_network_name]

        regex: 'monitoring_.+'

        action: keep

      - source_labels: [__meta_dockerswarm_task_desired_state]

        regex: 'running'

        action: keep

      - source_labels: [__meta_dockerswarm_node_hostname]

        target_label: node_name

      - source_labels: [__meta_dockerswarm_node_id]

        target_label: node_id

Thanks in advance for your time.

Kind Regards,

Thomas