How to use Docker Swarm Service Discovery with Traefik Proxy?

[Mike B.]

Prometheus.io docs describe how to use Docker Swarm service discovery with a label

docker service create \

  --name cadvisor \

  --label prometheus-job=cadvisor \

  ... 

You find the corresponding prometheus.yml at https://prometheus.io/docs/guides/dockerswarm/

This might work with an explicit prometheus exporter container which exposes a single metrics port, but this does not work for a real application like the Traefik Proxy server, which exposes ports 80 and 443, but metrics are available at port 8082.

The challenges I found: 

1. Prometheus automatically discovers two targets (ports 80 and 443)

2. Prometheus does not know about the correct metrics port (8082)

Is it possible to use Docker Swarm Service Discovery with relabeling to reduce the targets to a single one per container and update the port by just adding a second label for the correct port?

docker service create \

  --name traefik \

  --label prometheus-job=traefik \

  --label prometheus-port=8082 \

  ...

[Mike B.]

One challenge solved: setting port by label, overwriting the default port from service discovery

# Set the target port from the `prometheus-port` label if it exists. 

- source_labels: [__address__, __meta_dockerswarm_service_label_prometheus_port] 

  target_label: __address__ 

  regex: '(.*):(\d+);(\d+)' 

  replacement: "${1}:${3}" 

This was actually solved in this group: https://groups.google.com/g/prometheus-users/c/nRo03ei6TW4