Newbie Relabeling Question (I don't understand)

[Ben Luchterhand]

I'm using Prometheus to track metrics of several machines on different networks launched through a Consul/Nomad architecture;  Prometheus tries to use the internal IP address (172.xx.xx.xxx) instead of the public IP (3.xxx.xx.xxx). I am attempting to relabel the __address__ label but I'm having trouble understanding how relabeling works... Any help would be greatly appreciated.

    relabel_configs:

      - source_labels: ['__address__']

        regex: '172.xx.xx.xxx'

        action: replace

        target_label: '__address__'

        replacement: '3.xxx.xx.xxx'

[Brian Candler]

Are you using consul_sd_configs to generate your targets?  Show your config.  Are your nodes/services registering their internal or external addresses in consul?

Relabeling is text pattern matching and replacement. It is unlikely to be useful unless you have 1:1 NAT, e.g. the address 172.1.2.3 can be rewritten to 3.1.2.3.  Otherwise you're better off with a static targets list (file_sd_configs) where you just list all the public IPs you want to scrape.

[Ben Luchterhand]

global:
  scrape_interval:     1s
  evaluation_interval: 1s


scrape_configs:
  - job_name: nomad_metrics
    metrics_path: /v1/metrics
    params:
      format: ['prometheus']


    consul_sd_configs:
      - server: 'xxx'
        services: ['nomad', 'nomad-client']


    relabel_configs:
      - source_labels: ['__meta_consul_tags']
        regex: '(.*)http(.*)'
        action: keep


  - job_name: 'webserver'
    consul_sd_configs:
        - server: 'xxx'
          services: ['webserver']


    relabel_configs:
      - source_labels: ['__address__']
        regex: 'xxx'
        action: replace
        target_label: '__address__'
        replacement: 'xxx'


  - job_name: consul_server_metrics
    metrics_path: /v1/agent/metrics
    params:
      format: ['prometheus']


    static_configs:
      - targets: ["xxx", "xxx", "xxx"]

That's my prometheus config; I didn't write most of it so I don't understand most of it... But yeah, consul_sd_configs appears in there.
And yes, Consul registers the services by their internal addresses

[Brian Candler]

> Consul registers the services by their internal addresses

Unless you can get consul to register the service's external addresses, then you can't use consul data to scrape the targets external addresses.