Multiple DNS SD config

[Andrew Meyer]

I would like to search for server in my prod and preprod env.  Currently I'm using a single SRV record to do this in prod.  Works great!

But I want to do this in preprod from prod.  Here is my config:

  - job_name: "node"
    metrics_path: /metrics
    dns_sd_configs:
    - names:
      - "metrics"
    relabel_configs:
    - source_labels: ['__meta_dns_name']
      regex:         'metrics\.(.+?)\..+?\.andrew1\.us'
      target_label:  'zone'
      replacement:   '$1'
    - source_labels: ['__meta_dns_name']
      regex:         'metrics\..+?\.(.+?)\.andrew1\.us'
      target_label:  'job'
      replacement:   '$1'
    - source_labels: ['__meta_dns_name']
      regex:         'metrics\.(.+?)\..+?\.andrew2\.us'
      target_label:  'zone'
      replacement:   '$1'
    - source_labels: ['__meta_dns_name']
      regex:         'metrics\..+?\.(.+?)\.andrew2\.us'
      target_label:  'job'
      replacement:   '$1'

Will this work the way I have it?  I do have forwarding set up and I have tested.  It is working.

[Brian Candler]

> Will this work the way I have it?

You should never override the "job" label. The job label tells you which Prometheus scrape job the metric came from. It also helps to ensure that metrics have a unique label set (in case you end up with two jobs scraping the same exporter)

I don't yet understand what you're doing. The DNS query you're doing is just "metrics". Are you relying on DNS search path?  What is the "single SRV record" pointing to - is there only one target that you're scraping?  What is different between the "prod" and "preprod" environments?  What you you mean by "do this in preprod from prod"?

Note that "dns_sd_configs" (plural) can have multiple configs, and likewise "names" can have multiple names. So the following is valid:

    dns_sd_configs:
    - names:
      - metrics.foo.baz.andrew1.us

    - names:
      - metrics.bar.qux.andrew2.us

As is this:

    dns_sd_configs:
    - names:
      - metrics.foo.baz.andrew1.us

      - metrics.bar.qux.andrew2.us

(The first version lets you set different record type, port and refresh interval for each name; the second does not). You should be able to avoid DNS search paths like this.

If your DNS query returns a mixed list of prod and preprod hosts, and you only want to scrape prod *or* preprod, then you can use "keep" or "drop" rules to select which ones you scrape, as long as you have a way to tell the difference.