Attaching the hotname in probe_success query.

[Yagyansh S. Kumar]

Hi. I am monitoring my Health Check URLs using Blackbox, and I am trying to attach the hostname of the server on which the Health Check is down. My Health Checks are of the form - ServerIP:PORT/my/healthcheck/url (Eg. x.x.x.x:8080/api/a1/healthcheck). I am extracting the ServerIP using regex and then performing group_left on that IP, but it is giving duplicate series error.

Job for Blackbox:

  - job_name: 'Test-Regex'
    metrics_path: /probe
    params:
      module: [http_healthcheck]
    file_sd_configs:
      - files:
        - /etc/blackbox/TestTargets.yml
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - source_labels: [__address__]
        regex: '(.+):(.+)'
        target_label: expo1
        replacement: '${1}:9100'
      - source_labels: [__address__]
        regex: '(.+):(.+)'
        target_label: expo2
        replacement: '${1}:4444'
      - source_labels: [__address__]
        regex: '(.+):(.+)'
        target_label: ip
        replacement: '${1}'
      - target_label: __address__
        replacement: BlackboxServer:Port

Please excuse the messy and stupid looking relabel configs. (I was testing 2-3 things simultaneously).

Target File:

- targets:
     - 172.20.10.138:4596/healthcheck
  labels:
     checkname: 'testingregex'

Alert Expression - (probe_success{job="Test-Regex"}) * on(expo2) group_left(cluster) node_uname_info

Error - Error executing query: found duplicate series for the match group {} on the right hand-side of the operation: [{__name__="node_uname_info", cluster="C1", domainname="(none)", env="PRE-PROD", instance="192.168.8.46:9100", job="node_exporter_A1", machine="x86_64", node="N1", nodename="mymachine-n2.mydomain.in", release="3.10.0-327.22.2.el7.x86_64", source="prometheus", sysname="Linux", version="#1 SMP Thu Jun 23 17:05:11 UTC 2016"}, {__name__="node_uname_info", cluster="C1", domainname="(none)", env="PRE-PROD", instance="192.168.8.45:9100", job="node_exporter_A1", machine="x86_64", node="N1", nodename="mymachine-n1.dailyhunt.in", release="3.10.0-327.22.2.el7.x86_64", source="prometheus", sysname="Linux", version="#1 SMP Thu Jun 23 17:05:11 UTC 2016"}];many-to-many matching not allowed: matching labels must be unique on one side

What I am not understanding is:

1. When I am performing my group_left using expo2 i.e 172.30.10.138:9100 why is it even giving duplicate series error?
2. Why is the error coming for 192.168.8.46:9100(mymachine-n2.mydomain.in) and 192.168.8.45:9100(mymachine-n1.mydomain.in).