Find available labels for an exporter

[Baptiste Mille-Mathias]

Hello,

I'm using blackbox_exporter and I'd like to know the list of builtin labels it provides (but I'm interested for any exporter).

More specifically, I've 2 probes (one for each proxy server I have to monitor) and I'd like to add a label to expose by which proxy the connexion used, so what is the label I should rename.

#### blackbox excerpt ####

  http_proxy_1:                          
    prober: http                          
    http:                                
      preferred_ip_protocol: ip4          
      proxy_url: http://proxy1:8888
      method: GET                        
  http_proxy_2:                          
    prober: http                          
    http:                                
      preferred_ip_protocol: ip4          
      proxy_url: http://proxy2:8888
      method: GET                        

#### prometheus excerpt ####

  - job_name: 'blackbox-proxy'          
    metrics_path: /probe                  
    params:                              
      module:                            
        - http_proxy_1                    
        - http_proxy_2                    
    static_configs:                      
      - targets:                          
        - https://google.com/            
        - https://microsoft.com/          
        - https://apple.com/              
    relabel_configs:                     
      - source_labels: [__address__]      
        target_label: __param_target      
      - source_labels: [__param_target]  
        target_label: instance            
      - target_label: __address__        
        replacement: 127.0.0.1:9115

Thanks

[Brian Candler]

To see the labels which blackbox_exporter adds, just use curl:

curl 'http://127.0.0.1:9115/probe?target=x.x.x.x&module=http_proxy_1'

Remember that prometheus itself adds "job" and "instance" labels.  And of course, you can just do a PromQL query in the web UI to see the full set of labels:

{job="blackbox-proxy"}

However, there's no need to "rename" an existing label, when you can add your own.  At simplest:

    static_configs:                      
      - targets:                          
        - https://google.com/     

        labels:

          foo: bar

      - targets:                          

        - https://microsoft.com/          

        labels:

          foo: baz

      - targets:                          
        - https://apple.com/    

        labels:

          foo: qux

Now, you're actually interested in knowing which module was used.  The config you've written with

    params:                              
      module:                            
        - http_proxy_1                    
        - http_proxy_2    

won't work anyway, as far as I know.  This will set ...&module=http_proxy_1&module=http_proxy_2 but blackbox_exporter will only look at the first one seen (as you can see if you test this with curl).

So you need to scrape each target twice.  One way is to have two jobs, one for module=http_proxy_1 and one for module=http_proxy_2.  This means you can share the same targets file between the jobs.  The metrics will be distinguished by the "job" label, and/or you can set some other label in rewriting rules:

      - target_label: module  
        replacement: http_proxy_1   # or http_proxy_2 in the other job

The other way to do it is to set the "module" within the targets file, and list the targets multiple times.  This is based on my own running config:

  - job_name: blackbox
    file_sd_configs:
      - files:
          - /etc/prometheus/blackbox.d/*.yml
    metrics_path: /probe
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

      - source_labels: [__address__]
        target_label: __param_target

      - source_labels: [module]
        target_label: __param_module
      - target_label: __address__
        replacement: 127.0.0.1:9115  # Blackbox exporter

Then in the targets file:

- labels:
    module: http_proxy_1

  targets:
    - https://google.com/            
    - https://microsoft.com/          
    - https://apple.com/      

- labels:
    module: http_proxy_2

  targets:
    - https://google.com/            
    - https://microsoft.com/          
    - https://apple.com/   

This gives you the maximum flexibility, since with a single scrape job you can scrape whatever blackbox modules you like, just by adding targets.