Probing Endpoints with different authentication strings.

[Yagyansh S. Kumar]

Hi. I have around 10 URLs that I want to monitor. All are under Basic Auth and have different auth credentials. Is it all possible to probe them using a single blackbox module? Or will I have to create 10 different modules, which seems a very tedious task.

Thanks!

[Eswar Rao Bevara]

Hi,

I am little behind you. I am trying to basic auth one end point though even I have more modules that should be monitored. But I am unable to probe with Basic Auth. can you give me you example config of prom and blackbox on how you have done basic auth?

Thanks

Eswar

[Yagyansh S. Kumar]

Hi. To monitor a single endpoint just use basic_auth in your http module.

modules:
  http_healthcheck:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      method: GET
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      tls_config:
        insecure_skip_verify: false
      preferred_ip_protocol: "ip4"
      basic_auth:
        username: username
        password: password

And use this module in for probing your endpoint that you define in prometheus.yml file.

[Brian Candler]

You'll need to create 10 different modules - there's no way to expand parameters into the module settings.

You can use a script to generate the YAML though.

[Eswar Rao Bevara]

strangely this config is not working for me. I still see 401 error. Not sure what I am missing here. Would you post a sample snippet of the prom.yml to scrape these.

[Yagyansh S. Kumar]

And that would mean making 10 different jobs to probe those 10 URLs. Right?

[Brian Candler]

On Tuesday, 17 March 2020 03:33:40 UTC, Yagyansh S. Kumar wrote:

And that would mean making 10 different jobs to probe those 10 URLs. Right?

No: you can set __param_module differently for targets in the same job.

A simple way is to use the file_sd file to add labels to groups of targets (each group can be a single target):

- labels:

    module: mod1

  targets:

    - foo.example.com

- labels:

    module: mod2

  targets:

    - bar.example.com

#... etc

And a simple relabel config:

    relabel_configs:

      - source_labels: [module]

        target_label: __param_module

(Note: you *could* just set the __param_module label directly in the targets file.  However this is dangerous if you ever probe the same host with two different blackbox modules; you need some stored distinguishing label to keep the timeseries separate.  In any case, the label giving the probe module used is useful for history and dashboards)

If the above targets config is still too verbose for you, then you can used something like this:

- targets:

    - foo.example.com;mod1

    - bar.example.com;mod2

with some more sophisticated relabeling (untested):

    relabel_configs:

      # Split "foo;bar" into instance="foo" module="bar"

      - source_labels: [__address__]

        regex: '(.+);(.+)'

        target_label: instance

        replacement: '${1}'

      - source_labels: [__address__]

        regex: '(.+);(.+)'

        target_label: module

        replacement: '${2}'

      # Fallback: bare "foo" becomes instance="foo" module="some_default"

      - source_labels: [__address__]

        regex: '([^;]+)'

        target_label: instance

        replacement: '${1}'

      - source_labels: [__address__]

        regex: '([^;]+)'

        target_label: module

        replacement: 'icmp'

      # copy instance to __param_target and module to __param_module

      - source_labels: [instance]

        target_label: __param_target

      - source_labels: [module]

        target_label: __param_module

      # send scrape to blackbox_exporter

      - target_label: __address__

        replacement: 127.0.0.1:9115

HTH, Brian.

[Yagyansh S. Kumar]

Thanks a lot.

[Eswar Rao Bevara]

Hi Brain,

This worked like a charm and saved couple of hundred lines of code as I have 20 end points to monitor. One quick question, I have scraping metrics for some of my spring boot applications and there are around 30 applications that I am scrapping metrics from. is there a way to use file_sd to reduce the redundancy of the code just like the way it worked for blackbox exporter.

Thanks

Eswar

[Brian Candler]

On Friday, 20 March 2020 08:42:20 UTC, Eswar Rao Bevara wrote:

I have scraping metrics for some of my spring boot applications and there are around 30 applications that I am scrapping metrics from. is there a way to use file_sd to reduce the redundancy of the code just like the way it worked for blackbox exporter.

Without specific information all I can say is "yes, you can do all sorts of things".

You might not even need relabelling, if it's just a question of connecting to multiple exporters on different ports on the same host:

- targets:
    - foo.example.com:9100
    - foo.example.com:9123
    - foo.example.com:9876

However it would be "normal practice" to have a different scrape job for each type of exporter you are talking to.  Remember that multiple scrape jobs can re-use the same file_sd files.

I believe it should also be possible to set __metrics_path__ differently for different targets, although I've never had to do this, e.g.

- labels:

    __metrics_path__: /something
  targets:
    - foo.example.com:9100

- labels:

    __metrics_path__: /otherthing

  targets:

    - foo.example.com:8080
    - foo.example.com:9999

HTH, Brian.