Monitoring HTTP and HTTPS Endpoints

[spir...@gmail.com]

Hi,

We have some apps that have HTTP and HTTPS endpoints.  Can you advise how to monitor them both?  I've created two separate jobs but now I'm getting alerts when Prometheus tries to scrape the non-https apps.

Here's my config:

 - job_name: 'kubernetes-pods'

      kubernetes_sd_configs:

      - role: pod

      relabel_configs:

      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]

        action: keep

        regex: true

      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]

        action: replace

        target_label: __metrics_path__

        regex: (.+)

      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]

        action: replace

        regex: ([^:]+)(?::\d+)?;(\d+)

        replacement: $1:$2

        target_label: __address__

      - action: labelmap

        regex: __meta_kubernetes_pod_label_(.+)

      - source_labels: [__meta_kubernetes_namespace]

        action: replace

        target_label: kubernetes_namespace

      - source_labels: [__meta_kubernetes_pod_name]

        action: replace

        target_label: kubernetes_pod_name

    - job_name: 'kubernetes-pods-https'

      kubernetes_sd_configs:

      - role: pod

      scheme: https

      tls_config:

        insecure_skip_verify: true

      relabel_configs:

      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]

        action: keep

        regex: true

      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]

        action: replace

        target_label: __metrics_path__

        regex: (.+)

      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]

        action: replace

        regex: ([^:]+)(?::\d+)?;(\d+)

        replacement: $1:$2

        target_label: __address__

      - action: labelmap

        regex: __meta_kubernetes_pod_label_(.+)

      - source_labels: [__meta_kubernetes_namespace]

        action: replace

        target_label: kubernetes_namespace

      - source_labels: [__meta_kubernetes_pod_name]

        action: replace

        target_label: kubernetes_pod_name

[Chris Marchbanks]

Hello,

One option I can think of is to add an annotation such as prometheus.io/https=true or prometheus.io/scheme="https" to your Kubernetes pods.

Then in your service discovery configs you could add another rule to each of your jobs to keep/drop the targets appropriately.

Hope this helps,

Chris

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8d3d9117-cc0e-4c1c-b9e0-c7e0a0cebc1b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Chris Marchbanks | Engineer

FreshTracks.io - Intelligent Alerting for Kubernetes and Prometheus

[Stefano Pirrello]

Thanks I will try this.

[spir...@gmail.com]

So it looks like I'm almost there.  If I check the Targets everything is green and I can see the two jobs intended to scrape http and https endpoints.  The only problem now is the HTTPS endpoints are showing up in the Alerts.

Does this make sense?

Latest config:

- job_name: 'kubernetes-pods-http'

      kubernetes_sd_configs:

      - role: pod

      relabel_configs:

      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]

        action: keep

        regex: true

      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]

        action: drop

        regex: https

      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]

        action: keep

        regex: https

      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]

        action: replace

        target_label: __metrics_path__

        regex: (.+)

      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]

        action: replace

        regex: ([^:]+)(?::\d+)?;(\d+)

        replacement: $1:$2

        target_label: __address__

      - action: labelmap

        regex: __meta_kubernetes_pod_label_(.+)

      - source_labels: [__meta_kubernetes_namespace]

        action: replace

        target_label: kubernetes_namespace

      - source_labels: [__meta_kubernetes_pod_name]

        action: replace

        target_label: kubernetes_pod_name

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8d3d9117-cc0e-4c1c-b9e0-c7e0a0cebc1b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Marchbanks]

Interesting, I don't think I have ever seen that before. I will let you know if I think of something to try, but perhaps someone else has seen state and up not match?

Also, I realized you can use the __scheme__ label in your relabel rules to consolidate to one job with something like:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]

  action: replace

  target_label: __scheme__

  regex: (.+)

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8d3d9117-cc0e-4c1c-b9e0-c7e0a0cebc1b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Chris Marchbanks | Engineer

FreshTracks.io - Intelligent Alerting for Kubernetes and Prometheus

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/97d989e0-c9dc-4707-93db-c2f1fb7df84b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[spir...@gmail.com]

Hi Chris,

Your suggestion down below worked.  Thank you!