Is there an example of what prometheusRules should look like in the Helm template?

[dh...@purestorage.com]

Hi everyone,

I'm currently trying to inject custom rules into my Prometheus Operator setup via a custom values.yaml for prometheus operator.   I am attempting to do so via setting "prometheusRules" in values.yaml, since I found that here:

https://github.com/coreos/prometheus-operator/blob/master/helm/kube-prometheus/values.yaml

     # default rules are in templates/general.rules.yaml

     # prometheusRules: {}

Despite many permutations, I can't get this to work.  My last attempt is at the bottom; the previous attempt didn't have a "groups:" layer.

Is there any examples of a working values.yaml file that contains a non-empty prometheusRules for custom rule injection?  Or am I misunderstanding how I should add custom rules with my helm template?

Ideally, I'd like to be able to do this all from yaml files I tell helm about when I do the install.

Thank you!!

Dan.

[dh...@purestorage.com]

Of course I forgot my promised latest-attempt example :)

Here it is:

prometheusRules:

  prometheus.rules: |+

    groups:

    - name: prometheus.rules

      rules:

      - alert: PrometheusConfigReloadFailed

        expr: prometheus_config_last_reload_successful == 0

        for: 10m

        labels:

          severity: warning

        annotations:

          description: Reloading Prometheus' configuration has failed for {{$labels.namespace}}/{{$labels.pod}}

[Daniel Hart]

OK, so I got the config format working ;)  (See below).

My next question is, how do I get helm template variables to process here, since prometheusRules is being sent in as a key - value?

Specifically, how can I make the "prefix" field work here?

prometheusRules:

  SRERules: |-

    groups:

    - name: sre.rules

      rules:

      - alert: NodeMemoryUtilizationDH

        expr: 100 - (avg by (instance) (avg_over_time(node_memory_MemAvailable[5m]) / node_memory_MemTotal) * 100) > 70

        for: 10m

        labels:

          severity: "warning"

        annotations:

          prefix: "{{ .Labels.cluster }}]"

          summary: "Node Memory Utilization Alert"

          description: "High Memory utilization detected for node {{ $labels.instance_id }} tagged as: {{ $labels.instance_name_tag }}, the utilization is currently: {{ $value }}%"

Thanks!

Dan.