metadata labels and rule labels

[Aydogan Osman]

Hey! 

Environment: k8s, kube prometheus stack

I’m new to writing Prometheus rules and wanted to reach out to get a better understanding and any guidance on best practices. I’m in the process of implementing custom rules and need to understand the elements that make up a rule namely metadata labels and rule labels. Outcomes where possible and applicable making the process repeatable and less static. 

In the example below, the rule labels $labels is a map; but what is in the map how can I see all the elements to know what's important? and available for example the namespace isn't working. 

apiVersion: monitoring.coreos.com/v1

kind: PrometheusRule

metadata:

  name: awscni

  labels:

    ruleset: custom

    mandatory: "true"

  namespace: monitoring

spec:

  groups:

    - name: rules-awscni

      rules:

        - alert: CNIAvailableIPsLow

          expr: sum(awscni_ip_max) - sum(awscni_assigned_ip_addresses) < 180

          for: 10m

          labels:

            severity: critical

            namespace: "{{ $labels.namespace }}"

          annotations:

            summary: "AWS EKS available IP addresses low"

            description: "The AWS EKS CNI available IP addresses is less than 50.\n  VALUE = {{ $value }}"