Prometheus integration with MSTeams

[Zhang Zhao]

I was trying to integrate MSTeams with Prometheus AlertManager so that the alert will be able to feed to MSTeams channels. I tried the solution on GitHub as URL below.

https://github.com/prometheus-msteams/prometheus-msteams

Below are the template and testing alert. The issue I ran into was the template was not able to parse my testing alert correctly because of the highlighted 3 fields. What shall I update to remove these 3 fields???

zz@ZhangsodMacBook Prometheus-MSTeams % curl -X POST -d @sample.json http://localhost:2000/testChannel

json: cannot unmarshal string into Go struct field Message.commonLabels of type template.KV%                                                       zz@ZhangsodMacBook Prometheus-MSTeams % curl -X POST -d @sample.json http://localhost:2000/testChannel

json: cannot unmarshal string into Go struct field Message.commonAnnotations of type template.KV%                                                  zz@ZhangsodMacBook Prometheus-MSTeams % curl -X POST -d @sample.json http://localhost:2000/testChannel

json: cannot unmarshal string into Go struct field Message.commonAnnotations of type template.KV%     

Below is my card.tmpl

{{ define "teams.card" }}

{

  "@type": "MessageCard",

  "@context": "http://schema.org/extensions",

  "themeColor": "{{- if eq .Status "resolved" -}}2DC72D

                 {{- else if eq .Status "firing" -}}

                    {{- if eq .CommonLabels.severity "critical" -}}8C1A1A

                    {{- else if eq .CommonLabels.severity "warning" -}}FFA500

                    {{- else -}}808080{{- end -}}

                 {{- else -}}808080{{- end -}}",

  "summary": "Prometheus Alerts",

  "title": "Prometheus Alert ({{ .Status }})",

  "sections": [ {{$externalUrl := .ExternalURL}}

  {{- range $index, $alert := .Alerts }}{{- if $index }},{{- end }}

    { 

      "facts": [

        {{- range $key, $value := $alert.Annotations }}

        {

          "name": "{{ reReplaceAll "_" "\\\\_" $key }}",

          "value": "{{ reReplaceAll "_" "\\\\_" $value }}"

        },

        {{- end -}}

        {{$c := counter}}{{ range $key, $value := $alert.Labels }}{{if call $c}},{{ end }}

        {

          "name": "{{ reReplaceAll "_" "\\\\_" $key }}",

          "value": "{{ reReplaceAll "_" "\\\\_" $value }}"

        }

        {{- end }}

      ],

      "markdown": true

    }

    {{- end }}

  ]

}

{{ end }}

However, there was a parsing issue with the template above. Below is an alert triggered for testing:

{

   "receiver":"prometheus-snow",

   "status":"firing",

   "alerts":[

      {

         "status":"firing",

         "labels":{

            "alertname":"KubeControllerManagerDown",

            "cluster":"espr-aksepme-dev-westus-cluster-01",

            "geo":"us",

            "prometheus":"espr-prometheus-nonprod/prometheus-prometheus-oper-prometheus",

            "region":"westus",

            "severity":"critical"

         },

         "annotations":{

            "message":"KubeControllerManager has disappeared from Prometheus target discovery.",

            "runbook_url":"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecontrollermanagerdown"

         },

         "startsAt":"2020-06-02T06:56:55.479Z",

         "endsAt":"0001-01-01T00:00:00Z",

         "generatorURL":"http://prometheus-prometheus-oper-prometheus.espr-prometheus-nonprod:9090/graph?g0.expr=absent%28up%7Bjob%3D%22kube-controller-manager%22%7D+%3D%3D+1%29&g0.tab=1",

         "fingerprint":"246a26f7e7ce2afc"

      }

   ],

   "groupLabels":"",

   "commonLabels":"",

   "commonAnnotations":"",

   "externalURL":"http://prometheus-prometheus-oper-alertmanager.espr-prometheus-nonprod:9093",

   "version":"4",

   "groupKey":"{}:{alertname=\"KubeControllerManagerDown\"}",

   "groupLabels_alertname":"KubeControllerManagerDown",

   "commonLabels_alertname":"KubeControllerManagerDown",

   "commonLabels_cluster":"espr-aksepme-dev-westus-cluster-01",

   "commonLabels_geo":"us",

   "commonLabels_prometheus":"espr-prometheus-nonprod/prometheus-prometheus-oper-prometheus",

   "commonLabels_region":"westus",

   "commonLabels_severity":"critical",

   "commonAnnotations_message":"KubeControllerManager has disappeared from Prometheus target discovery.",

   "commonAnnotations_runbook_url":"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecontrollermanagerdown"

}