Remove labels in receiver Amazon SNS

[Emanuel Gonzalez]

Hi all, I have configured Alertmanager to send alerts to Amazon SNS, everything is working correctly but I see a lot of information in the alert that arrives. I have read the documentation but I have not understood how to resolve it.

The alert sends the following information:

Alerts Firing:
Labels:
 - alertname = HpaMaxedOut
 - app_kubernetes_io_component = metrics
 - app_kubernetes_io_instance = kube-state-metrics
 - app_kubernetes_io_managed_by = Helm
 - app_kubernetes_io_name = kube-state-metrics
 - app_kubernetes_io_part_of = kube-state-metrics
 - app_kubernetes_io_version = 2.7.0
 - cluster = test-dev
 - helm_sh_chart = kube-state-metrics-4.29.0
 - horizontalpodautoscaler = apache-hpa
 - instance = 10.x.x.x:8080
 - job = kubernetes-service-endpoints
 - namespace = test-apache
 - node = ip-10-x-x-x.ec2.internal
 - service = kube-state-metrics
 - severity = critical
Annotations:
 - description = The maximum number of desired Pods has been hit
  VALUE = 2
  CLUSTER = test-dev
 HPA = apache-hpa
 - summary = Kubernetes HPA scale capability (instance 10.x.x.x:8080)
Source: /graph?g0.expr=kube_horizontalpodautoscaler_status_desired_replicas+%3E%3D+kube_horizontalpodautoscaler_spec_max_replicas&g0.tab=1

I would like to put together something simpler without so much

 information, for example.

 - alertname = HpaMaxedOut
VALUE = 2
HPA = apache-hpa
CLUSTER = test-dev
- description = The maximum number of desired Pods has been hit
 - summary = Kubernetes HPA scale capability (instance 10.x.x.x:8080)

Here my config:

    groups:
      - name: HpaMaxedOut
        rules:
        - alert: DeploymentReplicasLimitReached
          expr: kube_horizontalpodautoscaler_status_desired_replicas >= kube_horizontalpodautoscaler_spec_max_replicas
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: Kubernetes HPA scale capability (instance {{ $labels.instance }})
            description: "The maximum number of desired Pods has been hit\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

Any helps??

[Brian Candler]

And what's your alertmanager sns_config ?

You can find the default sns.message.template here: https://github.com/prometheus/alertmanager/blob/v0.25.0/template/default.tmpl#L94-L103

{{ define "sns.default.message" }}{{ .CommonAnnotations.SortedPairs.Values | join " " }}
{{ if gt (len .Alerts.Firing) 0 }}
Alerts Firing:
{{ template "__text_alert_list" .Alerts.Firing }}
{{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}
Alerts Resolved:
{{ template "__text_alert_list" .Alerts.Resolved }}
{{ end }}
{{ end }}

You'll need to override this with a template you write yourself, which contains just the labels and/or annotations of interest.

[SysAdmin EM]

Hello, this is the AlertManager config:

alertmanager_config: |
  route:
    group_wait: 30s
    group_interval: 2m
    repeat_interval: 2m
    receiver: 'amazon-sns'
    routes:
    - receiver: 'amazon-sns'
      matchers:
      - alertname='HpaMaxedOut'
  receivers:
  - name: 'amazon-sns'
    sns_configs:
    - topic_arn: arn:aws:sns:xxxxxx:xxxxx:xxxxx
      sigv4:
        region: 'us-east-1'
        subject: '{{ .CommonAnnotations.summary }}'
        message: 'Value: {{ .CommonLabels.value }}\nCluster Name: {{ .CommonLabels.clustername }}\nService: {{ .CommonLabels.service }}'

Any help to configure it the way I need it

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/33fe26cf-76bb-4643-8e03-3a865c96b540n%40googlegroups.com.

[Brian Candler]

You will need to remove a couple of spaces before subject: and message: so they are not nested under sigv4, but line up with topic_arn.  (I'm surprised it accepts the config with the bad indentation).

Then you will need to change your template to include the text and fields you want to include in the message. You can find the template documentation here:

https://prometheus.io/docs/alerting/latest/notifications/

https://prometheus.io/docs/alerting/latest/notification_examples/

[SysAdmin EM]

I have read the documentation and tried to create a template with this code, but it doesn't show the expected result.

   
{{ define "sns.default.message" }}
    {{- $status := .Status -}}
    {{- if eq $status "firing" -}}
    Alert: {{ .Labels.alertname }}
    Cluster: {{ .Labels.cluster }}
    Value: {{ printf "%.2f" .Annotations.current_value }}
    {{- else if eq $status "resolved" -}}
    OK
    {{- end -}}
    {{- end }}

Is there someone with more experience who can help me?

[Brian Candler]

> it doesn't show the expected result

What result *does* it show, and how does it differ from what you expect?

I suggest you first start by making a template that generates a dummy message, like "Hello". Once you know how to make your template replace the default one, then start adding to it field by field.