Rancher alertmanager Jason payload editable or not

[Suvendu Nanda]

Hi All , we have rancher based setup . We are receiving webhook payload from alertmanager, but alarms are not generating at webhook due to format issue.

For example we need to remove \ 

Here we need 
severity=warning, instead of severity=\"warning\"}",     

Can we update the payload jason at rancher alertmanager?      

Jason preview ----

 instance=\"10.42.26.5:8080\", job=\"kube-state-metrics\", job_name=\"ttncbfullbackup-full-28242241\", n                      amespace=\"tmobile\", pod=\"rancher-monitoring-kube-state-metrics-5f4b467b9b-8sftb\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheu                      s\", service=\"rancher-monitoring-kube-state-metrics\", severity=\"warning\"}","groupLabels":{"alertname":"KubeJobFailed","condition":"true","container"                      :"kube-state-metrics","endpoint":"http","instance":"10.42.26.5:8080","job":"kube-state-metrics","job_name":"ttncbfullbackup-full-28242241","namespace":"                      tmobile","pod":"rancher-monitoring-kube-state-metrics-5f4b467b9b-8sftb","prometheus":"cattle-monitoring-system/rancher-monitoring- 

[Chris Siebenmann]

> Hi All , we have rancher based setup . We are receiving webhook payload
> from alertmanager, but alarms are not generating at webhook due to format
> issue.
>
> For example we need to remove \
> Here we need

> *severity=warning*, instead of *severity=\"warning\"}",*
> *Can we update the payload jason at rancher alertmanager? *

The short answer is 'no', because what you want is not valid JSON
syntax. Whatever webhook receiver you're using needs to be able to
process JSON, including JSON strings. You can see examples of the
Alertmanager JSON for alerts in, for example:

https://gist.github.com/mobeigi/5a96f326bc06c7d6f283ecb7cb083f2b
https://www.puppeteers.net/blog/testing-alertmanager-webhooks-with-curl/

In JSON alerts, the value of labels are strings, which means that in the
raw JSON they always have quotes around them because this is the JSON
string format.

Since you seem to be using something to transform the JSON to another
format (the raw JSON does not have eg 'severity="warning"'), it seems
likely that this other component is what needs to be fixed and updated.

(The overall JSON format is documented in
https://prometheus.io/docs/alerting/latest/configuration/#webhook_config
but you need to know a certain amount of JSON to understand what eg
'<object>' means here.)

- cks

[Suvendu Nanda]

Thanks a lot, Chris for your valuable and informative feedback.