Delete an alert from an external script

1,202 views
Skip to first unread message

abhijit...@gmail.com

unread,
Oct 16, 2017, 9:16:53 AM10/16/17
to Prometheus Users
We are trying to generate an alert from an external script by calling the alert manager API based on a certain condition. Based on another condition we would like to update the same alert's status to "resolved" using the same script.
While we have been able to generate an alert we would need some help to update the alert's status.
Could anyone point to some resources on this?

jdbj...@gmail.com

unread,
Oct 16, 2017, 9:24:55 AM10/16/17
to abhijit...@gmail.com, Prometheus Users
Use the same API call you doing to create an active alert, but pass the 'endsAt' field so pass it to 'resolved'.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4fda1890-ae5c-41af-9cb4-7c24f5cbe8a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

abhijit...@gmail.com

unread,
Oct 16, 2017, 9:35:57 AM10/16/17
to Prometheus Users
Sure. I will give it a try. Will update what I find.


On Monday, October 16, 2017 at 6:54:55 PM UTC+5:30, Jose Donizetti wrote:
Use the same API call you doing to create an active alert, but pass the 'endsAt' field so pass it to 'resolved'.
On Mon, Oct 16, 2017 at 11:16 AM, <abhijit...@gmail.com> wrote:
We are trying to generate an alert from an external script by calling the alert manager API based on a certain condition. Based on another condition we would like to update the same alert's status to "resolved" using the same script.
While we have been able to generate an alert we would need some help to update the alert's status.
Could anyone point to some resources on this?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

abhijit...@gmail.com

unread,
Oct 16, 2017, 11:03:53 AM10/16/17
to Prometheus Users
Hi Jose,

to generate the alert with the following payload:
payload_prometheus= [{
                "labels": {
                "alertname": "TestAlert",
                "dev": "sda2",
                "instance": "testalert"
                },
                "annotations": {
                "info": "The disk sda2 is running full",
                "summary": "please check the instance example1"
                }
                    }]

Now as per your instructions, I have used the same API but with endsAt field added as given below in the payload:

payload_prometheus= [{
                "labels": {
                "alertname": "TestAlert",
                "dev": "sda2",
                "endsAt": "resolved",
                "instance": "testalert"
                },
                "annotations": {
                "info": "The disk sda2 is running full",
                "summary": "please check the instance example1"
                }
                    }]

But after doing this, I could still see the alert.
So I passed the current unix timestamp to the "endsAt" field.
But that too did not resolve the problem and I am still seeing the alert.
What am I missing here?




On Monday, October 16, 2017 at 6:54:55 PM UTC+5:30, Jose Donizetti wrote:
Use the same API call you doing to create an active alert, but pass the 'endsAt' field so pass it to 'resolved'.
On Mon, Oct 16, 2017 at 11:16 AM, <abhijit...@gmail.com> wrote:
We are trying to generate an alert from an external script by calling the alert manager API based on a certain condition. Based on another condition we would like to update the same alert's status to "resolved" using the same script.
While we have been able to generate an alert we would need some help to update the alert's status.
Could anyone point to some resources on this?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

jdbj...@gmail.com

unread,
Oct 17, 2017, 12:15:51 PM10/17/17
to Abhijit Mitra, Prometheus Users
Hey Abhijit,

The time format for startsAt, and endsAt must be rfc3339. Also, both fields are not labels. It's documented here https://prometheus.io/docs/alerting/clients/#sending-alerts

Also, I just noticed that you must send both startsAt, and endsAt to resolve the alert. Eg:

alerts1='[
  {
    "labels": {
       "alertname": "DiskRunningFull",
       "dev": "sda1",
       "instance": "example1"
     },
     "annotations": {
        "info": "The disk sda1 is running full",
        "summary": "please check the instance example1"
      },
      "startsAt": "2017-10-17T14:00:11-02:00",
      "endsAt": "2017-10-17T14:06:11-02:00"
  }
]'
curl -XPOST -d"$alerts1" http://localhost:9093/api/v1/alerts


To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d0c1ac89-b631-4377-811d-ba80fa8ec372%40googlegroups.com.

jdbj...@gmail.com

unread,
Oct 18, 2017, 9:23:45 AM10/18/17
to Abhijit Mitra, Prometheus Users
Both amtool and the ui will call the same API for alerts. So you should see the same info in both. Do you have a curl example of the way you are creating and resolving the alert?


On Wed, Oct 18, 2017 at 5:00 AM, Abhijit Mitra <abhijit...@gmail.com> wrote:
Hi,

Thanks a lot for the solution.
As per your suggestion I had passed the startsAt and endsAt in rfc3339 format. 
Also I had passed them as separate fields and not within labels.

This worked perfectly well. I could verify the alerts were removed using amtool.
But I can still see the alerts in the UI. 
Are there any ways to get rid of them from the UI as well?

Thanks again for your solution.

abhijit...@gmail.com

unread,
Oct 23, 2017, 2:26:38 AM10/23/17
to Prometheus Users
Hi,

I am using this to create an alert:

            payload_prometheus= [{
                    "labels": {
                        "alertname": "MyAlert",
                        "dev": "sda1",
                        "instance": "example4"
                    },
                    "annotations": {
                        "info": "The disk sda1 is running full",
                        "summary": alertStr
                    }
                }]
            
r = requests.post(url_prometheus, data=json.dumps(payload_prometheus))

And this to resolve an alert:

            payload_prometheus= [{
                    "labels": {
                        "alertname": "MyAlert",
                        "dev": "sda1",
                        "instance": "example4"
                    },
                    "annotations": {
                        "info": "The disk sda1 is running full",
                        "summary": alertStr
                    }, 
                    "startsAt": "2017-10-17T14:00:11-02:00",
                    "endsAt": "2017-10-17T14:06:11-02:00"
                }]
            
r = requests.post(url_prometheus, data=json.dumps(payload_prometheus))

In the UI the expired alerts are still showing while through amtool if I check I do not see the alert.
But when I restart alertmanager the expired alerts does not show up anymore.
What could be the issue here?

jdbj...@gmail.com

unread,
Oct 23, 2017, 6:54:35 PM10/23/17
to Abhijit Mitra, Prometheus Users
Hey, 

I've tried both scripts:

To create:

alertStr="summary"
payload_prometheus= [{
        "labels": {
            "alertname": "MyAlert",
            "dev": "sda1",
            "instance": "example4"
        },
        "annotations": {
            "info": "The disk sda1 is running full",
            "summary": alertStr
        },
        "startsAt": rfc3339.rfc3339(datetime.now()),
    }]

r = requests.post(url_prometheus, data=json.dumps(payload_prometheus))

To resolve:

alertStr="summary"
payload_prometheus= [{
        "labels": {
            "alertname": "MyAlert",
            "dev": "sda1",
            "instance": "example4"
        },
        "annotations": {
            "info": "The disk sda1 is running full",
            "summary": alertStr
        },
        "startsAt": rfc3339.rfc3339(datetime.now()),
        "endsAt": rfc3339.rfc3339(datetime.now())
    }]
r = requests.post(url_prometheus, data=json.dumps(payload_prometheus))

They work fine. Make sure you running the last version of alertmanager.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/143649ac-1407-47a7-bb99-ddd65923120f%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages