Auto resolve stackdriver alerts

510 views
Skip to first unread message

Aditya Prakash

unread,
Aug 16, 2018, 11:25:43 AM8/16/18
to gce-discussion
Hi,

I have set alert on my http(s) lb request_count metrics, when any time series has response_code_class equal to 500 for 1 minute. It is obvious that my metrics doesn't have any data points when no server errors are occuring. As described in fine prints of this page
An incident will also be listed as open if there is no data to indicate that the condition is no longer met.
Which means that once triggered, it remains in open state unless one resolves it manually by selecting `resolve for 5 minutes`. Is there a way I can auto resolve my alerts for no data points? I tried to look into setting up a script which would resolve alerts after a period for me, but stackdriver api doesn't seem to expose `resolve for 5 minutes`. Any of kind help of would be really nice. Thanks.

Mary Koes

unread,
Aug 16, 2018, 9:15:36 PM8/16/18
to Aditya Prakash, gce-discussion, Rory Petty

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/f3a84e9f-5dd1-4d44-ad4a-78495aca2c92%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Aditya Prakash

unread,
Aug 17, 2018, 1:20:16 PM8/17/18
to adbro...@google.com, ahill...@google.com, rpe...@google.com, mk...@google.com, palm...@google.com, gce-dis...@googlegroups.com
If you prefer that make a ticket for feature request on https://issuetracker.google.com, I can do that.
Alerts staying in open state makes us miss alerts when next time condition is met. An api endpoint for `resolve for 5 minutes` ui option would be good enough too.

On Fri, Aug 17, 2018 at 10:12 PM Adam Browning <adbro...@google.com> wrote:
The incidents will automatically resolve after 7 days of no points. There's a product feature flag that controls that length of time, but I don't know what the status of manipulating that flag is with the cutover from AutoMoB to Spanner. For the sake of clarity, the product feature flag isn't currently exposed via an API, so it'd have to be set by the dev team.

On Fri, Aug 17, 2018 at 11:30 AM Amber Hillhouse <ahill...@google.com> wrote:

On Fri, Aug 17, 2018 at 11:25 AM Rory Petty <rpe...@google.com> wrote:

Digil (Google Cloud Platform Support)

unread,
Aug 17, 2018, 5:14:21 PM8/17/18
to gce-dis...@googlegroups.com
Hello Aditya,

If you think its a useful feature, feel free to open one using the issue-tracker along with your use case scenario.

Juliusz Gonera

unread,
Nov 18, 2018, 4:21:49 PM11/18/18
to gce-discussion
Hi Aditya,

Did you create an issue for this? If yes, could you link to it?

Thanks,
Juliusz

Shashank hegde

unread,
Nov 19, 2018, 3:05:28 AM11/19/18
to jgo...@gmail.com, gce-discussion
I haven't created an issue but posted a query in the groups


--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Thanks & Regards,
Shashank Hegde

xchri...@google.com

unread,
Nov 20, 2018, 9:19:02 PM11/20/18
to gce-discussion
Hello, everyone who is interested in this being added as a feature in GCP, I've created a feature request[0] to implement this. Please be sure to star it to let us know you are affected, & also get email updates about the status of its implementation. OP ad...@gmail.com I've made you the reporter of the issue. Please be sure to star it as well.

Aditya Prakash

unread,
Nov 21, 2018, 9:43:56 AM11/21/18
to gce-discussion
Sorry I didn't get back here.
In private email exchange, I was told that beside creating ticket for a feature request, I could ask for reduction on an gcp internal flag, which already auto resolves alerts (defaults to 7 days). I filed my private ticket here https://issuetracker.google.com/issues/116075434, asking to reduce it to 30 min for my project. I didn't actually got my project's threshold reduced, ticket is just pending as feature request. I suppose, one needs enterprise support for such requests.
Reply all
Reply to author
Forward
0 new messages