Alertmanager issues

[Jack Neely]

Greetings,

I've noticed this a couple times and last time this happened I was sure to upgrade Alertmanager to 0.1.1.  Alertmanager is using %80 - %125 of CPU, very sluggish to recognize silences/changes (slow AJAX responses I believe).  It's dumping the following to the log but not often:

{"level":"error","msg":"Error on notify: context deadline exceeded","source":"notify.go:152","time":"2016-04-28T17:03:03Z"}

{"level":"error","msg":"Notify for 2011 alerts failed: context deadline exceeded","source":"dispatch.go:238","time":"2016-04-28T17:03:03Z"}

{"level":"error","msg":"Error on notify: context deadline exceeded","source":"notify.go:152","time":"2016-04-28T18:11:31Z"}

{"level":"error","msg":"Notify for 2011 alerts failed: context deadline exceeded","source":"dispatch.go:238","time":"2016-04-28T18:11:31Z"}

A restart used to get things back to normal, but the behavior seems to return quickly.  Any clues here?

--

Jack Neely
Operations Engineer
42 Lines, Inc.

[Jack Neely]

Running with debug logging, this seemed to be going through every past alert and marking things as resolved.  I have removed the SQLite DB and started fresh and I'm back to normal behavior.

[Fabian Reinartz]

This is one of the more explainable behaviors where we see issues. There are other ones less clear. 

The original plan was to wait until we've completed a quite elaborate storage layer that allows additional features.

Given the number of issue reports we will now rather work on a replacement that provides enough functionality for the current feature set.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.