I've set Prometheus up from scratch several times now using the following configuration files:
I uncomment the section in the prometheus config to use the alertmanager. Whenever I force an alert to go off, they then stay in PENDING mode forever (well, at least several hours in my testing). Trying to figure out the cause, I tried removing the "for:" configuration option in any of the alerts and suddenly everything is working fine. The alert fires and I see it show up in the alertmanager. So no issues of alerts actually going out or communicating with the alert manager.
No matter how low I set the "for:", it sits there PENDING and never firing. I don't recall ever having this issue before, but right now I can only test 2.5.0 and 2.6.0 and it's happening for both. When I have a chance I'll try and see if I can install an older version and see if it's still happening.
I've turned DEBUG logging on and nothing is showing up in the logs. And I mean literally nothing, unless I remove the "for" clause. So not sure where things are getting hung up.
Appreciate any insight on something I may be overlooking.
Keith