AlertManager: how the time in message generated?

[leiwa...@gmail.com]

Configed a simple rule：

groups:

- name: network-delay

  rules:

  - alert: "network delay"

    expr: probe_duration_seconds * 1000 > 3000

    for: 1s

    labels:

      severity: warning

      team: ops

    annotations:

      description: "{{$labels.instance}} : {{ $value }}"

A pushed the alert message wo wechat: 

  [Warning]:
2020-08-10 03:03:53
description = zhaoqing : 1081.122236  

I want to know how the time i  marked as red is generated and what it represents?

It is  about 40 mins earlier than the time when the msg is received.

Thanks,

Lei

[Christian Hoffmann]

Hi,

On 8/10/20 8:51 AM, leiwa...@gmail.com wrote:
> A pushed the alert message wo wechat: 
>
>   [Warning]:
> 2020-08-10 03:03:53
> description = zhaoqing : 1081.122236  
>
> I want to know how the time i  marked as red is generated and what it
> represents?
> It is  about 40 mins earlier than the time when the msg is received.

As far as I know, this should come from the Prometheus side and would be
part of the HTTP request to alertmanager.

There may be some delays between a firing alert in Prometheus and a sent
notification in Alertmanager (e.g. group_wait). Can you show your config?

A difference of 40 minutes sounds like a lot. Not sure, but another wild
guess: it could also be related to a flapping alert?

Kind regards,
Christian

[leiwa...@gmail.com]

alertmanager.yml:

global:

  resolve_timeout: 5m

templates:

- /opt/alertmanager-0.21.0.linux-amd64/wechat.tmpl 

route:

  group_by: ['alertname']

  group_wait: 10s

  group_interval: 10s

  repeat_interval: 2m

  receiver: 'wechat'

receivers:

inhibit_rules:

  - source_match:

      severity: 'critical'

    target_match:

      severity: 'warning'

    equal: ['alertname', 'dev', 'instance']

What a  flapping alert mean?

Thanks,
Lei