unit test passes with and without alert

[rbauduin]

Hi,

I am writing unit tests for prometheus alert as described at https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/

I have an alert defined to trigger after 5 minutes of an expression being <1 (See below for its definition). I am defining the series values as 1+0x4 0+0x12 1+0x3 with an interval of 1 minute, meaning that from minute 5, we have values under the alert's detection value. So from time 10m an alert should be required in the test, howerer, I can write two test, one with the alert

      - alertname: NoServer
        eval_time: 10m
        exp_alerts:
          - exp_annotations:
              for: 5m
              message: There is no server.
            exp_labels:
              severity: error

and one without alert:

      - eval_time: 10m
        exp_alerts: []

and both pass successfully!

I have this behaviour with promtool installed with go get, and with the downloaded release:

promtool --version
promtool, version 2.24.1 (branch: HEAD, revision: e4487274853c587717006eeda8804e597d120340)
  build user:       root@0b5231a0de0f
  build date:       20210120-00:09:36
  go version:       go1.15.6
  platform:         linux/amd64

I only saw this behaviour for this alert triggering only after 5 minutes. Is this linked to that?

I attach the whole test and alert definitions to this message.

Thanks in advance

Raphaël

The alert definition:

"groups":
- "name": "prom-alerts"
  "rules":
  - "alert": "NoServer"
    "annotations":
      "for": "5m"
      "message": "There is no server."
    "expr": "sum(kube_pod_container_info{container='server'}) < 1"
    "for": "5m"
    "labels":
      "severity": "error"

[rbauduin]

Here is a even simpler illustration:

# alert.yml definition: trigger is metric is < 1

"groups":
- "name": "prom-alerts"
  "rules":
  - "alert": "NoServer"
    "annotations":

      "message": "There is no server."

    "expr": "metric < 1"
    "labels":
      "severity": "error"

# test.yml: all 0 series, so alert should always be expected, yet this passes:
evaluation_interval: 1m
rule_files:
  - alert.yml
tests:
  - alert_rule_test:

      # this passes
      - alertname: NoServer
        eval_time: 5m
        exp_alerts:
          - exp_annotations:

              message: There is no server.
            exp_labels:
              severity: error

      # this too!

      - eval_time: 6m
        exp_alerts:
    input_series:
      - series: metric
        values: 0+0x20
    interval: 1m

The page at https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/ states:
  If you want to test if an alerting rule should not be firing, then you can mention the above fields and leave 'exp_alerts' empty.

Am I doing something wrong or should I report a bug?

Raphaël

[rbauduin]

Seems the alertname is needed for the alert to be checked, so this works:

# test.yml: all 0 series, so alert should always be expected, yet this passes:
evaluation_interval: 1m
rule_files:
  - alert.yml
tests:
  - alert_rule_test:

      # this passes
      - alertname: NoServer
        eval_time: 5m
        exp_alerts:
          - exp_annotations:

              message: There is no server.
            exp_labels:
              severity: error

      # this now reports an error as wanted!

      - alertname: NoServer
        eval_time: 6m

        exp_alerts:
    input_series:
      - series: metric
        values: 0+0x20
    interval: 1m