Need help for Promtool rule test

[Shivanand Shete]

 Dear all,

Please find the below my prometheous alert and rules file , but I am not able test the alert using promtool.

I not able to understand where is the exact issue.

Please kindly help me.

Alert rule :

groups:

- name: distribution-service.alerts

rules:

- alert: Distribution_Service

expr: changes(process_start_time_seconds{job=~"distribution-service"}[5m]) > 1

for: 5m

labels:

severity: critical

annotations:

summary: Distribution-service too many restarts (instance {{ $labels.instance }})

description: Distribution-service has restarted more than once in the last 5 minutes. It might be crashlooping. VALUE = {{ $value }} LABELS = {{ $labels }}

Rule_test_File:

rule_files:

- ./distribution-service.yaml

evaluation_interval: 5m

tests:

- interval: 5m

input_series:

- series: 'process_start_time_seconds{job="distribution-service", instance="test-host"}'

values: '4'

alert_rule_test:

- eval_time: 5m

alertname: Distribution_Service

exp_alerts:

- exp_labels:

severity: warning

job: distribution-service

instance: test-host

exp_annotations:

summary: "Distribution-service too many restarts (instance test-host)"

description: "Distribution-service has restarted more than once in the last 5 minutes. It might be crashlooping. VALUE = 4 LABELS = map[instance:test-host job:distribution-service]"

[Brian Candler]

You only put one value in your test series data.  But you have a range vector expression, with changes(), and filtered with >1.  For the value to have changed more than 1 time (i.e. 2 or more times) you need at least 3 data points.

For an example of a test series with multiple data points, see

https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/#example

[Shivanand Shete]

As you suggested I updated input series data Still getting below error.

Could you please help me on which input series data I need to pass.

Alert:

groups:

- name: distribution-service.alerts

rules:

- alert: Distribution_Service

expr: changes(process_start_time_seconds{job=~"distribution-service"}[5m]) >= 1

for: 1m

labels:

severity: critical

annotations:

summary: Distribution-service too many restarts (instance {{ $labels.instance }})

description: Distribution-service has restarted more than once in the last 5 minutes. It might be crashlooping. VALUE = {{ $value }} LABELS = {{ $labels }}

Test Alert:

rule_files:

- ./distribution-service.yaml

evaluation_interval: 5m

tests:

- interval: 1m

input_series:

- series: 'process_start_time_seconds{job="distribution-service", instance="test-host"}'

values: '4 4 4 4 4'

alert_rule_test:

- eval_time: 5m

alertname: Distribution_Service

exp_alerts:

- exp_labels:

severity: critical

job: distribution-service

instance: test-host

exp_annotations:

summary: "Distribution-service too many restarts (instance test-host)"

description: "Distribution-service has restarted more than once in the last 5 minutes. It might be crashlooping. VALUE = 4 LABELS = map[instance:test-host job:distribution-service]"

Gerring Below Error:

Unit Testing:  distribution-service-test.yaml
  FAILED:
    alertname: Distribution_Service, time: 5m,
        exp:[
            0:
              Labels:{alertname="Distribution_Service", instance="test-host", job="distribution-service", severity="critical"}
              Annotations:{description="Distribution-service has restarted more than once in the last 5 minutes. It might be crashlooping. VALUE = 4 LABELS = map[instance:test-host job:distribution-service]", summary="Distribution-service too many restarts (instance test-host)"}
            ],
        got:[]

[Brian Candler]

Well, if the values of the timeseries that you provide are 4 4 4 4 4, answer me this: how many times has it changed in that period?

[Shivanand Shete]

Thanks for your reply.

I did some changes in input series like below.

In the input series I am passing '1 0 1 0 1' means within 5 min interval changes 3 times. restarts 3 times.

Alert:

groups:

- name: distribution-service.alerts

rules:

- alert: Distribution_Service

expr: changes(process_start_time_seconds{job=~"distribution-service"}[5m]) > 0

for: 5m

labels:

severity: critical

annotations:

summary: Distribution-service too many restarts (instance {{ $labels.instance }})

description: Distribution-service has restarted more than once in the last 5 minutes. It might be crashlooping. VALUE = {{ $value }} LABELS = {{ $labels }}

Test_rule.yaml:

rule_files:

- ./distribution-service.yaml

evaluation_interval: 1m

tests:

- interval: 1m

input_series:

- series: 'process_start_time_seconds{job="distribution-service", instance="test-host"}'

values: '1 0 1 0 1'

alert_rule_test:

- eval_time: 5m

alertname: Distribution_Service

exp_alerts:

- exp_labels:

severity: critical

job: distribution-service

instance: test-host

exp_annotations:

summary: "Distribution-service too many restarts (instance test-host)"

description: "Distribution-service has restarted more than once in the last 5 minutes. It might be crashlooping. VALUE = 6 LABELS = map[instance:test-host job:distribution-service]"

Receive Output error:

Unit Testing:  distribution-service-test.yaml
  FAILED:
    alertname: Distribution_Service, time: 5m,
        exp:[
            0:
              Labels:{alertname="Distribution_Service", instance="test-host", job="distribution-service", severity="critical"}

              Annotations:{description="Distribution-service has restarted more than once in the last 5 minutes. It might be crashlooping. VALUE = 6 LABELS = map[instance:test-host job:distribution-service]", summary="Distribution-service too many restarts (instance test-host)"}
            ],
        got:[]

[Brian Candler]

It's the "for: 5m" in the alerting rule which is suppressing it.  Try commenting that out.