got: [] in promtool unit test

[radhamani...@gmail.com]

Hi,

I need help in this alert test.

This is the alert rule I have,

- alert: ServiceMemoryUsage90Percent

expr: sum(container_memory_usage_bytes{job="cadvisor",container_label_io_rancher_stack_name!="", image=~"rancher.*"}) by (container_label_io_rancher_stack_name, instance, dns_name) /1024/1024/1024 *100 > 90

for: 5m

labels:

severity: Critical

alert_owner: alerts-infra

annotations:

description: "Service memory has been at over 90% for 1 minute. Service: {{ $labels.container_label_io_rancher_stack_name }}, DnsName: {{ $labels.dns_name }}. \n"

I am trying to write a test for this,

tests:

# Infra Alert Tests

- input_series:

- series: 'container_memory_usage_bytes{job="cadvisor",container_label_io_rancher_stack_name="network-service",image="rancher.test"}'

values: '0 100000000 0 100000000 100000000'

alert_rule_test:

- alertname: ServiceMemoryUsage90Percent

# eval_time: 5m

exp_alerts:

- exp_labels:

alertname: ServiceMemoryUsage90Percent

severity: Critical

alert_owner: alerts-infra

exp_annotations:

description: "Service memory has been at over 90% for 1 minute. Service: (), DnsName: ()."

But I am always getting ,

Unit Testing:  tests/infra_db_alerts-tests.yml
  FAILED:
    alertname: ServiceMemoryUsage90Percent, time: 0s,
        exp:[
            0:
              Labels:{alert_owner="alerts-infra", alertname="ServiceMemoryUsage90Percent", severity="Critical"}
              Annotations:{description="Service memory has been at over 90% for 1 minute. Service: (), DnsName: ()."}
            ],
        got:[]

I tried many options in alert rule testing. But nothing helped.

Any help on this please?

not sure why I am getting got as empty. And always alert is getting triggered even if I keep input series value as 0.

[Brian Candler]

> And always alert is getting triggered even if I keep input series value as 0.

No, the alert is *not* being triggered. That's what "got: []" means: "I got an empty list of alerts" (although your test is written to *expect* a non-empty list of alerts, so the reality doesn't match expectations)

> Any help on this please?

Well, to start with:

100000000/1024/1024/1024 *100 = 9.31322574

So you need to add a zero to your test values.

Secondly, your alerting rule has "for: 5m". If you keep this, then the alert condition has to have been true for 5 minutes, and has to be evaluated at 5 minutes.  So you need:

        values: '1000000000 1000000000 1000000000 1000000000 1000000000 1000000000'

and uncomment "eval_time: 5m"

Alternatively, you can comment out "for: 5m" in your alerting rule, in which case you only need

        values: '1000000000'

After this, your alert triggers (it "got" an alert), but you'll find the labels and annotations don't match exactly what you'd put in the expectations:

  FAILED:
    alertname: ServiceMemoryUsage90Percent, time: 5m,

        exp:[
            0:
              Labels:{alert_owner="alerts-infra", alertname="ServiceMemoryUsage90Percent", severity="Critical"}
              Annotations:{description="Service memory has been at over 90% for 1 minute. Service: (), DnsName: ()."}
            ],
        got:[
            0:

              Labels:{alert_owner="alerts-infra", alertname="ServiceMemoryUsage90Percent", container_label_io_rancher_stack_name="network-service", severity="Critical"}
              Annotations:{description="Service memory has been at over 90% for 1 minute. Service: network-service, DnsName: . \n"}
            ]

You can then modify your expectations to make them match what the alerting rule actually generates - or you can modify the alerting rule itself to generate a different set of labels and annotations.

[radhamani...@gmail.com]

Great...Thankyou very much Brian..

After I fixed labels and annotations, everything works fine :)