Need help in promtool test - its failing for one alert or the other

[radhamani...@gmail.com]

Hi,

I have below prometheus unit test 

I tried a lot of manipulations on input_series. But it is always throwing failure. 
It only evaluates the first value in the input series.


Unit Testing:  tests/infra_db_alerts-tests.yml
  FAILED:
    alertname: MySQLReplicationDown, time: 10m,
        exp:[
            0:
              Labels:{alert_owner="alerts-infra", alertname="MySQLReplicationDown", severity="High"}
              Annotations:{description="MySQL replication is broken. \n"}
            ],
        got:[]

rule_files:

- ../rules/db_alerts.yml

tests:

# MySQL Alert Tests

- input_series:

- series: mysql_up

values: 0

- series: mysql_global_variables_innodb_replication_delay

values: '1 1 1 1 1 0 0 0 0 -1 -1 -1 -1 -1 -1'

alert_rule_test:

- alertname: MySQLReplicationLag

eval_time: 5m

exp_alerts:

- exp_labels:

severity: High

alert_owner: alerts-infra

exp_annotations:

description: "There is a MySQL replication delay. \n"

- alertname: MySQLReplicationDown

eval_time: 10m

exp_alerts:

- exp_labels:

severity: High

alert_owner: alerts-infra

exp_annotations:

description: "MySQL replication is broken. \n"

[Brian Candler]

You haven't included your alerting rules file.

I also note you provide only one value for the mysql_up series. You might want something like "0x15" there. See:

https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/#series

[radhamani...@gmail.com]

Thankyou Brian for helping..

I am passing multiple values as , values: '1 1 1 1 1 0 0 0 0 -1 -1 -1 -1 -1 -1' Is this not correct?

Alert rules file contains,

- alert: MySQLReplicationLag

expr: mysql_global_variables_innodb_replication_delay > 0

for: 5m

labels:

severity: High

alert_owner: alerts-infra

annotations:

description: "There is a MySQL replication delay. \n"

- alert: MySQLReplicationDown

expr: mysql_global_variables_innodb_replication_delay == -1

for: 5m

labels:

severity: High

alert_owner: alerts-infra

annotations:

description: "MySQL replication is broken. \n"

[radhamani...@gmail.com]

When I tried with ,

tests:

# MySQL Alert Tests

- input_series:

- series: mysql_up

values: 0

- series: mysql_global_variables_innodb_replication_delay

values: '1*5 0*15 -1*5'

I get parse error,

Unit Testing:  tests/infra_db_alerts-tests.yml
  FAILED:
1:50: parse error: unexpected character in series sequence: '*'

[radhamani...@gmail.com]

Now I tried this,

- series: mysql_global_variables_innodb_replication_delay

values: '1x5 0x15 -1x5'

This didnt create any parser error.

But its failing with same error

Unit Testing:  tests/infra_db_alerts-tests.yml
  FAILED:
    alertname: MySQLReplicationDown, time: 5m,

        exp:[
            0:
              Labels:{alert_owner="alerts-infra", alertname="MySQLReplicationDown", severity="High"}
              Annotations:{description="MySQL replication is broken. \n"}
            ],
        got:[]

[Brian Candler]

On Tuesday, 19 September 2023 at 21:38:12 UTC+1 radhamani...@gmail.com wrote:

I am passing multiple values as , values: '1 1 1 1 1 0 0 0 0 -1 -1 -1 -1 -1 -1'

(Not for the mysql_up series you're not - and it was unknown at that point whether you were using that series in your alerting rules.

Since you're not using it, you might as well remove it.)

I found the solution, but it's a bit weird and I'd like to raise it as an issue.  You need to explicitly set the "interval" on your rule test to 1m. Once you've done that, the additional problems are:

- "for: 5m" means you need the expression to be alerting for *6* data points

- the 'eval_time' needs to point to a time where it has been alerting for 6 data points

It works with the following changes:

--- test-rule3.yaml.orig 2023-09-20 08:12:40.181790387 +0100
+++ test-rule3.yaml 2023-09-20 08:32:27.828676576 +0100
@@ -7,8 +7,9 @@

       - series: mysql_up
         values: 0
       - series: mysql_global_variables_innodb_replication_delay

-        values: '1 1 1 1 1 0 0 0 0 -1 -1 -1 -1 -1 -1'
+        values: '1 1 1 1 1 1 0 0 0 0 -1 -1 -1 -1 -1 -1'

+    interval: 1m

     alert_rule_test:
       - alertname: MySQLReplicationLag
         eval_time: 5m

@@ -20,7 +21,7 @@

               description: "There is a MySQL replication delay. \n"

       - alertname: MySQLReplicationDown

-        eval_time: 10m
+        eval_time: 15m

[Brian Candler]

Raised as https://github.com/prometheus/prometheus/issues/12871