and operation with metric but different labels

[Baitanik Talukder]

I have the following metric in prometheus:

(test_case_failed_count_2nd{job="audit-test"}> 0) or (test_case_failed_count_1st{job="audit-test"} > 0)

Element	Value
test_case_failed_count_2nd{endpoint="http",job="audit-test",namespace="push-gateway",pod="prometheus-pushgateway-dc9cd85b6-8bk76",service="prometheus-pushgateway",summary_html_severe="None"}	1
test_case_failed_count_1st{endpoint="http",job="audit-test",namespace="push-gateway",pod="prometheus-pushgateway-dc9cd85b6-8bk76",service="prometheus-pushgateway",summary_html_warning="http://localhost/reports/2020-06-22_13-13-39/summary.html"}	8

Now when I am doing and operation:

(test_case_failed_count_2nd{job="audit-test"}> 0) and (test_case_failed_count_1st{job="audit-test"} > 0)

it returns blank... 

i need to get a bool true value or some data so that above expression I can put in prometheus rule for alert.
could any one please help..

[Brian Candler]

To get a bool value (0 or 1) add the "bool" qualifier, i.e.  foo > bool bar

See: https://prometheus.io/docs/prometheus/latest/querying/operators/#comparison-binary-operators

The and/or operators must match labels exactly, but using another operator like + or * you can match on different label sets:

foo * on(endpoint,namespace,pod,service) bar

Another approach is to use aggregate sum() or max() over them: e.g.

sum by (endpoint,namespace,prod,service) (test_case_failed_count_1st)

- this has the advantage that it will work even if there are multiple metrics with different summary_html_XXX flags for the same endpoint/namespace/prod/service combination.