PromQL queries not supporting OR and | logical operator

[Moksha Reddy G]

Hey there,

I am struggling to find simple and efficient OR logic operator for my Prometheus queries which are running through Grafana dashboards. 

Problem Statement: 

Case1: Below query is not at all working as it contains other condition in beginning. We have many queries like that and need the correct syntax to use logic OR condition. Label value should be common as $clustername in both label keys

sum(irate(pilot_xds_pushes{type="cds",clusterName=" $clustername|", k8s_cluster=" $clustername|"}[5m]))

Case2: Below query is responding with results but it is giving Empty query result when I choose the date/time before I introduced clusterName as a label. Label value should be common as $clustername in both label keys.

sum(kube_pod_info{k8s_cluster=~"$clustername|", clusterName=~"$clustername|"}) by (clusterName, k8s_cluster)

For example: If I choose 1 month old timeframe when there is no label called clusterName then NO results displayed. If I choose current timeframe then it works. I guess it is because the availability of the label in Prometheus database. What is the logic from Prometheus backend?

Case3: If I put OR operator in between these two conditions then its working but OR is not working in all cases due to binary expressions in queries. Below is the query:

sum(kube_pod_info{k8s_cluster=~"$clustername"} or {clusterName=~"$clustername"}) by (clusterName, k8s_cluster)

Could you please look into this issue on high priority and kindly share your inputs to use this OR operator without any issues in all these scenarios?

Best,

Reddy

[Brian Candler]

In your case 1, 

clusterName=" $clustername|"

should be
clusterName=~" $clustername|"

Your screenshot shows this mistake as well.

You stated "Below query is not at all working as it contains other condition in beginning."  I think you need to clarify both parts of that statement:

(1) in what way is it not working? Show the input metrics, the result of the query, and what result you're actually looking for.

If the problem is that it returns an empty result set (as per your screenshot), that's because you used the wrong label match operator, "=" instead of "=~".  It will only match a clusterName which has the exact literal value "d3-prd-w2|" (including the vertical bar).

(2) "as it contains other condition in beginning" doesn't mean anything to me. What conditions? Do you mean the label filter type="cds"?

Then clearly it will only match metrics that have that label. Is that not what you want?

In your case 3, I think you have another bug:

(kube_pod_info{k8s_cluster=~"$clustername"} or {clusterName=~"$clustername"})

should be

(kube_pod_info{k8s_cluster=~"$clustername"} or kube_pod_info{clusterName=~"$clustername"})

>  OR is not working in all cases due to binary expressions in queries

That doesn't mean anything. The semantics of the OR operator are clearly defined. Show examples of what metrics you are feeding into this query, what you get as the output, and what you would *like* to see instead, and we may be able to help you formulate a query that does what you want.

In other words, the problem is not that "OR is not working" - the problem is that you haven't formulated your PromQL query in a way which gets you the results you're looking for.

> Could you please look into this issue on high priority

I refer you to http://www.catb.org/~esr/faqs/smart-questions.html#urgent

(however, the whole of that document is well worth reading)

[Moksha Reddy G]

Hi Brian,

Thanks for your response.

Now I am clear about my case1, but still I have questions on case2 and case3. I prefer to go one by one, please find my question below on Case2:

Case2: Below queries are responding with results but it is giving Empty query results when I choose the date/time before I introduced clusterName as a label. Label value should be common as $clustername in both label keys.

sum(kube_pod_info{k8s_cluster=~"dd-stg|", clusterName=~"dd-stg|"}) by (clusterName, k8s_cluster)

sum(irate(pilot_xds_pushes{type=~"rds", clusterName=~"dd-stg|", k8s_cluster=~"dd-stg|"}[5m])) by (clusterName, k8s_cluster)

For example: If I choose a 1 month old timeline(2023-05-01) when there is no label called clusterName then NO results displayed. If I choose the current timeframe then it works. clusterName label has been introduced recently one month back, It is because of the availability of the label at that point in time. BUT we need old data also with both(clusterName, k8s_cluster) labels so what is the correct way of defining the query in such cases?

Best,

Reddy

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/4XO_43ajVuc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/54816f0f-ffe5-4684-9ed3-489eb0debcd8n%40googlegroups.com.

[Brian Candler]

If you group by label, and the label is missing or empty (which are the same thing in prometheus), you get no results. e.g.

    sum by (foo) (node_filesystem_avail_bytes)

I think what you have to do is to use label_replace to synthesize a dummy value for the missing label:

    sum by (foo) (label_replace(node_filesystem_avail_bytes,"foo","unset","foo",""))

This converts

    mymetric{bar="baz"}

into

   mymetric{bar="baz",foo="unset"}