PROMQL: how to add values when there is no data returned?

[John Tipper]

Hi all,

I have a data model where some metrics are namespaced by client, environment and deployment name. I am interested in creating a summary of each deployment, where that summary is based on the number of alerts that are present for each deployment.

I can get the deployments in the dev, uat, and prod environments using this query:

group by(tenant, environment, deployment)(up{environment=~"dev|uat|prod"}) - 1 # returns the following by way of example: {deployment="default",environment="dev",tenant="tenant1"} 0 {deployment="default",environment="prod",tenant="tenant3"} 0 {deployment="default",environment="prod",tenant="tenant2"} 0 {deployment="default",environment="uat",tenant="tenant1"} 0

So we can see that tenant 1 has 2 deployments in 2 different environments, whereas the other 2 have only one.  group by returns a value of 1, so we subtract 1 to get 0 for each deployment and I now wish to add to this the number of alerts that are applicable to each deployment.

To get the alerts, I do this:

ALERTS{severity="warning"} 

# returns something like this when there is an alert, the details in the alert will vary, but will always have the `tenant`, `environment` and `deployment` labels 

ALERTS{alertname="HostSystemdServiceCrashed",alertstate="firing",instance="example",job="node",deployment="default",environment="dev",tenant="tenant1",name="example.service",severity="warning",state="failed",type="oneshot"} 1 

# however, when there are no alerts, I get "no data" returned

I can't work out how to add the alerts to the deployments whilst retaining the deployments for which there were no alerts returned:

(group by(tenant, environment, deployment)(up{environment=~"dev|uat|prod"}) -1) + on(tenant, environment, deployment) (ALERTS{severity="warning"}) 

# returns only data for the deployment for which there is an alert {deployment="default",environment="dev",tenant="tenant1"} 1  

# if there are no alerts, I get no data returned at all

What I want as output is this:

{deployment="default",environment="dev",tenant="tenant1"} 1 {deployment="default",environment="uat",tenant="tenant1"} 0 {deployment="default",environment="prod",tenant="tenant2"} 0 {deployment="default",environment="prod",tenant="tenant3"} 0

How can I achieve this?

NOTE:

If I use sum with or, then I get this, depending on the order of the arguments to or:

(group by(tenant, environment, deployment)(up{environment=~"dev|uat|prod"}) -1) or sum by (tenant, environment, deployment) (ALERTS{severity="warning"} )

# returns this, note the value in `tenant1|dev|default` 

{deployment="default",environment="dev",tenant="tenant1"} 0 {deployment="default",environment="uat",tenant="tenant1"} 0 {deployment="default",environment="prod",tenant="tenant2"} 0 {deployment="default",environment="prod",tenant="tenant3"} 0

If I reverse the order of the parameters to or, I get what I am after:

{deployment="default",environment="dev",tenant="tenant1"} 1 {deployment="default",environment="uat",tenant="tenant1"} 0 {deployment="default",environment="prod",tenant="tenant2"} 0 {deployment="default",environment="prod",tenant="tenant3"} 0

But I'm stuck now if I want to do something like apply a weight to alerts of a different severity level, e.g. (pseudocode):

summary = 0 + sum(warning alerts) + 2*sum(alerts(critical alerts))

This gives the same single value series, or no data if there are no alerts.

Question is also posted to Stack Overflow, in case someone would prefer to answer there: https://stackoverflow.com/questions/64803483/promql-how-to-add-values-when-there-is-no-data-returned

Many thanks,

John