Mutiple PromQLs for each expr in alert rules

[nina guo]

Hello guys,

Is there a way to report an alert like this:

expr: if containerCPUusage>80 and serverCPUusge>80

Then trigger the alert.

[Henrik Peng]

you can combine them in a monitor rather than in a alert.

[Brian Candler]

Yes, but you need to be really clear on what you're doing.  What you need to realise is that this is a *vector* expression:

    expr: containerCPUusage > 80

This may return zero, one, or many results in a vector: that is, a set of all timeseries with metric name "containerCPUusage" that meet the condition.  In general, a given metric can refer to multiple timeseries, distinguished by having different label sets.

The expression "serverCPUusage>80" is the same, but it could return a vector with a different set of labels.  So the clever part is being able to associated containerCPUusage with serverCPUusage, by finding a subset of labels which allows you to do the matching in a unique way, for example N containers link to 1 server.  It's documented here.

If you show real examples of the complete metrics, with their full sets of labels, we may be able to give more specific help.

[nina guo]

Thank you for your reply.

Let me clarify what we are going to do more.

We have the following alert rules for a container. But we still want to include the CPU usage of the corresponding host which host this container. That is:
 - when CPU usage for container is larger than 85%, and CPU usage for host is larger than 85%, then the alert is triggered.

#rules for container

- alert: ContainerCpuUsage

        expr: (sum(rate(container_cpu_usage_seconds_total{name=~".+"}[3m])) BY (instance, name) * 100) > 85

        for: 10m

        labels:

          severity: warning

        annotations:

          summary: "Container CPU usage (instance {{ $labels.instance }})"

          description: "Container CPU usage is above 80%\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

#rules for host

- alert: HighCpuLoad

          expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85

          for: 5m

          labels:

            severity: critical

          annotations:

            summary: "High CPU load (instance {{ $labels.instance }})"

            description: "[TEST] CPU load is > 80%\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

#rules for 

[Stuart Clark]

Are there any labels for container_cpu_usage_seconds_total &
node_cpu_seconds_total which match?

--
Stuart Clark

[nina guo]

May I ask how to check all the available labels for this? Can we define new labels?

For instances, we use file discovery to get the monitored target list.

[Stuart Clark]

On 06/12/2021 08:43, nina guo wrote:
> May I ask how to check all the available labels for this? Can we
> define new labels?
>
> For instances, we use file discovery to get the monitored target list.

Just enter the metric name in the query page of the Prometheus UI and
you'll see all the different labels & their values.

--
Stuart Clark

[nina guo]

Thank you.

in  container_cpu_usage_seconds_total, there are following labels:

instance

name

kubernetes_io_arch

hostname

job

[nina guo]

in  container_cpu_usage_seconds_total, there are following labels:

instance

name

kubernetes_io_arch

hostname

job

cpu (the value of this label is "total")

[nina guo]

Could you provide some help on this topic?

[Brian Candler]

As I said before: if you can show real examples of the complete metrics, with their full sets of labels, then we may be able to help.

What do I mean by this?  For example, if I go into the PromQL web interface and type "node_filesystem_avail_bytes" as the query, I get the following:

node_filesystem_avail_bytes{device="/dev/mapper/ganeti-root", fstype="ext4", instance="nuc1", job="node", mountpoint="/", netbox_type="device"} 10666057728

node_filesystem_avail_bytes{device="/dev/mapper/ubuntu--vg-ubuntu--lv", fstype="ext4", instance="nuc2", job="node", mountpoint="/", netbox_type="device"} 14423576576

node_filesystem_avail_bytes{device="/dev/sda1", fstype="vfat", instance="nuc2", job="node", mountpoint="/boot/efi", netbox_type="device"} 530321408

... etc

What I'm asking for is for you to do this separately for both "container_cpu_usage_seconds_total" and "node_cpu_usage_seconds_total" (or whatever the actual metrics are).

For each one, show us a handful of *actual* results, with real labels and values.  Then it will be possible to suggest a query which will combine them.