Recording Rules are not working
Priyank Agrawal
We have configured some recording rules in promethues master server to get different metric usage. Till last week those rules were working but now some of them are not working.
If we try to execute its expression we are getting data but if we try to execute with rule name it is showing “Empty Query Result”.
If possible, could anyone please help here.
Not Working Rules:
record:Disk_Usage_Node
expr:sum by(cluster, kubernetes_io_hostname) (rate(container_fs_usage_bytes{container_name!="POD"}[5m])) / 1.073741824e+09 * 100
labels:
Disk_Usage_Node: Disk_Usage_Node
OK -2m 18s ago 1.821ms
record:CPU_Usage_Node
expr:sum by(cluster, kubernetes_io_hostname) (rate(container_cpu_usage_seconds_total{container_name!="POD"}[5m])) / sum by(cluster, kubernetes_io_hostname) (machine_cpu_cores) * 100
labels:
CPU_Usage_Node: CPU_Usage_Node
OK -2m 18s ago 2.731ms
record:Memory_Usage_Node
expr:sum by(cluster, kubernetes_io_hostname) (rate(container_memory_usage_bytes{container_name!="POD"}[5m])) / 1.073741824e+09 * 100
labels:
Memory_Usage_Node: Memory_Usage_Node
Working Rules:
record:Current_Cluster_CPU_Usage
expr:sum by(cluster, kubernetes_io_hostname, node_role_kubernetes_io_worker) (rate(container_cpu_usage_seconds_total{id="/"}[1h]))
labels:
Current_Cluster_CPU_Usage: Current_Cluster_CPU_Usage
OK -2m 18s ago 0.779ms
record:Current_Cluster_Memory_Usage
expr:sum by(cluster, kubernetes_io_hostname, node_role_kubernetes_io_worker) (container_memory_working_set_bytes{id="/"}) / 1.073741824e+09
labels:
Current_Cluster_Memory_Usage: Current_Cluster_Memory_Usage
OK -2m 18s ago 0.426ms
record:Current_Cluster_Disk_Usage
expr:sum by(cluster, kubernetes_io_hostname, node_role_kubernetes_io_worker) (container_fs_usage_bytes{device=~"^/dev/.*$",id="/"}) / 1.073741824e+09
labels:
Current_Cluster_Disk_Usage: Current_Cluster_Disk_Usage
Thanks and Regards,
Priyank