Help with Prometheus subquery

[katzz...@gmail.com]

I am using following query:

sum(count_over_time(container_tasks_state{state="running", container_label_com_docker_swarm_task_name=~".+"}[35s]) ) by (container_label_com_docker_swarm_service_name, container_label_com_docker_swarm_node_id) < 2

How to change this query to make sure that state of the  found container is currently running?

 

Thanks

[Meier]

If what you are asking is to be alerted if certain metrics are missing you will need a "presence alert", see https://docs.google.com/presentation/d/1CD3ZFsvi1xMzm2wepPLLnLMt9sX0bqxfxEgcr65mgBw/edit#slide=id.g2364009b9b_0_58

In this case you would need to specify container_label_com_docker_swarm_task_name explicitly 

[katzz...@gmail.com]

Thank you for your answer but I want to define an alert without explicitly define a name of container

[Meier]

I thought so. Maybe I am completely wrong here but to me it sounds impossible. How would you find if sometihng is missing from an unknown collection of things if you don't ask for a specific thing?

[katzz...@gmail.com]

My scrape_interval is 15s. If my query returns value less than 2, this means that a running container was stopped or  a new container was started. This works fine for me, but I need to separate cases when a container was  stopped and when a container was started.