Help understanding graph of a PromQL query

[Rajat Varyani]

Hello,

I am trying to find the execution time of a cronjob in k8s cluster which is obtained using:

kube_job_status_completion_time{namespace="some-namespace", job_name=~"some-job-\\d+"} - kube_job_status_start_time{namespace="some-namespace", job_name=~"some-job-\\d+"}

The result of the query is time in seconds. However when I plot this as a graph the start point and end point of the line are way too off. I have attached a graph of a job having execution time of 184 seconds but the start time shown on graph is 4:33:30am and end time is 06:06:15am for the value of 184.

I don't seem to understand:

•  If the Kubernetes job lasted for few minutes how can the value have such start time and end time.
• How to potentially debug such scenarios? 

Regards,

Rajat

[Brian Candler]

Try plotting graphs separately of:

kube_job_status_completion_time{namespace="some-namespace", job_name=~"some-job-\\d+"}

kube_job_status_start_time{namespace="some-namespace", job_name=~"some-job-\\d+"}

and I think it will become clearer.

Each metric has values at different times, from when the metric was first exported until it stopped being exported.  In this case each metric has a constant value, but it still exists for a range of times.

A query like "A - B", taken at instant t, returns a result whenever both metrics (with matching labels) have a value at that time.

If you give such a query to grafana, then it will sweep t over a range, which is the selected time range for the graph, and the graph will show a point whenever the expression has a value.

I am guessing that your exporter was returning values for both kube_job_status_completion_time and kube_job_status_start_time (for that particular job) between 4:43am and 6:06am, and therefore Grafana will show the value of the expression when both metrics have a value.

For example: maybe kube_job_status_start_time existed from 4:40am to 06:06am, and kube_job_status_completion_time existed from 4:43am to 06:10am.  Your graph will show values for all the times that both metrics exist, i.e. 4:43am to 6:06am.

[Rajat Varyani]

Thanks Brian. You are correct.  We are using kube-state-metrics to export. The behavior of the exporter is to export unless and until the object is deleted from the cluster. As we were tracking jobs in this case, they are not  cleaned up automatically. Hence the exporter exports the metric even after the job finishes execution.

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/HTkKApeIJmE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/101a37d2-03c9-480c-88f1-03a0a0ea6048%40googlegroups.com.