Prometheus showing decreasing graph for a non-decreasing gauge

[Sunil Anand]

We are using Prometheus for displaying several gauge type metrics. This specific gauge would always be increasing as it demonstrates total value.

From the query we could able to retrieve value that shows always in the increasing order across several intervals of 10s.

1680090660000000000 vsdptas1pcar-tas01.tc.corp see-1 cap_tcapProvider1 NEXUS_CAP_SERVICE_VOICE cap_tcapProvider1 18646863   <-- human time 13:51:00

1680090670000000000 vsdptas1pcar-tas01.tc.corp see-1 cap_tcapProvider1 NEXUS_CAP_SERVICE_VOICE cap_tcapProvider1 18647216   <-- human time 13:51:10

But when we plot the graph, sometimes this shows dip in the line which is abnormal for a non-decreasing metrics.

I have attached the graph for reference.

Does the UI have some bug or are we doing anything wrong in here, ?

[Brian Candler]

> 1680090660000000000 vsdptas1pcar-tas01.tc.corp see-1 cap_tcapProvider1 NEXUS_CAP_SERVICE_VOICE cap_tcapProvider1 18646863

That's not a prometheus-format metric. Where are you seeing that?

And those graphs aren't from prometheus either: it would show the metric name as

tas_see_TacpProvider_totalBeginSent{host="vsdptas1pcar-tas01.tc.corp",...}

not

{__name__="tas_see_TacpProvider_totalBeginSent",host="vsdptas1pcar-tas01.tc.corp",...}

(although I don't know what *version* of prometheus you're running, and if it's something ancient I suppose it could behave differently)

In any case: I suggest that you go to the Prometheus web interface, go to the PromQL expression editor, and enter

tas_see_TacpProvider_totalBeginSent{host="vsdptas1pcar-tas01.tc.corp"}[2m]

to see 2 minutes worth of raw metrics from the prometheus database, with timestamps.

Then change the expression to

tas_see_TacpProvider_totalBeginSent{host="vsdptas1pcar-tas01.tc.corp"}

and switch to the Graph tab.

I trust what it says is in the database is real. Therefore, if the graph shows values going down, and a database query shows values going down, then these are coming from your scraping process somehow.