Increase query not returning expected number.

[Ryan Rank]

I have an increase query that is not returning the results I'm expecting. It's consistently returning a smaller number than I expect.

I'm querying against a counter that only increases by an integer value. 

Here's the query:
sum without (field1, field2, field3, field4, field5, field6, field7, field8, field9, field10, field11,  field12, field13) (increase (apigateway_apicount_counter{field12="field12Value", field13=" field13 Value"}[15m]))

Results:

{node="node1"}   2.615 K

{node="node2"}   2.541 K

{node="node3"}   2.530 K

{node="node4"}   2.541 K

When I do a count against the real vales, I'm coming up with the following:

node1   4416
node2   4271
node3   3868
node4   3849

I do see the documentation stating "The increase is extrapolated to cover the full time range as specified in the range vector selector, so that it is possible to get a non-integer result even if a counter increases only by integer increments." Because of this, I do expect some variance, but not as much of a difference as I am seeing.

[Brian Candler]

How often are you scraping this metric?

Do these counters reset often?

What do you mean by "do a count against the real values" - what values are you looking at?

Assuming the counters don't reset often, you may find it useful to compare the output of

apigateway_apicount_counter

apigateway_apicount_counter offset 15m

and also the differences:

apigateway_apicount_counter - apigateway_apicount_counter offset 15m

You may also find sum by (node) (....) simpler than the long list of "without" fields.

[Ben Kochie]

The easiest way to see what Prometheus is looking at is to query for the raw data:

apigateway_apicount_counter{field12="field12Value", field13=" field13 Value"}[15m]

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e855f387-490b-4f93-96b3-5f72c3172e6dn%40googlegroups.com.

[Ryan Rank]

How often are you scraping this metric?

Every 30 seconds. Nowhere near enough to account for the difference. 

Do these counters reset often?

Nope. I'm using a custom exporter where I don't reset the counters. That said, if I restart the exporter, the counters will obviously reset.

What do you mean by "do a count against the real values" - what values are you looking at?

A little backstory. This is tracking API Calls to an API Gateway. We put a summary line of this information into our logs which is forwarded to Splunk. So Splunk captures the raw data. The API Gateway also forwards the API call data to the custom exporter. It's the exact same data. So when I say "Real values" I'm looking at the count of the summary line in Splunk filtering using the same parameters as my Prometheus Query.

Assuming the counters don't reset often, you may find it useful to compare the output of

apigateway_apicount_counter

apigateway_apicount_counter offset 15m

and also the differences:

apigateway_apicount_counter - apigateway_apicount_counter offset 15m

With all the labels involved, I'm dealing with a couple thousand rows in these queries. The last one (counter - offset) does have a difference in the numbers which is a value roughly equal to all the nodes combined in my original query.

You may also find sum by (node) (....) simpler than the long list of "without" fields.

Good call. Can you tell I'm new? ;-)