Failure rate ratio exceeding 1.0 with two independent counters

[tejaswini vadlamudi]

Hi,

A failure rate query uses two separate counters:

  sum(rate(application_soap_unsuccessful_responses_sent_total[1m]))
  / sum(rate(application_soap_requests_recv_total[1m]))

 The observed value was approximately 1.0004 (displayed as 100.04% in the dashboard).

These are two independent counters incremented at different points in the application:

  - requests_recv_total - incremented at request ingress
  - unsuccessful_responses_sent_total - incremented when a failure response is sent back

The standard error rate pattern uses label filtering on a single metric:

  sum(rate(http_requests_total{status="500"}[5m])) / sum(rate(http_requests_total[5m]))

In that case the numerator is a subset of the denominator, so the ratio is bounded to 0-1.

With two independent counters, the ratio can exceed 1.0 due to:

  1. A request received at the end of one rate window producing its failure response in the next window
  2. Counter reset compensation applied independently per metric during pod restarts
  3. Scrape boundary misalignment between the two counters

Is this understanding correct? Is clamp(..., 0, 1) the recommended approach to bound the ratio, or is there a better pattern for calculating failure rates from two independent counters with a short (1m) rate window?

Thanks,

Teja

[Brian Candler]

I'd say that the derived value is potentially inaccurate always. You could clamp it to (0,1) which would hide the problem, but equally when the result is e.g. 50% it doesn't necessarily mean what you think it means, for the same reasons you gave.

Do you have an available metric:

application_soap_successful_responses_sent_total

or

application_soap_responses_sent_total   # includes both successful and unsuccessful responses

?

If so, using that would allow you to calculate an accurate metric.

[tejaswini vadlamudi]

  Hi Brian,

  Good point. You are right that the inaccuracy applies across the range, not only at >100%.

  Unfortunately, neither of those metrics exists. The application only exposes two counters:

  - application_soap_requests_recv_total - incremented at request ingress
  - application_soap_unsuccessful_responses_sent_total - incremented when a failure response is sent

  There is no application_soap_successful_responses_sent_total or combined application_soap_responses_sent_total. The two counters are incremented at different code points in the request processing pipeline, so the numerator is not a subset of the denominator.

  Given that constraint, would clamp() be the least-bad approach for bounding the display, or is widening the rate window (from 1m to 5m) a better trade-off to reduce the drift frequency?

  For reliable alerting, I assume the only proper fix is adding application_soap_successful_responses_sent_total counter to the application so the query can use:

  unsuccessful / (successful + unsuccessful)

  Thanks,
  Teja

[Brian Candler]

I think you're doing the best that's possible given the wrong metrics.

Whether increasing the window from 1 minutes to 5 minutes would be an improvement depends on the specific behaviour and how you use this metric. It may give you a more accurate answer when it has stabilised, but it will be slower to converge and will hide spikes - which might be a benefit or a disadvantage, depending on your use case.

If this data is used for alerting then you may have to make a trade-off between timely alerts versus spurious alerts.