Gauge Exemplars

[Fabian Stäber]

Hi,

I'm experimenting with Exemplars for Gauge metrics in client_java (background: at the dev summit on 10 November 2022 we decided that "Prometheus will ingest Exemplars on all time series").

For comparison: A counter MUST have the following methods:

• inc(): Increment the counter by 1
• inc(double v): Increment the counter by the given amount. MUST check that v >= 0.

Exemplars will contain the increment, i.e. if inc() is called the Exemplar will have the value 1.0, if inc(v) is called the Exemplar will have the value v.

Now, A gauge MUST have the following methods:

• inc(): Increment the gauge by 1
• inc(double v): Increment the gauge by the given amount
• dec(): Decrement the gauge by 1
• dec(double v): Decrement the gauge by the given amount
• set(double v): Set the gauge to the given value

Which value should we choose for Gauge Exemplars?

Fabian

[Julien Pivotto]

On 06 Dec 23:06, 'Fabian Stäber' via Prometheus Developers wrote:
> Hi,
>
> I'm experimenting with Exemplars for Gauge metrics in client_java
> (background: at the dev summit on 10 November 2022

> <https://docs.google.com/document/d/11LC3wJcVk00l8w5P3oLQ-m3Y37iom6INAMEu2ZAGIIE/edit>

> we decided that "Prometheus will ingest Exemplars on all time series").
>
> For comparison: A counter MUST have the following methods

> <https://prometheus.io/docs/instrumenting/writing_clientlibs/#counter>:
>
> - inc(): Increment the counter by 1
> - inc(double v): Increment the counter by the given amount. MUST check

> that v >= 0.
>
> Exemplars will contain the increment, i.e. if inc() is called the Exemplar
> will have the value 1.0, if inc(v) is called the Exemplar will have the
> value v.
>
> Now, A gauge MUST have the following methods

> <https://prometheus.io/docs/instrumenting/writing_clientlibs/#gauge>:
>
> - inc(): Increment the gauge by 1
> - inc(double v): Increment the gauge by the given amount
> - dec(): Decrement the gauge by 1
> - dec(double v): Decrement the gauge by the given amount
> - set(double v): Set the gauge to the given value

>
> Which value should we choose for Gauge Exemplars?

That is a good question and my guess would be that it will depend on the
use case. Do you have examples of gauges would would attach exemplars
to?

--
Julien Pivotto
@roidelapluie

[Fabian Stäber]

Thanks Julien. I guess there are examples for both, having Exemplars with the delta as well as the absolute value.

Example 1) Number of Bytes in a queue. You would call myGauge.inc(nBytes) if a message is added to the queue, and myGauge.dec(nBytes) if a message is fetched from the queue. If adding / fetching is done via REST API, it would make sense to have Exemplars for the add/fetch operations.

Example 2) An IoT device reaching out to a Web hook for reporting temperature. To track the current temperature, you'd call myGauge.set(temperature). If the IoT device is capable of adding trace headers when calling the Web hook it would be good to have Exemplars of these calls.

The Java library supports automatic Exemplars, i.e. users don't call explicit API to add Exemplars, but the library figures out if a current trace context exists and samples Exemplars automatically under the hood. For that, we need a default behavior. So what would a good default be? Always the total value of the Gauge, or always the delta, or make it depend on whether inc(), dec(), or set() was called?

Fabian

[Matthias Rampke]

At scrape time, how would I know which method was used? "always the total value of the gauge" seems like the least surprising choice to me. If users need to track separate exemplars for increment and decrement, they could use two counters (basically, a non-native UpDownCounter), which conceptually and physically preserves that the final value is the result of repeated additions and removals.

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAPX310ipsufix2s4_P4PXOvzGr4MFDbHaaBmmJZXpnW-k%2Bas6g%40mail.gmail.com.

[Fabian Stäber]

> At scrape time, how would I know which method was used?

The library keeps a list of Exemplars that are provided at scrape time (in case of a Gauge the list only has 1 element, because Gauges only have 1 Exemplar).

The question is what to put in the list when inc(), dec(), or set() are called.

My feeling is also that the total value of the gauge is least surprising. But that would also mean that for cases like "number of Bytes in a queue" you'd not see the number of Bytes added as an Exemplar value, just the total queue size.

Fabian

[Matthias Rampke]

Sorry, unclear phrasing – I mean as the one scraping (whether by hand or as a Prometheus server), how would I know what the semantics of the value in the provided exemplar are, whether they are a delta or an absolute value? Again, I think "it's always an absolute value" makes this least surprising.

This overlaps with the discussion we had about exemplars from recording rules – if there was the option to have multiple exemplars for one gauge, it could separately present exemplars for increment, decrement, and set, whichever ones have been used …

/MR

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAPX310ggv%2BthbzFYx5j71B%2ByyOOy9d-64nPAa00-yNd6jcZbkQ%40mail.gmail.com.

[Fabian Stäber]

> if there was the option to have multiple exemplars for one gauge

That's an interesting thought. Actually, OpenTelemetry metrics have an optional list of Exemplars for each metric, no matter whether it's a counter, gauge, histogram, or exponential histogram. https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto

In Prometheus we are currently headed to complex Exemplar rules: Counters and Gauges can have one, Histograms can have one per bucket plus one for count and one for sum, Native histograms can have an arbitrary list, ...

As we are creating a new version of OpenMetrics protobuf anyway, we could consider switching to the OTel model and just have an arbitrary list of Exemplars for each metric.

Fabian