Adding timestamps to Gauge Metric

[Prince]

Hi everyone, I am new to prometheus. I am using type Gauge. I wanted to get the timestamp along with the value. It will be great if anyone can help on this.

example:

go_duration.WithLabelValues("type").Set(12345.678)

so here collector is getting only 12345.678, I am reading this data from a file where along with the vale there is a corresponding timestamp is there. I want that too.

[Julius Volz]

Hi,

You should be able to use the NewMetricWithTimestamp() function for this: https://pkg.go.dev/github.com/prometheus/client_golang/prometheus?utm_source=godoc#NewMetricWithTimestamp

Note that client-side timestamps should only be used in exceptional circumstances, and if you still expect those timestamps to be regularly updated (because otherwise Prometheus will just collect a dot here and there and mostly show empty graphs). If that is not the case, consider omitting the client-side timestamp and instead sending a metric that includes the last-update timestamp in its sample value (like the node_exporter does for the mtime metric in its "textfile" collector module: https://github.com/prometheus/node_exporter/blob/b6215e649cdfc0398ca98df8e63f3773f1725840/collector/textfile.go#L38)

Regards,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/f365ad0c-75b0-4bad-a75d-3980f0f61669n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Prince]

Thank you, As I understood the  NewMetricWithTimestamp() takes two parameters one the time and the other one is metric. So as my metric name is go_duration, so I did this :

1st way:

go func(){

go_duration.WithLabelValues("type").Set(12345.678)

prometheus.NewMetricWithTimestamp(time_var, go_duration )

}()

2nd way: 

go func(){

prometheus.NewMetricWithTimestamp(time_var,go_duration.WithLabelValues("type").Set(12345.678))

}()

Using 1st way not getting the timestamp only values are getting scarped.

Using 2nd way getting error as: "go_duration.WithLabelValues("type").Set(12345.678) used as a value"

[Julius Volz]

So NewMetricWithTimestamp() returns a Metric interface object that you can then emit from a Collector's Collect() method. See this example from cadvisor: https://github.com/google/cadvisor/blob/19df107fd64fa31efc90e186af91b97f38d205e9/metrics/prometheus.go#L1931-L1934

You can see more usage example here: https://sourcegraph.com/search?q=context:global+prometheus.NewMetricWithTimestamp&patternType=literal

In general, it seems like you are building an exporter (a process that proxies/translates existing values into the Prometheus format, in your case those existing values are coming from a file), so you are not instrumenting the exporting process itself, and thus you probably don't want to use the "NewGauge()" / "mygauge.WithLabelValues().Set()" functions that are for direct instrumentation of a process. Instead, you'll want to implement a Collector interface that just returns a set of proxied metrics, as outlined here:

* https://pkg.go.dev/github.com/prometheus/client_golang/prometheus#hdr-Custom_Collectors_and_constant_Metrics

* https://prometheus.io/docs/instrumenting/writing_exporters/#collectors

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/a69144c4-cfe1-42d8-9a46-a9fd1230f421n%40googlegroups.com.

[Prince]

Thank you, understood. I have used the following:

in Collect()

   t := time.Date(2021, time.August, 28, 07, 0, 0, 12345678, time.UTC)

    s := prometheus.MustNewConstMetric(c.metric, prometheus.GaugeValue, float64(s.value))

    ch<- prometheus.NewMetricWithTimestamp(t,s)

I am getting my_metric_name 152.401 1630134000012     (both things the value and timestamp), but I am not getting this timestamp in the x-axis of prometheus graph. Can You please let me how can I get that timestamps in The x-axis of prometheus graph?

[Prince]

Is it possible to give a custom timestamp in Prometheus for X-axis?

Example: I am getting my_metric_name 152.401 1630134000012 at metric endpoint but in Prometheus graph, I am getting the value 152.401 when it is scraped, but I want it should be displayed at 1630134000012(Saturday, August 28, 2021, 7:00:00.012 AM) this time in Prometheus graph.

Is it possible? if yes, can you please let me know how?

Thank you.

[Stuart Clark]

That very much depends on whatever tool you are using to display graphs.

However it is sounding like Prometheus may not be the right system depending on what you are trying to do. Prometheus is a metric system, which works by sampling the current state of a system at regular time periods, meaning the exact timestamp doesn't generally matter.

It sounds like you are instead wanting to record events - things that happen at a specific period of time, not at a regular frequency. For that use case you should look at an event store - something like Elasticsearch, InfluxDB or a standard relational or no-SQL database.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Ben Kochie]

Yup, Prometheus is a monitoring system, not a general use time-series database.

Specifically, Prometheus has a "look back window" where Prometheus will take the timestamp of your query and look back in time for samples to match.

This is fundamental to how PromQL works.

So no, what you are asking for is not possible. PromQL graph queries base the X-axis on the input parameters of the range query. The start, end, and step.

See the documentation on the API: https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/9C1C3B56-D825-47B3-8738-BF2E502A3E67%40Jahingo.com.

[Prince]

So let's suppose we are monitoring an event in such a way that the logs of this event are in a file and there is some value for that event and correspondingly time at when we got that value.

So in the Prometheus graph we can not show the value at that time???? 

Because I have a similar situation where I have a value and timestamp for that value. But when the Prometheus server starts running it shows the value at the current time not at it's a timestamp.

[Stuart Clark]

The key things you just said were "event" and "logs", both of which are not the metrics that Prometheus is designed for. Now it is possible to convert events/logs into metrics, but this sounds different to what you are wanting. Metrics created from logs would have regularly scraped metrics (where no timestamp is included) which might contain a counter of the number of events or the value of the last event (or maybe even a gauge containing the last timestamp of the event). These are then perfect for alerting when events stop happening, happen too often or produce values outside of allowed parameters.

If instead you are wanting to be able to store individual data points when events happen (which might be at any point, not aligned with a regular scrape interval) as it sounds then you want something different to Prometheus. You can use a standard SQL or no-SQL database (such as MySQL or DynamoDB) or a time series database (such as InfluxDB or Timescale DB). For many of the different options you can visualise them using Grafana, which allows you to show data from both Prometheus and your event store

[Prince]

So that means in the Prometheus graph the data will be getting displayed from the time of scraping and at a regular intervals (scrape interval).

Example: my_metric  1669.574 1630299163151(data and it's timestamp). So this data 1669.574 will be displayed at starting scrape time not at this  1630299163151 time.

**  163029916315 this is older time than the starting scrape time.

[Stuart Clark]

On 2021-08-30 07:19, Prince wrote:
> So that means in the Prometheus graph the data will be getting
> displayed from the time of scraping and at a regular intervals (scrape
> interval).
> Example: my_metric 1669.574 1630299163151(data and it's timestamp).
> So this data 1669.574 will be displayed at starting scrape time not at
> this 1630299163151 time.
>
> ** 163029916315 this is older time than the starting scrape time.
>

In general you shouldn't set the timestamp for a metric at all. There
are very few use cases where it should be used, with the main one being
when connecting another scrape based metric system to Prometheus (e.g.
CloudWatch).

For everything else you set the metric to the latest value (for a gauge)
and it will then update Prometheus during the next scrape. If you must
know the exact time of the last event (for example to alert if events
stop happening) you'd have a gauge whos value is that timestamp. But in
none of those situations would you set the metrics timestamp.

--
Stuart Clark

[Prince]

Thank you. Got It.

for NewMetricWithTimestamp(time.Time,metric)  Is it compulsory that the time should be in UTC?

[Bjoern Rabenstein]

On 31.08.21 04:27, Prince wrote:
>
> for NewMetricWithTimestamp(time.Time,metric) Is it compulsory that the
> time should be in UTC?

Short answer: No.

The Go `time.Time` type includes the time zone. It can use any time
zone. The library will then use the capabilities of the Go `time.Time`
type to convert it into Unix-time, as required by the exposition
format, which is independent of time zones.

In different news: Using the Prometheus instrumentation libraries to
instrument your code counts as using Prometheus and should be
discussed on the Prometheus users mailing list, see
https://groups.google.com/forum/#!forum/prometheus-users

This here is the mailing list to discuss development of the various
Prometheus components itself.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

[Prince]

Thanks