find timestamps for metric

[Dennis Ordanov]

Hi,

I am trying to visualize system metrics across different releases of software on just one host as a proof of concept.

I have a metric that is akin to the role metric that I am accepting discontinuity for and I would like to figure out the timestamps using the http api or anyway from prometheus really.

Let's say, I use the textfile collector to send a metric like this using the node_exporter:

version{version="1.1"}  #  Apr 2 - Apr 3 

version{verson="1.2"}   #  Apr 4  - Apr 5 

If I want to graph node_load1 and compare between the two versions I can do something like this (I think this is working like I want it to?) I have a build from HEAD of prometheus with the group_left changes.

node_load1 * on (instance, job) group_left(version) version{version=“1”} offset 24h

node_load1 * on (instance, job) group_left(version) version{version=“2”} offset 24h

Can I figure out the start and end timestamps for version{version="1"} so I can programmatically set the offsets?

If this is not possible (though I really really hope it is) would the recommendation be to post the offsets somewhere else pre-calculated for each version and patch Grafana or the prometheus console templates to get it or post the offsets to the Grafana dashboards API?

Are there other ways to accomplish what I am doing? Even if I add a label for the version for every metric that this system and application exports I am still left with super imposing them on the same graph which means calculating the offset.

Thanks for your time,

Dennis O.

[Björn Rabenstein]

On 17 May 2016 at 23:38, Dennis Ordanov <daod...@gmail.com> wrote:
>
> I have a metric that is akin to the role metric that I am accepting
> discontinuity for and I would like to figure out the timestamps using the
> http api or anyway from prometheus really.
>
> Let's say, I use the textfile collector to send a metric like this using the
> node_exporter:
>
> version{version="1.1"} # Apr 2 - Apr 3
> version{verson="1.2"} # Apr 4 - Apr 5

You still need to add a sample value here (usually "1" for a metric like that).

> If I want to graph node_load1 and compare between the two versions I can do
> something like this (I think this is working like I want it to?) I have a
> build from HEAD of prometheus with the group_left changes.
>
> node_load1 * on (instance, job) group_left(version) version{version=“1”}
> offset 24h
>
> node_load1 * on (instance, job) group_left(version) version{version=“2”}
> offset 24h
>
>
>
> Can I figure out the start and end timestamps for version{version="1"} so I
> can programmatically set the offsets?

There is no way to set the offset programmatically _within_ PromQL.

You could write a separate program that queries the time series for
`version`, picks the first timestamp for each series, and then
assembles a PromQL query with the right offset and runs it against the
Prometheus server (or pushes it into a Grafana dashboard).

It would require a lot of rewiring within the core of PromQL to make
your use case work within PromQL.

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

[Dennis]

On Wed, May 18, 2016 at 5:10 AM, Björn Rabenstein <bjo...@soundcloud.com> wrote:

On 17 May 2016 at 23:38, Dennis Ordanov <daod...@gmail.com> wrote:
>
> I have a metric that is akin to the role metric that I am accepting
> discontinuity for and I would like to figure out the timestamps using the
> http api or anyway from prometheus really.
>
> Let's say, I use the textfile collector to send a metric like this using the
> node_exporter:
>
> version{version="1.1"}  #  Apr 2 - Apr 3
> version{verson="1.2"}   #  Apr 4  - Apr 5

You still need to add a sample value here (usually "1" for a metric like that).

> If I want to graph node_load1 and compare between the two versions I can do
> something like this (I think this is working like I want it to?) I have a
> build from HEAD of prometheus with the group_left changes.
>
> node_load1 * on (instance, job) group_left(version) version{version=“1”}
> offset 24h
>
> node_load1 * on (instance, job) group_left(version) version{version=“2”}
> offset 24h
>
>
>
> Can I figure out the start and end timestamps for version{version="1"} so I
> can programmatically set the offsets?

There is no way to set the offset programmatically _within_ PromQL.

You could write a separate program that queries the time series for
`version`, picks the first timestamp for each series, and then

Is this the query_range endpoint and can I set the start time to be 1 and end time to be time.now() or do I have to page through time and set start and end times to be a week part for example?

I wonder if it's easier of I just store the start and end times for when a version of software was scheduled on some machine(s) elsewhere and not even have a version metric?

assembles a PromQL query with the right offset and runs it against the
Prometheus server (or pushes it into a Grafana dashboard).

It would require a lot of rewiring within the core of PromQL to make
your use case work within PromQL.

Dennis O.

[Björn Rabenstein]

On 18 May 2016 at 20:19, Dennis <daod...@gmail.com> wrote:
>
> Is this the query_range endpoint and can I set the start time to be 1
> and end time to be time.now() or do I have to page through time and
> set start and end times to be a week part for example?

There is no fundamental reason against querying the whole time range.
You might get a copious amount of results, obviously, but since it is
a single series, there are queries way worse.

[Dennis]

Hi,

This is kind of working, but guessing the step value is proving challenging or I am doing something else wrong to figure out the beginning of a metric + label combination.

I am using some made up dates and they are only about a day apart afaict and I receive an error.

$ curl 'localhost:9090/api/v1/query_range?query=wrk_http_benchmark_up&start=2016-05-18T20:10:30.781Z&end=2016-05-22T20:10:30.781Z&step=30s'

{"status":"error","errorType":"bad_data","error":"exceeded maximum resolution of 11,000 points per timeseries. Try decreasing the query resolution (?step=XX)"}

1) If I change this to 40 seconds it starts working, where is the 11k timeseries number coming from and how would I use start = 1 for instance in this case and if I use a time_range even farther in the past then I have to use an even larger resolution.

2) I tried using a label in hopes that I get better results, but I just get no results, is the below even the correct syntax?

$ curl -v 'localhost:9090/api/v1/query_range?query=wrk_http_benchmark_up\{run="31"\}&start=2016-05-18T20:10:30.781Z&end=2016-05-23T20:10:30.781Z&step=50s'

I was imagining I can do something like 

$ curl -v 'localhost:9090/api/v1/query_range?query=wrk_http_benchmark_up\{run="31"\}&start=1&end=2016-05-23T20:10:30.781Z&step=10s'  # I only need the first timestamp (though getting the last time stamp would be nice too.

[Dennis]

On Thu, May 19, 2016 at 4:18 AM, Dennis <daod...@gmail.com> wrote:

Hi,

This is kind of working, but guessing the step value is proving challenging or I am doing something else wrong to figure out the beginning of a metric + label combination.

I am using some made up dates and they are only about a day apart afaict and I receive an error.

$ curl 'localhost:9090/api/v1/query_range?query=wrk_http_benchmark_up&start=2016-05-18T20:10:30.781Z&end=2016-05-22T20:10:30.781Z&step=30s'

{"status":"error","errorType":"bad_data","error":"exceeded maximum resolution of 11,000 points per timeseries. Try decreasing the query resolution (?step=XX)"}

1) If I change this to 40 seconds it starts working, where is the 11k timeseries number coming from and how would I use start = 1 for instance in this case and if I use a time_range even farther in the past then I have to use an even larger resolution.

2) I tried using a label in hopes that I get better results, but I just get no results, is the below even the correct syntax?

$ curl -v 'localhost:9090/api/v1/query_range?query=wrk_http_benchmark_up\{run="31"\}&start=2016-05-18T20:10:30.781Z&end=2016-05-23T20:10:30.781Z&step=50s'

I was imagining I can do something like 

$ curl -v 'localhost:9090/api/v1/query_range?query=wrk_http_benchmark_up\{run="31"\}&start=1&end=2016-05-23T20:10:30.781Z&step=10s'  # I only need the first timestamp (though getting the last time stamp would be nice too.

I should mention that I can use /query, but I suspect this will break down under some conditions (under which?) and I was hoping I could avoid it in some fashion?

$ curl -v 'localhost:9090/api/v1/query?query=wrk_http_benchmark_up\[365d\]'|python -m json.tool

[Björn Rabenstein]

On 19 May 2016 at 13:34, Dennis <daod...@gmail.com> wrote:
> I should mention that I can use /query, but I suspect this will break down
> under some conditions (under which?) and I was hoping I could avoid it in
> some fashion?
>
> $ curl -v
> 'localhost:9090/api/v1/query?query=wrk_http_benchmark_up\[365d\]'|python -m
> json.tool

Yes, that's what I would do. If it gets too big, you have to partition it.

Prometheus really has no notion of when a series starts or end.
(That's part of the problem in
https://github.com/prometheus/prometheus/issues/398 .)