Results of internal labels included queries

[Cemalettin Koc]

Hi,

I would like to query against internal labels as specified in the documentation.

avg_over_time({__name__=~".*_pool_Usage",quantile="0.999"}[1m]) 

This is resulting without __name__ or metric name in the beginning of data. I am expecting to include __name label or name in the beginning of the results. Why name of the series are not included in the results? What is wrong with it?

Thanks

[Brian Brazil]

Most functions and operators will remove the __name__, as the output is no longer that name (e.g. if I multiply users by 0, it's no longer a count of users).

You should look at using labels to distinguish these series rather than the metric name, or explicitly list each metric name you want.

--

Brian Brazil

www.robustperception.io

[Cemalettin Koc]

Thank you for your suggestion. We are in the process of migrating our metrics to client_java of prometheus. We will send some PR's for missing modules it has. However currently we are using coda hale's metric and unfortunately it is including names pool names in the metric name. 

I have still 2 questions:

1. I can understand that there is not an easy way to declaratively to specify which labels should be included in the result. I can only say for this that if a user includes __name__ in the query, it might be a good idea to return with it. At least __name__ with regex search might return __name__ since it is not clear which data is belonging to which series. Don't you think so?

2. Why are not results returning name of the series? I can only see "{job="kubernetes-applications-pods",kubernetes_namespace="prod",kubernetes_pod_name="xxx-yyy-app-2673296454-g451v",pod_template_hash="2673296454",quantile="0.999"} 21". I expect name of the series at the beginning as well. What is wrong with it?

[Brian Brazil]

On 1 May 2017 at 11:55, Cemalettin Koc <cemalet...@gmail.com> wrote:

Thank you for your suggestion. We are in the process of migrating our metrics to client_java of prometheus. We will send some PR's for missing modules it has. However currently we are using coda hale's metric and unfortunately it is including names pool names in the metric name. 

If you're looking for the Java memory stats, the simpleclient_hotspot module has all that. https://github.com/prometheus/client_java#included-collectors

 

I have still 2 questions:

1. I can understand that there is not an easy way to declaratively to specify which labels should be included in the result.

There is, the by/without clauses control this for aggregators.

 

I can only say for this that if a user includes __name__ in the query, it might be a good idea to return with it. At least __name__ with regex search might return __name__ since it is not clear which data is belonging to which series. Don't you think so?

We will as long, as there's no operations that might make the name invalid.

 

2. Why are not results returning name of the series? I can only see "{job="kubernetes-applications-pods",kubernetes_namespace="prod",kubernetes_pod_name="xxx-yyy-app-2673296454-g451v",pod_template_hash="2673296454",quantile="0.999"} 21". I expect name of the series at the beginning as well. What is wrong with it?

__name__ is special. Once you e.g. take an average over time we know longer know what the name is, so exclude it.

Brian 

On Monday, May 1, 2017 at 1:41:42 PM UTC+3, Brian Brazil wrote:

On 1 May 2017 at 11:35, Cemalettin Koc <cemalet...@gmail.com> wrote:

Hi,

I would like to query against internal labels as specified in the documentation.

avg_over_time({__name__=~".*_pool_Usage",quantile="0.999"}[1m]) 

This is resulting without __name__ or metric name in the beginning of data. I am expecting to include __name label or name in the beginning of the results. Why name of the series are not included in the results? What is wrong with it?

Most functions and operators will remove the __name__, as the output is no longer that name (e.g. if I multiply users by 0, it's no longer a count of users).

You should look at using labels to distinguish these series rather than the metric name, or explicitly list each metric name you want.

--

Brian Brazil

www.robustperception.io

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3f752983-5071-49cb-b5b0-3cd841cf967f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Cemalettin Koc]

__name__ is special. Once you e.g. take an average over time we know longer know what the name is, so exclude it.

If I query metric name with regex search, how I can know which data belongs to which series?  

 

[Julius Volz]

On Mon, May 1, 2017 at 1:24 PM, Cemalettin Koc <cemalet...@gmail.com> wrote:

__name__ is special. Once you e.g. take an average over time we know longer know what the name is, so exclude it.

If I query metric name with regex search, how I can know which data belongs to which series?  

As Brian mentioned, as long as you don't apply transformations that make the metric name invalid, the metric name will be returned. For example:

{__name__=~"node_disk.*"}

http://demo.robustperception.io:9090/graph?g0.range_input=1h&g0.expr=%7B__name__%3D~%22node_disk.*%22%7D&g0.tab=1

If you do avg_over_time(), the original metric names are no longer semantically correct for the resulting samples, so they get removed.

If you explicitly wanted to keep the name label somehow, you could use label_replace() to copy it into a normal label first.

For example:

label_replace({__name__=~"node_disk.*"}, "newlabelname", "$1", "__name__", "(.*)")

Unfortunately, we don't support label_replace() on range vectors yet, like:

label_replace({__name__=~"node_disk.*"}[1m], "newlabelname", "$1", "__name__", "(.*)")

Maybe we should support that though, hmm...

[Brian Brazil]

We don't have any functions that return range vectors, and I don't think this is a case to add it. It's presumed you know what your metric names are, doing regexes on __name__ is not encouraged and should only be done when debugging performance of Prometheus or exploring metric names.

--

Brian Brazil

www.robustperception.io

[Julius Volz]

Well, I guess that's your vote on https://github.com/prometheus/prometheus/issues/2671 then :)

[Cemalettin]

Please forgive my ignorance. I am quite new to prometheus. I can not follow why series semantically are not correct anymore.

{__name__=~"node_disk.*"} -> this returns with names.

{__name__=~"node_disk.*"}[60s] -> this range vector also returns with names

avg_over_time({__name__=~"node_disk.*"}[60s]) -> this does not return names

For me this should also return names since the range vector also is containing series are including names. Would you please explain me why this is semantically not correct? I would be glad that if you give an example too.

[Julius Volz]

On Mon, May 1, 2017 at 7:30 PM, Cemalettin <cemalet...@gmail.com> wrote:

Please forgive my ignorance. I am quite new to prometheus. I can not follow why series semantically are not correct anymore.

{__name__=~"node_disk.*"} -> this returns with names.

{__name__=~"node_disk.*"}[60s] -> this range vector also returns with names

avg_over_time({__name__=~"node_disk.*"}[60s]) -> this does not return names

For me this should also return names since the range vector also is containing series are including names. Would you please explain me why this is semantically not correct? I would be glad that if you give an example too.

A metric name describes the thing that is being measured. If you select a raw metric samples, the metric name matches what the sample value represents. If you do an operation like avg_over_time() on said samples, you now have a different metric, and the old name is no longer the right description for it. Rather, the new correct metric name would be something like "instance_device:node_disk_sectors_read:avg1m" - actually not sure what our naming convention for an over-time-averaged metric name suffix would be, but at least it's not the original metric anymore, it's a transformation thereof.

[Cemalettin Koc]

A metric name describes the thing that is being measured. If you select a raw metric samples, the metric name matches what the sample value represents. If you do an operation like avg_over_time() on said samples, you now have a different metric, and the old name is no longer the right description for it. Rather, the new correct metric name would be something like "instance_device:node_disk_sectors_read:avg1m" - actually not sure what our naming convention for an over-time-averaged metric name suffix would be, but at least it's not the original metric anymore, it's a transformation thereof.

I got it. Thank you for the explanation.

For those who hit this thread and curios about my solution, I used scraping config like this:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: (.+)_pool_.*

    target_label: pool_name 

This was really educative for me to learn some parts of it. I also agree with Brian about wrong modeling and trying to keep library compact. 

Thanks again Brian and Julius.