topk() sorting

[Peter Zaitsev]

Hi,

I think topk() is absolutely fantastic function which allows to pick top K items from the many time series which might be available

I wonder though when topk() returns time series how does it sort result ? It looks like it is sorting by one of the labels or all of them.   Is it possible to tell it some way to sort it some way to put the highest series on top ?

Here is example what I get with Grafana  showing MySQL commands from mysqld_exporter

--

Peter Zaitsev, CEO, Percona
Tel: +1 888 401 3401 ext 7360   Skype:  peter_zaitsev

[Julius Volz]

Prometheus itself returns topk() results sorted from highest to lowest value (as can be evidenced if you evaluate your expression in Prometheus's expression browser).

So what you are running into here is more of a Grafana visualization issue, maybe best to check with them or look at their docs to see the possibilities around series sorting in the legend?

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julius Volz]

Keep in mind also that topk() can produce different orderings over time, so in a graph, there's no "one" correct way to sort things.

See also https://github.com/prometheus/prometheus/issues/586

[Peter Zaitsev]

Julius,

Well I went to  Grafana guys first... They told me what in their model it is the plugin responsibility to sort the data and even though there is a way to sort the data in the table on the side the graphs will be drawn in he order they are returned, for example for purposes of stacking.

I recognize what in the different times topK  can well produce different values    but in the end there is going to be  N (different than K) of series returned which will have at least some values filled ?

Whenever it is implemented on Prometheus side or Grafana  Plugin side it is good if   for example sorting by the average (over period plotted) can be implemented

[Julius Volz]

Yeah, I have to correct myself, in that Prometheus does *not* sort topk() results for *graph* queries (because of the aforementioned ambiguity about what is the right sorting), only for instant (tabular) queries.

The problem about sorting by the average over the plotted period is that this is not really supported by Prometheus computational model. A query that is computed over time has no notion of what the graph range is, or that it is even going to be used as a graph. Prometheus simply evaluates the query as an instant query again and again, at different timestamps, then stitches the result together at the end. So this would either require a completely new way of handling queries in Prometheus, or needs to be handled on the client side somehow. This is also discussed in https://github.com/prometheus/prometheus/issues/586

[Peter Zaitsev]

Thanks,

Are you guys working with Grafana folks to make sure the  Prometheus Plugin is a first class citizen ?

The sorting of the data on UI part is surely possible and I really do not care where it happens.  It just looks like right now both Dev teams think it is someone else's responsibility :)

[Nicholas Capo]

I'm pretty sure (grafana 2.6) you can click the header of the table (small blue text) and it will sort by that column.

Or am I confused about what you want?

Nicholas

[Julius Volz]

Yeah, if Grafana 2.6 enables that, that would be great.

We are in loose contact with Grafana people about the Prometheus integration, but haven't dealt with this issue yet.

[Peter Zaitsev]

Nicholas.

Yes. You can sort the table on the side, however  this will not show the data in the graph, as such data in the tooltip will remain unsorted as well as graphs will be plotted in random order - which matters for stacked graphs.

[Nicholas Capo]

Ah, I see.