Preserving metric name when using recording rule

[Chih-Chieh Huang]

Would like to know either it is a bad idea or not possible for the scenario described below.

The motivation is to reduce the frequency and number of queries Grafana sends to Prometheus by combining multiple metrics into one new metric at the same time preserving the origin metric name.

Define a new metric using recording rule
 
    C = A OR B

A and B are two different metrics (such as one is cpu_memory the other is disk_usage) but they have same or very similar labels.
C is the new metric generated by the recording rule using OR operation

Is it also possible to make existing metric name become a label in the new metric C. (or any other way to preserve A and B's metric name).

Any suggestion?

Thank you.

[Brian Brazil]

I'd strongly advise against going down this path. Are you having particular performance problems?

--

Brian Brazil

www.robustperception.io

[Chih-Chieh Huang]

In some sense, yes.

If for each "environment" a Grafana panel is configured with 10 queries, then I would have N * 10 queries sent from browser to Grafana then to Prometheus (N being the number of environments).  When using browser's dev tool, some queries have high TTFB (time to first byte) but if I executed those queries directly in Prometheus, the response time is not bad at all. Makes me wonder if browser has very few connections to send queries and they are lots of queries to be sent and some of them will just have to wait.  If there is a way I can reduce the number of queries for each environment, then the hope is the overall user experience will be better.

What are other solutions I can try?

thanks

[Jasmine Hegman]

Chih-Chieh, you have wondered correctly. Browsers have connection limits against particular servers which affect AJAX requests. One web development oriented solution to avoid this is by using multiple (sub)domains that point to the same machine - this allows you to surpass the limit, based on the cardinality of dimensions used. The problem is the complexity that arises. It's one factor that made CDNs quite popular.

To solve this may require work more on Grafana's end than Prometheus'. Here are some ideas I see:

* Introduce batch queries to Prometheus and alter Grafana to use them. Grafana supports other datasources and how this would work with them deserves to be considered. As Brian mentioned is the complexity worth introducing?

* Grafana could be modified to try and stream queries in the background, perhaps ahead of time. This is more complexity and even likely wastes bandwidth with prediction misses but could potentially give you better UX when looking at those charts.

* Grafana could store last retrieved results in local storage and utilize that data while the query is running. This is simplest approach to reduce time spent looking at grafana panels with progress indicators spinning.

* Users could expose Prometheus through multiple endpoints and have Grafana round-robin through them or use them randomly. This would be the best - Grafana needs to handle multiple server configs, but already has precedent with CSVs doing something similar in many areas of it. Otherwise it pushes all onus on the end user. e.g. Now instead of "prometheus.com" datasource endpoint, we feed Grafana a.prometheus.com, b.prometheus.com" endpoint. If A records point to the same prometheus the queries should still execute properly but all in parallel, giving you the quick response you desire with no change to Prometheus.

I think the fourth is best but not sure how to advise in a way that will work for all cases - I imagine domains and how people are scraping is quite different between users.

[Jasmine Hegman]

I am so sorry, I forgot to include the link regarding parallel requests http://stackoverflow.com/questions/985431/max-parallel-http-connections-in-a-browser

[Chih-Chieh Huang]

thank you for the pointers, Jasmine.