Prometheus Specific Time Range Every Day Query

[Robert Parker]

I've got a similar problem to this unanswered SO issue being posed here: 

https://stackoverflow.com/questions/43400092/query-for-the-same-time-period-of-every-day-in-prometheus

I need to produce a table report in Grafana (not via ad-hoc API queries to Prometheus) that looks something like this:

```

Average CPU Utilization by Namespace

Namespace  | 12am-6am   6am-12pm   12pm-6pm    6pm-12am
---------  | -----------------------------------------
ns1        | 12.4       33.4       43.6        56.7
ns2        | 12.3       56.7       45.6        89.6

ns3        | 15.7       45.6       23.5        23.8

...

``` 

The basi  query I'm using is: 

avg_over_time(namespace_name:container_cpu_usage_seconds_total:sum_rate[6h]) 

Which is an out of the box K8S recording rule: 

rules:

    - expr: |

        sum(rate(container_cpu_usage_seconds_total{job="kubelet", image!=""}[5m])) by (namespace)

      record: namespace:container_cpu_usage_seconds_total:sum_rate

What I think I need is something like this that will give me the average over time for only those elements in the series selected by the user in the Grafana UI that are recorded between 12am and 6am EST: 

avg_over_time(namespace_name:container_cpu_usage_seconds_total:sum_rate[12am-6am])

That is, if the user has "last 24 hours" selected in the Grafana UI, there will be one contiguous range of data in the series, and if they have "last 48 hours" selected, there will be two 12am-6pm series in the range to give me the avg_over_time on.

Of course, "query['time of day start'-'time of day end']" appears not to be valid in PromQL.  I thought about using the offset function, but that would require that the user set a specific time in the UI range, which is not what I want.  I want something like "the average of all values over the past week (or whatever the user has selected in the UI) but only among those scraped between 12am and 6am no matter what day those were recorded on".

Is this doable with Prometheus+Grafana and if so, how?

Thanks in advance.