Bug? Prometheus query incorrectly showing metrics data in Grafana

[chuanjia xing]

Hi there,

      I hit following issue when using Prometheus + Grafana, and want to check if this is a bug in Prometheus query. 

Scenario:

    I am using Prometheus to collect CPUUtilization for aws instances. In one dashboard, I want to show the metrics for services which are underutilized. To do this, I first use Grafana Variable "underutilized_services" to find out the service names; then I use this Variable in my Prometheus query to show the actual CPUUtilization data for these services. 

     I am using following Prometheus query:

     "avg by (service_id)($Metric{service_id=~"$underutilized_services"})"

and $underutilized_services" is a list of service names, like service1, service2...

     It is working well when the $underutilized_services is not empty, like folllwing (first graph)

     But if the $underutilized_services is empty, then my understanding is that it should now "No Data", but somehow it is showing something wired, like following (second graph).

      If you look at the legend of the second graph, it shows:

       "avg by (service_id)(aws_ec2_cpuutilization_p50{service_id=~"()"})"

      It looks like above query actually returns some value, which in my opinion it shouldn't. 

      This is showing some confusing data for my customer. Ideally it should show "No Data" in the second case. 

       Is there a bug in Prometheus? Or I am not using the query correctly?

[chuanjia xing]

Ping for this. Anyone can give some pointers for this? Thanks.

[Justin Toh]

Are you getting any data when you run the aws_ec2_cpuutilization_p50{service_id=~"()"} query?
The query above should return all the underlying time-series that were used to compute the values you're seeing in the 2nd screenshot.

Might be also good to share which version of Prometheus + Grafana you're running.

[Jiacai Liu]

The point is that service_id have regexp matchers, so it will
match all data instead of null.

On Thu, Apr 15, 2021 at 12:23:24 PM +0800, chuanjia xing wrote:

> Ping for this. Anyone can give some pointers for this? Thanks.
>
> On Tuesday, April 13, 2021 at 4:41:55 PM UTC-7 chuanjia xing
> wrote:
>
>> Hi there,
>>
>> I hit following issue when using Prometheus + Grafana,
>> and want to
>> check if this is a bug in Prometheus query.
>>
>> Scenario:
>>
>> I am using Prometheus to collect CPUUtilization for aws
>> instances. In
>> one dashboard, I want to show the metrics for services which
>> are
>> underutilized. To do this, I first use Grafana Variable
>> "underutilized_services" to find out the service names; then I
>> use this
>> Variable in my Prometheus query to show the actual
>> CPUUtilization data for
>> these services.
>>
>> I am using following Prometheus query:
>>

>> "*avg by
>> (service_id)($Metric{service_id=~"$underutilized_services"})*"

>>
>> and $underutilized_services" is a list of service names, like
>> service1,
>> service2...
>>
>> It is working well when the $underutilized_services is not
>> empty,
>> like folllwing (first graph)
>>
>> But if the $underutilized_services is empty, then my
>> understanding is
>> that it should now "No Data", but somehow it is showing
>> something wired,
>> like following (second graph).
>>
>> If you look at the legend of the second graph, it shows:
>>

>> "*avg by
>> (service_id)(aws_ec2_cpuutilization_p50{service_id=~"()"})*"

[chuanjia xing]

Thanks. In this case, it looks like a bug to me since for the following query, 

avg by (service_id)($Metric{service_id=~"$underutilized_services"})

if $underutilized_services" is empty, ideally it should return nothing instead of all data points by applying regex match.

btw, is there a way for me to get over this by using a different query?