Diff between recording metrics and origin metrics

[David Miller]

the result has an obvious diff,like pictures

my recording rules:

 - record: sum:instance_cpu_quota:cell:product:marker:functional_marker

      expr: sum(instance_cpu_quota)by(cell,product,marker,functional_marker)

    - record: sum:instance_mem_quota:cell:product:marker:functional_marker

      expr: sum(instance_mem_quota)by(cell,product,marker,functional_marker)

    - record: sum:instance_mem_quota:cell:product:marker:functional_marker

      expr: sum(instance_mem_quota_bytes)by(cell,product,marker,functional_marker)

    - record: sum:instance_ssd_quota:cell:product:marker:functional_marker

      expr: sum(instance_ssd_quota_bytes)by(cell,product,marker,functional_marker)

    - record: sum:instance_hdd_quota:cell:product:marker:functional_marker

      expr: sum(instance_hdd_quota_bytes)by(cell,product,marker,functional_marker)

    - record: sum:instance_cpu_used:cell:product:marker:functional_marker

      expr: sum(instance_cpu_used)by(cell,product,marker,functional_marker)

    - record: sum:instance_mem_used:cell:product:marker:functional_marker

      expr: sum(instance_mem_anon_bytes)by(cell,product,marker,functional_marker)

how should I fix it?thank you

[Julien Pivotto]

Can we have a screenshot of the graph view, from the same timerange?
also, which version of prometheus are you using?

Thanks

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/79f17ddc-d2e9-4b7f-82d9-a0e7e4ebc0f2%40googlegroups.com.




--
(o- Julien Pivotto
//\ Open-Source Consultant
V_/_ Inuits - https://www.inuits.eu

[David Miller]

yes，this is my graph and version

I have set the log level to debug, but can not find any information usable for this

thank you

在 2020年3月27日星期五 UTC+8上午12:45:01，Julien Pivotto写道：

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Julien Pivotto]

Can we see the metrics and the rules screenshot?

Note that recording rules are evaluated every X seconds, depending on
your settings, which might explain the difference.

> > an email to promethe...@googlegroups.com <javascript:>.

> > > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/prometheus-users/79f17ddc-d2e9-4b7f-82d9-a0e7e4ebc0f2%40googlegroups.com.
> >
> >
> >
> >
> >
> > --
> > (o- Julien Pivotto
> > //\ Open-Source Consultant
> > V_/_ Inuits - https://www.inuits.eu
> >
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/297e7d50-a22c-4829-b161-dc529d674868%40googlegroups.com.

[David Miller]

this is the config and metric screenshot

在 2020年3月27日星期五 UTC+8上午12:55:38，Julien Pivotto写道：

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Julien Pivotto]

Are you using federation or exporting metrics with timestamps?

> > https://groups.google.com/d/msgid/prometheus-users/297e7d50-a22c-4829-b161-dc529d674868%40googlegroups.com.
> >
> >
> >
> >
> >
> > --
> > (o- Julien Pivotto
> > //\ Open-Source Consultant
> > V_/_ Inuits - https://www.inuits.eu
> >
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d145cd22-ed7f-4068-8242-f7c171a56b2a%40googlegroups.com.

[David Miller]

yes, I use federation to scrape from other prometheus instances to aggeragate some metrics. these metrics are with its own timestamps

在 2020年3月27日星期五 UTC+8上午1:02:28，Julien Pivotto写道：

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[David Miller]

there is some logs with debug level 

在 2020年3月27日星期五 UTC+8上午1:10:04，David Miller写道：

[Julien Pivotto]

Can you share your prometheus config files?

> > https://groups.google.com/d/msgid/prometheus-users/d145cd22-ed7f-4068-8242-f7c171a56b2a%40googlegroups.com.
> >
> >
> >
> >
> >
> > --
> > (o- Julien Pivotto
> > //\ Open-Source Consultant
> > V_/_ Inuits - https://www.inuits.eu
> >
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/18aaa55b-e9a8-40ae-a150-1916e86b2191%40googlegroups.com.

[David Miller]

sure

and I have use this in the old version prometheus, there is no diff

在 2020年3月27日星期五 UTC+8上午1:13:09，Julien Pivotto写道：

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.