Disabling compaction/aggregation when using "tsdb create-blocks-from openmetrics"

269 views
Skip to first unread message

In Park

unread,
Mar 12, 2021, 2:28:17 AM3/12/21
to Prometheus Users
Hi,

I am using "tsdb create-blocks-from openmetrics <input file>" to backfill metrics into Prometheus. When I query for metrics, I see that the metrics are not being shown in the time which was specified within the input file to command but rather showed as an aggregated sum for every 2 hours. I am assuming this is because metrics are being  aggregated/compacted into 2 hour time blocks.  Is there a way to disable this compaction and have the metrics backfilled in the exact time which was specified in the file?

Julien Pivotto

unread,
Mar 12, 2021, 3:34:30 AM3/12/21
to In Park, Prometheus Users
If that happens then it's a bug. The metrics should be present exactly
as they were in the file.

Can you please share the request that makes you think it is not ingested
as in the file? You might use a high step to see your metrics.

Thank you

>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f4de77c9-31c1-48e0-aa81-4b6c6afb12b7n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

In Park

unread,
Mar 12, 2021, 4:14:52 AM3/12/21
to Prometheus Users
My apologizes, it seems like an issue with data points missing rather than what I described in my previous email.
I don't think the steps are the problem, I have tried with minute granularity:  /api/v1/query_range?query=group_committed_offset&start=1615453398.522&end=1615539798.522&step=60

The input file includes something like the following, which has a decreasing value every 5 mins  starting with the value of 576 until 1:
group_committed_offset{instance="host:24042",job="kafka",group="backfill_group",clientId="backfill_consumer",topic="backfill_topic",partition="0"} 576 1615533138
group_committed_offset{instance="host:24042",job="kafka",group="backfill_group",clientId="backfill_consumer",topic="backfill_topic",partition="0"} 575 1615532838
group_committed_offset{instance="host:24042",job="kafka",group="backfill_group",clientId="backfill_consumer",topic="backfill_topic",partition="0"} 574 1615532538
......
group_committed_offset{instance="host:24042",job="kafka",group="backfill_group",clientId="backfill_consumer",topic="backfill_topic",partition="0"} 1 1615360638

Looking at the query_range with step=60, you can see that it doesn't give me all the values specified in the input file :
  • values: 
    [
    • [
      • 1615456638.522,
      • "321"
      ],
    • [
      • 1615456698.522,
      • "321"
      ],
    • [
      • 1615456758.522,
      • "321"
      ],
    • [
      • 1615456818.522,
      • "321"
      ],
    • [
      • 1615456878.522,
      • "321"
      ],
    • [
      • 1615463838.522,
      • "345"
      ],
    • [
      • 1615463898.522,
      • "345"
      ],
    • [
      • 1615463958.522,
      • "345"
      ],
    • [
      • 1615464018.522,
      • "345"
      ],
    • [
      • 1615464078.522,
      • "345"
      ],
    • [
      • 1615471038.522,
      • "369"
      ],
    • [
      • 1615471098.522,
      • "369"
      ],
    • [
      • 1615471158.522,
      • "369"
      ],
    • [
      • 1615471218.522,
      • "369"
      ],
    • [
      • 1615471278.522,
      • "369"
      ],
    • [
      • 1615478238.522,
      • "393"
      ],
    • [
      • 1615478298.522,
      • "393"
      ],
    • [
      • 1615478358.522,
      • "393"
      ],
    • [
      • 1615478418.522,
      • "393"
      ],
    • [
      • 1615478478.522,
      • "393"
      ],
    • [
      • 1615485438.522,
      • "417"
      ],
    • [
      • 1615485498.522,
      • "417"
      ],
    • [
      • 1615485558.522,
      • "417"
      ],
    • [
      • 1615485618.522,
      • "417"
      ],
    • [
      • 1615485678.522,
      • "417"
      ],
    • [
      • 1615492638.522,
      • "441"
      ],
    • [
      • 1615492698.522,
      • "441"
      ],
    • [
      • 1615492758.522,
      • "441"
      ],
    • [
      • 1615492818.522,
      • "441"
      ],
    • [
      • 1615492878.522,
      • "441"
      ],
    • [
      • 1615499838.522,
      • "465"
      ],
    • [
      • 1615499898.522,
      • "465"
      ],
    • [
      • 1615499958.522,
      • "465"
      ],
    • [
      • 1615500018.522,
      • "465"
      ],
    • [
      • 1615500078.522,
      • "465"
      ],
    • [
      • 1615507038.522,
      • "489"
      ],
    • [
      • 1615507098.522,
      • "489"
      ],
    • [
      • 1615507158.522,
      • "489"
      ],
    • [
      • 1615507218.522,
      • "489"
      ],
    • [
      • 1615507278.522,
      • "489"
      ],
    • [
      • 1615514238.522,
      • "513"
      ],
    • [
      • 1615514298.522,
      • "513"
      ],
    • [
      • 1615514358.522,
      • "513"
      ],
    • [
      • 1615514418.522,
      • "513"
      ],
    • [
      • 1615514478.522,
      • "513"
      ],
    • [
      • 1615521438.522,
      • "537"
      ],
    • [
      • 1615521498.522,
      • "537"
      ],
    • [
      • 1615521558.522,
      • "537"
      ],
    • [
      • 1615521618.522,
      • "537"
      ],
    • [
      • 1615521678.522,
      • "537"
      ],
    • [
      • 1615528638.522,
      • "561"
      ],
    • [
      • 1615528698.522,
      • "561"
      ],
    • [
      • 1615528758.522,
      • "561"
      ],
    • [
      • 1615528818.522,
      • "561"
      ],
    • [
      • 1615528878.522,
      • "561"
      ],
    • [
      • 1615533138.522,
      • "576"
      ],
    • [
      • 1615533198.522,
      • "576"
      ],
    • [
      • 1615533258.522,
      • "576"
      ],
    • [
      • 1615533318.522,
      • "576"
      ],
    • [
      • 1615533378.522,
      • "576"
      ]
    ]

2021년 3월 12일 금요일 오후 5시 34분 30초 UTC+9에 Julien Pivotto님이 작성:

Julien Pivotto

unread,
Mar 12, 2021, 4:55:56 AM3/12/21
to In Park, Prometheus Users
This seems OK to me. It is the normal behaviour of the Prometheus query
API, which will get you the result every X step.

If you want to see what Prometheus actually has, can you try:

/api/v1/query?query=group_committed_offset[24h]&time=1615539798.522&step=60

Thanks!

On 12 Mar 01:14, In Park wrote:
> My apologizes, it seems like an issue with data points missing rather than
> what I described in my previous email.
> I don't think the steps are the problem, I have tried with minute
> granularity:
> /api/v1/query_range?query=group_committed_offset&start=1615453398.522&end=1615539798.522&step=60
>
> The input file includes something like the following, which has a
> decreasing value every 5 mins starting with the value of 576 until 1:
> group_committed_offset{instance="host:24042",job="kafka",group="backfill_group",clientId="backfill_consumer",topic="backfill_topic",partition="0"}
> 576 1615533138
> group_committed_offset{instance="host:24042",job="kafka",group="backfill_group",clientId="backfill_consumer",topic="backfill_topic",partition="0"}
> 575 1615532838
> group_committed_offset{instance="host:24042",job="kafka",group="backfill_group",clientId="backfill_consumer",topic="backfill_topic",partition="0"}
> 574 1615532538
> ......
> group_committed_offset{instance="host:24042",job="kafka",group="backfill_group",clientId="backfill_consumer",topic="backfill_topic",partition="0"}
> 1 1615360638
>
> Looking at the query_range with step=60, you can see that it doesn't give
> me all the values specified in the input file :
>
> - values:
> [
> -
> [
> - 1615456638.522,
> - "321"
> ],
> -
> [
> - 1615456698.522,
> - "321"
> ],
> -
> [
> - 1615456758.522,
> - "321"
> ],
> -
> [
> - 1615456818.522,
> - "321"
> ],
> -
> [
> - 1615456878.522,
> - "321"
> ],
> -
> [
> - 1615463838.522,
> - "345"
> ],
> -
> [
> - 1615463898.522,
> - "345"
> ],
> -
> [
> - 1615463958.522,
> - "345"
> ],
> -
> [
> - 1615464018.522,
> - "345"
> ],
> -
> [
> - 1615464078.522,
> - "345"
> ],
> -
> [
> - 1615471038.522,
> - "369"
> ],
> -
> [
> - 1615471098.522,
> - "369"
> ],
> -
> [
> - 1615471158.522,
> - "369"
> ],
> -
> [
> - 1615471218.522,
> - "369"
> ],
> -
> [
> - 1615471278.522,
> - "369"
> ],
> -
> [
> - 1615478238.522,
> - "393"
> ],
> -
> [
> - 1615478298.522,
> - "393"
> ],
> -
> [
> - 1615478358.522,
> - "393"
> ],
> -
> [
> - 1615478418.522,
> - "393"
> ],
> -
> [
> - 1615478478.522,
> - "393"
> ],
> -
> [
> - 1615485438.522,
> - "417"
> ],
> -
> [
> - 1615485498.522,
> - "417"
> ],
> -
> [
> - 1615485558.522,
> - "417"
> ],
> -
> [
> - 1615485618.522,
> - "417"
> ],
> -
> [
> - 1615485678.522,
> - "417"
> ],
> -
> [
> - 1615492638.522,
> - "441"
> ],
> -
> [
> - 1615492698.522,
> - "441"
> ],
> -
> [
> - 1615492758.522,
> - "441"
> ],
> -
> [
> - 1615492818.522,
> - "441"
> ],
> -
> [
> - 1615492878.522,
> - "441"
> ],
> -
> [
> - 1615499838.522,
> - "465"
> ],
> -
> [
> - 1615499898.522,
> - "465"
> ],
> -
> [
> - 1615499958.522,
> - "465"
> ],
> -
> [
> - 1615500018.522,
> - "465"
> ],
> -
> [
> - 1615500078.522,
> - "465"
> ],
> -
> [
> - 1615507038.522,
> - "489"
> ],
> -
> [
> - 1615507098.522,
> - "489"
> ],
> -
> [
> - 1615507158.522,
> - "489"
> ],
> -
> [
> - 1615507218.522,
> - "489"
> ],
> -
> [
> - 1615507278.522,
> - "489"
> ],
> -
> [
> - 1615514238.522,
> - "513"
> ],
> -
> [
> - 1615514298.522,
> - "513"
> ],
> -
> [
> - 1615514358.522,
> - "513"
> ],
> -
> [
> - 1615514418.522,
> - "513"
> ],
> -
> [
> - 1615514478.522,
> - "513"
> ],
> -
> [
> - 1615521438.522,
> - "537"
> ],
> -
> [
> - 1615521498.522,
> - "537"
> ],
> -
> [
> - 1615521558.522,
> - "537"
> ],
> -
> [
> - 1615521618.522,
> - "537"
> ],
> -
> [
> - 1615521678.522,
> - "537"
> ],
> -
> [
> - 1615528638.522,
> - "561"
> ],
> -
> [
> - 1615528698.522,
> - "561"
> ],
> -
> [
> - 1615528758.522,
> - "561"
> ],
> -
> [
> - 1615528818.522,
> - "561"
> ],
> -
> [
> - 1615528878.522,
> - "561"
> ],
> -
> [
> - 1615533138.522,
> - "576"
> ],
> -
> [
> - 1615533198.522,
> - "576"
> ],
> -
> [
> - 1615533258.522,
> - "576"
> ],
> -
> [
> - 1615533318.522,
> - "576"
> ],
> -
> [
> - 1615533378.522,
> - "576"
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f1322d53-8b4a-48ce-b94b-ebf7198bdb30n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

Julien Pivotto

unread,
Mar 12, 2021, 4:57:01 AM3/12/21
to In Park, Prometheus Users
On 12 Mar 10:55, Julien Pivotto wrote:
> This seems OK to me. It is the normal behaviour of the Prometheus query
> API, which will get you the result every X step.
>
> If you want to see what Prometheus actually has, can you try:
>
> /api/v1/query?query=group_committed_offset[24h]&time=1615539798.522&step=60

/api/v1/query?query=group_committed_offset[24h]&time=1615539798.522

there is no need for step here.
--
Julien Pivotto
@roidelapluie

In Park

unread,
Mar 12, 2021, 5:01:00 AM3/12/21
to Prometheus Users
Even without the step, it doesn't show certain data values which were in the input file.
For example, from the result I shared above between certain data points, there is around 2 hours worth of gap and the data between in not shown (or backfilled). I would expect values between 321-345 to be there but doesn't exist:

  • [
    • 1615456878.522,
    • "321"
    ],
  • [
    • 1615463838.522,
    • "345"
    ],

2021년 3월 12일 금요일 오후 6시 57분 1초 UTC+9에 Julien Pivotto님이 작성:

In Park

unread,
Mar 12, 2021, 5:12:05 AM3/12/21
to Prometheus Users
So if I execute group_committed_offset[24h] with the /query API without any steps, still I see missing data points. The following is the result from Prometheus table UI:

group_committed_offset{clientId="backfill_consumer", group="backfill_group", instance="kafka-0.default.svc.cluster.local:24042", job="kafka", partition="0", topic="backfill_topic"}
345 @1615463838 
369 @1615471038 
393 @1615478238 
417 @1615485438 
441 @1615492638 
465 @1615499838 
489 @1615507038 
513 @1615514238
537 @1615521438 
561 @1615528638 
576 @1615533138

data in the input file was increasing incrementally by 1 but in the result,  not all the data values are shown.
2021년 3월 12일 금요일 오후 7시 1분 0초 UTC+9에 In Park님이 작성:

In Park

unread,
Mar 12, 2021, 5:20:34 AM3/12/21
to Prometheus Users
Oh I see what is the problem, the blocks were overriding the existing ones, it is a mistake on my end. Sorry and thank you for looking into this.
2021년 3월 12일 금요일 오후 7시 12분 5초 UTC+9에 In Park님이 작성:
Reply all
Reply to author
Forward
0 new messages