Prometheus and Thanos metrics are removed when Pushgateway metrics is deleted

[Shaahul Hameed]

Hi there,

I am running pushgateway, prometheus and thanos nodes and facing an issue while deleting metrics in Pushgateway.

When I delete the group metrics in Pushgateway, the corresponding(consumed) metrics in Prometheus and Thanos store gets removed as well. I can see the metrics is uploaded to S3 by sidecar but don't understand why the metrics is lost when it disappears in Pushgateway.

I thought the metrics should stay in Prometheus/Thanos even if it gets lost in Pushgateway. 

Can you please help with this? I am kind of lost and not able to figure out why this is happening.

[Bjoern Rabenstein]

On 15.07.21 16:41, Shaahul Hameed wrote:
>
> When I delete the group metrics in Pushgateway, the corresponding(consumed)
> metrics in Prometheus and Thanos store gets removed as well. I can see the
> metrics is uploaded to S3 by sidecar but don't understand why the metrics
> is lost when it disappears in Pushgateway.
>
> I thought the metrics should stay in Prometheus/Thanos even if it gets lost
> in Pushgateway.

What do you mean by "lost"? If you run an instant query for the
current timestamp, they will not show up anymore once they cannot be
scraped from the Pushgateway anymore. But if you run an instant query
for the relevant past, or just a range query covering the relevant
past, you should still see them.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in