How to keep data of removed hosts?

[Martin Emrich]

Hello!

Our prometheus contains metrics of our kubernetes cluster.

Today, I wanted to look at data of a node (EC2 instance on AWS) that has been removed yesterday. But all the data regarding the VM is gone.
In contrast, data of a Kubernetes Pod that has been terminated is still visible later.

The nodes are discovered using ec2_sd_configs, the Kuberneted Pod data is scraped via federation from another prometheus instance running inside the K8s cluster.

Apparently there is a difference in behaviour regarding removed targets. But I cannot find a configuration parameter.

How can I keep historic data of removed VMs (or other targets) when the target itself is no longer there? I bet that is a FAQ, but I did not find information on how data removal works (beyond the storage configuration, which I expect works the same for all series)

Thanks,

Martin

[Stuart Clark]

How are you trying to see the older data & what is your retention period?

If a pod/EC2 instance goes away it just means no new data will be
scraped and any timeseries that were being scraped will go stale, but
the existing data should remain until the retention period is reached.

[Martin Emrich]

Hello Stuart!

My retention period is 90 days. I try to see the data via a grafana dashboard, as well as via the prometheus graphs feature, same effect.

All the kubernetes-related data is pulled via federate from another prometheus instance running inside the K8s cluster. 

Does the federate protocol include a mechanism to tell the pulling prometheus to delete series?

Do the scrape jobs delete series implicitly?

Thanks

Martin

[Stuart Clark]

On 2020-09-11 11:03, Martin Emrich wrote:
> Hello Stuart!
>
> My retention period is 90 days. I try to see the data via a grafana
> dashboard, as well as via the prometheus graphs feature, same effect.
>
> All the kubernetes-related data is pulled via federate from another
> prometheus instance running inside the K8s cluster.
> Does the federate protocol include a mechanism to tell the pulling
> prometheus to delete series?
>
> Do the scrape jobs delete series implicitly?
>

How are you trying to see things? Do you have the date range when the
instances existed selected?

What is the query you are trying to use?

--
Stuart Clark

[hark...@timescale.com]

Prometheus does not delete metrics by itself. I think taking the querying window to a day back should give you the results, provided you provide the correct metric nanme and labels.

[Martin Emrich]

Hi!

I tried it today to reproduce it… I took the same URL with the query from my browser history (as I did not write it down), and today the data history is there! 

Using the /prometheus/graph route, it works as expected: the graph begins where the instance is created, and ends where it is destroyed.

I have no Idea what went wrong earlier.

Thanks anyways!

Martin

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/5G_fUheUT0Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/126e1851-a8ad-43fa-a60b-4e19d1f1fc37n%40googlegroups.com.