prometheus agent 2.49.1 - remote_write metrics while disconnected

[Alexander Wilke]

Hello,

I am running a prometheus server as a central instance.

I configured prometheus in agent mode on another system and send the logs via remote_write from agent to central prometheus. This works.

I changed the password in web.config.yml on the central server so it does not match for the prometheus agent anymore. I could see that metrics did not arrive anymore.

I waited form 2-3 minutes and then reconfigured the correct password on the central prometheus, di a reload and I coud see metrics arrive immediately after that change. Unfortunately I did not receive the metrics from the outage time.

my main intention was exactly this - that I can get the metrics if there is an issue in the network or whatever between agent and central prometheus.

Is this expected behaviour or misconfiguration/misunderstanding on my site?

[Alexander Wilke]

I found several posts which ask if it is possible that prometheus agent can store the metrics locally if remote_write destination is not available.

This post describes that it should work at least for 2hrs.:
https://prometheus.io/blog/2021/11/16/agent/

However here is a guthub issue which seems to describe the same issue I saw in my environment:

https://github.com/prometheus/prometheus/issues/11919

Can someone confirm this is working with 2.49.1 to store metrics for some time on prometheus agent until remotw_write is available again?

if yes how did you configure this?