WAL configuration to extend retention window

[sri L]

Hi all,

I am looking for options in configuration which helps WAL to store data for more than 2 hours in Prometheus. So that, incase of any issues with remotewrite we have data available for more than 2hours.

Please suggest the parameters that we can use WAL configuration to achieve this

[Brian Candler]

I'm not aware that you can. The issue regarding the limitation is still open:

https://github.com/prometheus/prometheus/issues/9607

which is linked from here:

https://prometheus.io/blog/2021/11/16/agent/

"This is currently limited to a two-hour buffer only, similar to non-agent Prometheus, hopefully unblocked soon." (written in Nov 2021)

[Brian Candler]

You could have a look at using Grafana Alloy to buffer prometheus remote_write:

https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.receive_http/#example

It has its own WAL with adjustable retention:

https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.remote_write/#wal-block

I have not tested this, but it might be worth investigating.