Prometheus remote write to Thanos issue

[Anup R Shetty]

Hi All,

I have Prometheus installed in my on-premise infra and Thanos in cloud.

I have given below remote write configuration in prometheus.yml file.

remote_write:

  - url: '<Thanos URL>'

    basic_auth:

      username: '<username>'

      password: '<password>'

    proxy_url: '<Proxy_URL>'

Remote write happening if connectivity from Prometheus to Thanos established correctly.

Problem:

Prometheus provides a default feature - retry on failure to send the data.

 When I remove the proxy for 10 min. Prometheus not able to send the data to Thanos. After 10 min, if I add the proxy back,

Expectation:

Prometheus should send the previous 10 min data to Thanos.

Actual:

But Prometheus is not retrying/sending previous 10 min data to Thanos. It will only send the data from the time when we added the proxy back.

Any suggestions are helpful.

Thank you.

Regards,

Anup