Setting explicit timestamp to metrics with tsdb out_of_order_time_window option enabled not working

[Kesavanand Chavali]

Hello Experts,

We have a Windows Box on which Prometheus 2.45 version is running. 

1. The system is in future time unfortunately (+4hrs). 

2. We are running Thanos 0.34.2 as remote write and adjusted Thanos parameters to accept metrics till 12h into past or future.

3. We set storage -> tsdb -> out_of_order_time_window to 12h

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tsdb

4. We have a custom exporter written in C# that collects and exposes the metrics

5. We implemented a logic in this custom exporter to set the timestamp explicitly (subtract 4hrs from current time and set as timestamp for metric) 

https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#text-format-details

6. honor_timestamps set to true for custom exporter job

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config

We still see in Prometheus -> Targets, our exporter job is down with "too old sample".

Any help is greatly appreciated.

--

Thanks and Regards,

Kesav