metric timestamps out of band

[Johny]

 we are integrating a legacy system, that allows storing metrics with future timestamps, into our centralized prometheus instance. The use case here is that the forecast series is generated from external machine learning models and used for capacity planning, anomaly detection and stress analysis -- and alerting on forecast metrics.

I'm trying to improve my understanding of the constraints in Prometheus enforcing timestamps to be current. Besides implementation complexity of handling TS blocks with future timestamps, are there  design considerations on why this is unsupported? In this case, how do we integrate Prometheus with these legacy systems? Even the long term scalable stores such as Thanos, Cortex don't support this requirement.

Note that it is not feasible to replicate the models in PromQL in real time as the models are too complex and have dependencies on other workflows.

thanks

[Ben Kochie]

If you're producing metrics into the future, I would recommend using the remote write receiver model, rather than the scrape model. It handles this kind of use case much more easily.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7b475aa8-be4e-49f6-b9ff-18200a123ba3n%40googlegroups.com.

[Johny]

Does Prometheus support remote-write of metrics with future timestamps (+1,2 years)? I am aware other time series databases based on Prometheus model - such as Cortex, Thanos validate timestamps and throw out of band exceptions.

[Ben Kochie]

Hmm, I don't know what the limits are.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d5a801ea-fdbe-4d5b-92bd-5b650f28cca5n%40googlegroups.com.