Metrics with Future Time stamp and value

[Curious]

Hello Everyone, 

I'm using Prometheus metrics to train Facebook Prophet to forecast data for next 15 minutes. 

I plan to put back the forecasting data back into Prometheus. 

Tried the following:

1. Started exposing forecasting metrics along with forecasted timestamp 

metric_name [ "{" label_name "=" `"` label_value `"` { "," label_name "=" `"` label_value `"` } [ "," ] "}" ] value [ timestamp ] 

But not able to insert them into Prometheus as these are dropped by Promethues:

level=debug ts=2020-11-19T19:49:34.477Z caller=scrape.go:1255 component="scrape manager" scrape_pool=ml_test target=http://localhost:4000/ msg="Out of bounds metric" series="Usage{instance=\"instance01.us.west.local\"}"

Is there a way I can insert the future metrics into prometheus ? This would then used for comparison of actual v/s forecasted and generating alerts. 

TIA.  

[Brian Brazil]

This isn't possible, future data prevents ingestion of current data so there's a 10m sanity check. You can only really ingest over the past hour.

--

Brian Brazil

www.robustperception.io

[Stuart Clark]

You could remove the timestamp and just have the system present metrics when scraped that are actually the forecast for 15 minutes ahead (i.e. a scrape at 13:45:03 would actual return the forecast for 14:00:03).

Then in your queries you could use the offset modifier (https://prometheus.io/docs/prometheus/latest/querying/basics/#offset-modifier).

[Curious]

Thank you, Brian and Stuart for your prompt responses.  

The workaround suggested by Staurt might just do the trick. Will try it out and update here so that it could be useful for others. 

Thank you.