Can't render graphs past 41 days

[Mike Cammilleri]

Hi Community,

We have a Grafana 8 instance that is using a Prometheus data source with metrics going back two years. We also remote write these metrics to a Victoria Metrics instance for the long term storage.

We use the Victoria Metrics snapshot API to create backups and store them into an on-prem S3 bucket. I'm doing a test restore of the data into a new location and I can point our production Grafana instance at that restored Prometheus/Victoria Metrics data and everything renders in the graphs (basic Node Exporter stuff) and I can go back the full two years. 

The issue is when I stand up a new, fresh instance of Grafana on a different host and set up the same restored Prometheus/Victoria Metrics as a data source. It will only render data going back 40 days. On the 41st day, the graphs all say No Data. I do not think there is an issue with the Victoria Metrics data and restore because it renders just fine on the production instance. The dashboard I"m using is Node Exporter Full (1860) and the only thing I can think of is that the dashboard on the production instance is an older import of the community dashboard 1860 while the newer instances of Grafana I set up will use the latest import. However, I tried exporting the dashboard JSON from production to the test Grafana instance and it still cannot render metrics past 40 days. 

Looking for any suggestions. Thanks!