Debugging sample timestamp out of order error

[Sumit Rindhe]

Hi Team,

I have multiple k8s clusters running in my staging environment. I am collecting metrics using vmagent on all the clusters and sending them to one central Cortex database. For one cluster I see the metrics being ingested properly in Cortex, but for the rest of the clusters I see the following error in vmagent:

"sample timestamp out of order; last timestamp: 1604621321.004, incoming timestamp: 1604620791.461 for series" 

Not sure what is causing this issue and since my changes work for one cluster and not others its more confusing.

Thanks,

Sumit