Large number of "Error on ingesting out-of-order samples" messages

[John Dexter]

I have a prometheus server scraping about 10 targets on each of about 5 different machines. We've not noticed any issues but looking at the log I can see some targets are reporting out-of-order sample warnings as often as once a minute or so (scraping happens every 15s).

When I inspect the URL manually everything appears fine and the scrape takes about 50ms. What exactly has to happen for prometheus to detect this situation? When I see something like:

>msg="Error on ingesting out-of-order samples" num_dropped=144

What does num_dropped refer to exactly - the number of individual time-series exposed in the dropped scrape?

Thanks.