Error on ingesting out-of-order with k8s Federation

[Evelyn Pereira Souza]

Hi

My log file is full with messages like this

level=warn ts=2021-05-14T05:23:49.382Z caller=scrape.go:1372
component="scrape manager" scrape_pool=federate
target="https://prometheus.qna-prod-in.k8s.example.org:9090/federate?match%5B%5D=%7Bjob%3D%22ambassador-admin%22%7D&match%5B%5D=%7Bjob%3D%22kubelet%22%7D&match%5B%5D=%7Bjob%3D%22kube-state-metrics%22%7D&match%5B%5D=%7Bjob%3D%22node-exporter%22%7D"
msg="Error on ingesting out-of-order samples" num_dropped=1714

full log fle:
https://privatebin.at/?010661c7e7e29590#AdjCnhUs9zrHQmYbfWcXPrCmAV9ofpkvDTgeY5JtfwSV
(warning: takes some time to load because 6.2 MB)

federation config:

- job_name: federate
scrape_interval: 30s
scrape_timeout: 30s
honor_labels: true
metrics_path: /federate
scheme: https
tls_config:
insecure_skip_verify: true
params:
match[]:
- '{job="ambassador-admin"}'
- '{job="kubelet"}'
- '{job="kube-state-metrics"}'
- '{job="node-exporter"}'
file_sd_configs:
- files:
- DC-43-east.yml
relabel_configs:
- source_labels:
- __address__
regex: (.*)
replacement: ${1}:9090
target_label: __address__

When I curl manually the federate endpoint I get 54 MB of metrics.

What you advise me to fix "Error on ingesting out-of-order"?
It's only a warning in log.

best regards
Evelyn