remote_read: inconsistency between remote storage(influxDB) and Prometheus results.

[Sohaib Omar]

Hi all, I am using Prometheus as a query server, it just remote reads from the influx and is not responsible for any kind of scraping(scraping and remote_writing is done by Prometheus which is different than Prometheus which acts as a query server.). 

The problem I am facing is that there is an inconsistency between the remote_storage and Prometheus at remote_read time.

For example, even with the read_recent: true, when a certain time-series has stopped being appended in remote_storage, Prometheus(the one which remote_reads) is still able to return the result at least for 5 to 6 minutes later. The last point for below time-series is ~5 minutes old in influxDB but it still shows up in Prometheus.

below are my Prometheus configuration:

version: v2.17.1

global:
  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 1m
  external_labels:
    prometheus: monitoring/k8s
alerting:
  alert_relabel_configs:
  - separator: ;
    regex: prometheus_replica
    replacement: $1
    action: labeldrop
rule_files:
- /etc/prometheus/rules/prometheus-k8s-rulefiles-0/*.yaml
remote_read:
- url: http://data-influxdb.tick:8086/api/v1/prom/read?db=monitoringdb&rp=rp_1h&u=<user>&p=<pass>=
  remote_timeout: 1m
  read_recent: true

command-line flags:

[Stuart Clark]

On 11/05/2020 08:06, Sohaib Omar wrote:

For example, even with the read_recent: true, when a certain time-series has stopped being appended in remote_storage, Prometheus(the one which remote_reads) is still able to return the result at least for 5 to 6 minutes later. The last point for below time-series is ~5 minutes old in influxDB but it still shows up in Prometheus.

This is expected. When you query Prometheus for the current value of a metric it will actually look back up to five minutes to find it. After this point the time series is marked as stale. This is because it is pretty unlikely that a scrape will fall exactly "now", and could have last been scraped several minutes ago.

-- 
Stuart Clark

[Sohaib Omar]

Thanks, Stuart for the reply, is the look back configurable? I need my Prometheus queries result to be in sync with my remote storage.

[Sohaib Omar]

there is a  command-line parameter called lookback-delta which controls the loopback. read more about it here: https://prometheus.io/docs/prometheus/latest/migration/#flags