Endless 'Remote storage resharding' when influxdb used as remote storage with retention

[wats...@gmail.com]

Hello. I use influxdb as remote storage for reading and writing. I've created retention policies and continuous queries in influxdb: 

CREATE RETENTION POLICY "one_week" ON prometheus DURATION 1w REPLICATION 1 SHARD DURATION 1w;

CREATE RETENTION POLICY "one_month" ON prometheus DURATION 4w REPLICATION 1 SHARD DURATION 1w;

CREATE RETENTION POLICY "one_year" ON prometheus DURATION 52w REPLICATION 1 SHARD DURATION 1w;

ALTER RETENTION POLICY "autogen" ON prometheus DURATION 1d REPLICATION 1 SHARD DURATION 1d DEFAULT

CREATE CONTINUOUS QUERY prometheus_1w_5m ON prometheus BEGIN SELECT mean(*) INTO "one_week".:MEASUREMENT FROM /.*/ GROUP BY time(5m),* END;

CREATE CONTINUOUS QUERY prometheus_1M_30m ON prometheus BEGIN SELECT mean(*) INTO "one_month".:MEASUREMENT FROM /.*/ GROUP BY time(30m),* END;

CREATE CONTINUOUS QUERY prometheus_1y_1h ON prometheus BEGIN SELECT mean(*) INTO "one_year".:MEASUREMENT FROM /.*/ GROUP BY time(1h),* END;

Now I see endless  'Remote storage resharding' in prometheus logs such as:

{"caller":"dedupe.go:111","component":"remote","from":23,"level":"info","msg":"Remote storage resharding","queue":"0:https://influxdb:8086/api/v1/prom/write?db=prometheus\u0026u=prometheus\u0026p=STRONG_PASSWORD","to":1,"ts":"2019-06-05T11:01:03.109Z"}

Why it happens? Can it cause problems?

[Chris Marchbanks]

Hello,

Prometheus reshards the remote write queues in order to scale up the parallelism of remote write automatically. There is an open issue around Prometheus resharding far too often: https://github.com/prometheus/prometheus/issues/5627

Resharding too often should not cause major problems, but it will slow down your remote write throughput because Prometheus will wait for all pending samples to be sent before starting up the new shards.

Hope this helps,

Chris

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4aa2d1d9-3a18-46df-9f84-f95c44438b6b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[wats...@gmail.com]

Thank you for answer :)