Prometheus metrics repeat, and cause the promql to be unavailable.

57 views
Skip to first unread message

sowhat7

unread,
Dec 10, 2020, 10:36:05 PM12/10/20
to Prometheus Users
I add a label for node 10.193.32.44, and I get repeat metrics from prometheus at a certain moment.
And then my promql is unavailable, because many to many are not allowed. 
1.png2.png3.png4.png 
prometheus version: 2.7.2
node-exporter version: 0.17.0  

Christian Hoffmann

unread,
Dec 11, 2020, 5:08:45 AM12/11/20
to promethe...@googlegroups.com
Hi,

On 2020-12-11 04:36, sowhat7 wrote:
> I add a label for node 10.193.32.44, and I get repeat metrics from
> prometheus at a certain moment.
Is this a permanent issue or does it affect the time around your change
only?
I suspect the latter. In that case, you could use some kind of
aggregation to work around that (e.g. topk(1, ...), min()). You could
also try to delete the overlapping data (probably plus some minutes due
to staleness) using the DELETE API.

If the former, then there might be some issue with service discovery and
you would need to share your relevant configs in order for someone to be
able to help.

> prometheus version: 2.7.2
> node-exporter version: 0.17.0
These versions sound really, really old. You should consider updating.

Kind regards,
Christian
Reply all
Reply to author
Forward
0 new messages