'optional' labels?

12 views
Skip to first unread message

John Dexter

unread,
Oct 26, 2020, 10:36:55 AM10/26/20
to promethe...@googlegroups.com
I have a use-case where we monitor multiple, very different systems through a variety of protocols but one common thing is each sends updates to our server and most allow us to send messages to them too.

This seems a perfect example of labels... subsystem_message_total[subsystem-name,direction] for instance.

But one of our systems has 2 components, the others don't. I'm unsure what happens if this subsystem uses a 3rd label [component] and the others do not. Is this bad practice? Will it cause problems in storing/querying data?
Subsystems could each have their own metric name instead of using labels, but that sounds pretty ugly too.

What is the best approach here?

Thanks. 

Tim Schwenke

unread,
Oct 26, 2020, 12:19:44 PM10/26/20
to Prometheus Users
Having metrics with different label compositions seems fine to me. In the end a metric is nothing more than a set of time series with the same value for the label `__name__`. So there is no difference for the storage engine. 

I would just ensure that your data stays aggregatable. What do I mean by this? Let's take the following example:

```
 subsystem_message_total{name="foo", direction="in"}=100
 subsystem_message_total{name="foo", direction="out"}=100
 subsystem_message_total{name="bar"}=100
```

The query `sum(subsystem_message_total)` returns `300` which is correct. But if it looks like this:

```
 subsystem_message_total{name="foo", direction="in"}=100
 subsystem_message_total{name="foo", direction="out"}=100
 subsystem_message_total{name="foo"}=200  
 subsystem_message_total{name="bar"}=100
```

The query  `sum(subsystem_message_total)` will return `500`. So to work around this you would have to always remember to only include series with xor without a `direction` label.
Reply all
Reply to author
Forward
0 new messages