same port/exporter, but different job labels

[mcamm...@gmail.com]

Hi Everyone,

We currently use an external tool to query our list of targets from an internal database. We then use file based service discovery in prometheus and provide it the list of targets in a .json file. Works nicely. This is for the node_exporter job.

- job_name: 'se-linux-node-exporter'
  file_sd_configs:
  - files:
    - 'se-linux-node-exporter.json'

This is 100% of the targets we care about and want to see in our Grafana dashboards.

We now have a new external search we can run on our internal database to pull servers based on the service they provide. This will result in a subset of servers from the above scrape. I would like to add this as a separate job, like

-job_name: 'se-linux-XYZ-service'

  file-_sd_configs:

  -files:

    - 'se-linux-XYZ-service.json'

My concern is that since this is still for node_exporter port 9100 for both jobs, I'll get duplicate scrapes ingested. 

The end goal is to be able to group servers by service in Grafana - using the job label. What is the proper way to go about ensuring we don't have duplicate scrapes/metrics but still maintain this grouping? I'm hesitant to add custom labeling/relabeling to the original scrape that has all of the hosts because it could start our graphs over in Grafana. We have about a two year history (using Victoria Metrics). It's also possible that any one server can be a part of more than one service - so we could have duplicates there too.

I'm thinking there's something I can set with relabeling (drop/keep) or with honor_labels, but I want to be sure I understand what I'm doing first.

[Stuart Clark]

If you have the same endpoints in multiple jobs that will work (as the
job label is different so there aren't duplicates from a time series
perspective) but it isn't very useful, as you'd have duplicates from a
coverage perspective.

If you can have a target being part of multiple services a single label
isn't going to work. Instead you need a label for each service (with a
true/false type value) to be able to encode that, although it really
depends what you are trying to achieve overall.

Normally you'd never try to scrape the same target multiple times, so if
you have something listed in one job you'd make sure it isn't listed in
any others.

The normal way to handle metadata like service, etc. would either be to
add those labels within the job config (often done when that info comes
from places like Kubernetes labels) or by having separate metrics that
you can join into your queries (so something which has something like an
instance label to allow joining but then adds in the service labels).
https://www.robustperception.io/how-to-have-labels-for-machine-roles

--
Stuart Clark

[Brian Candler]

> My concern is that since this is still for node_exporter port 9100 for both jobs, I'll get duplicate scrapes ingested. 

You will, but it's not really a problem, because they will have different labels {job="se-linux-node-exporter"} and {"job=se-linux-XYZ-service"} and therefore will be stored in different timeseries.

It's a waste of resources, but otherwise isn't an issue - although you have to be careful that if you're doing aggregation queries and you don't match on the correct job label, you may count the same host twice.

It would be more efficient if you could categorise your servers by adding labels, rather than duplicating the scrapes.  For example, in your se-linux-node-exporter.json you could have

- labels:

    category: foo

  targets:

    - host1

    - host2

    - host3

- labels:
    category: bar

  targets:
    - host4

    - host5

(I'm showing JSON rather than YAML as it's easier to type :-)  Then all hosts will be scraped exactly once, but some will have {category="foo"} and some {category="bar"} as extra labels.  This is a pretty simple approach.  You can add as few or many labels to each group of hosts as you like.

The other way you can handle it is as described here:

https://www.robustperception.io/how-to-have-labels-for-machine-roles
https://www.robustperception.io/exposing-the-software-version-to-prometheus

In this approach, you create a completely separate static timeseries for each host, giving whatever metadata you like:

meta{instance="host1",category="foo",rack="R1",owner="alice"} 1
meta{instance="host2",category="bar",rack="R1",owner="bob"} 1
...

However, using that metadata requires more complex PromQL queries, as you have to join between your main metric and your metadata metric.

https://prometheus.io/docs/prometheus/latest/querying/operators/#many-to-one-and-one-to-many-vector-matches

https://www.robustperception.io/left-joins-in-promql

[Brian Candler]

> (I'm showing JSON rather than YAML as it's easier to type :-)

Swap those. Of course.