Incorrect Job Name in metrics prometheus_target_interval_length_seconds

[Ponson T]

Hello Everyone,

Can anyone help me with this issue?

What did you do?

1. Deployed Prometheus with default configuration
2. Deployed Node Exporter
3. Updated prometheus.yml configuration file and added node_exporter scrape job with 10s interval and set prometheus scrape job to 12s interval

Open prometheus web console http://localhost:9090/graph and browse metrics prometheus_target_interval_length_seconds

What did you expect to see?
It should have returned following time series
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="node_exporter",quantile="0.01"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="node_exporter",quantile="0.05"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="node_exporter",quantile="0.5"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="node_exporter",quantile="0.9"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="node_exporter",quantile="0.99"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.01"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.05"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.5"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.9"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.99"}

What did you see instead? Under which circumstances?
It returns following time series. Every job label value is stamped as prometheus.
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="prometheus",quantile="0.01"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="prometheus",quantile="0.05"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="prometheus",quantile="0.5"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="prometheus",quantile="0.9"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="prometheus",quantile="0.99"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.01"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.05"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.5"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.9"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.99"}

System information:

Linux 5.4.0-52-generic x86_64

OS-Release:

cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Prometheus version:

prometheus, version 2.22.0 (branch: HEAD, revision: 0a7fdd3)
build user: root@6321101b2c50
build date: 20201015-12:29:59
go version: go1.15.3
platform: linux/amd64

Prometheus Configuration file:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    scrape_interval: 12s
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    scrape_interval: 10s
    static_configs:
    - targets: ['localhost:9100', 'localhost:8081']
      labels:
        group: 'production'
    - targets: ['localhost:8082']
      labels:
        group: 'canary'

[Ponson T]

logged the following github issue earlier but it was closed recommending me to start the discussion in the mailing list -

https://github.com/prometheus/prometheus/issues/8152

[Brian Brazil]

The prometheus_target_interval_length_seconds metric does not have a job label, this is coming from your scrape config.

Brian

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a0a3fa33-0ba6-4ddb-b67c-22b942f14f6dn%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

[Brian Candler]

On Thursday, 5 November 2020 16:18:08 UTC, Ponson T wrote:

What did you expect to see?
It should have returned following time series
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="node_exporter",quantile="0.01"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="node_exporter",quantile="0.05"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="node_exporter",quantile="0.5"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="node_exporter",quantile="0.9"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="node_exporter",quantile="0.99"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.01"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.05"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.5"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.9"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.99"}

No.

prometheus_target_interval_length_seconds is not a metric returned by node_exporter.  It's a metric returned by prometheus (as you can see by instance="localhost:9090").  You scraped localhost:9090/metrics in the "prometheus" job, and so the label job="prometheus" is added to these metrics.

 

What did you see instead? Under which circumstances?
It returns following time series. Every job label value is stamped as prometheus.
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="prometheus",quantile="0.01"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="prometheus",quantile="0.05"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="prometheus",quantile="0.5"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="prometheus",quantile="0.9"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="10s",job="prometheus",quantile="0.99"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.01"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.05"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.5"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.9"}
prometheus_target_interval_length_seconds{instance="localhost:9090",interval="12s",job="prometheus",quantile="0.99"}

That is correct, as explained above.