Prometheus jobs are not showing in prometheus UI

[Venkatraman Natarajan]

Hi Team,

We have 100+ prometheus jobs in prometheus UI but not showing all the jobs. 

The below are sample 2 jobs. In this, first job is not showing in UI but second job metrics showing fine.  

- job_name: 'blackbox-fundconnect_retail'

  scrape_interval: 1020s

  metrics_path: /probe

  honor_timestamps: true

  params:

    module: [http_2xx]

  file_sd_configs:

    - files: ['./dynamic/blackbox/blackbox_retail_FundConnect_targets.yml']

  relabel_configs:

  - source_labels: [__address__]

    separator: ;

    regex: (.*)

    target_label: __param_target

    replacement: $1

    action: replace

  - source_labels: [__param_target]

    separator: ;

    regex: (.*)

    target_label: instance

    replacement: $1

    action: replace

  - separator: ;

    regex: (.*)

    target_label: __address__

    replacement: {{ XXX }}:9122

    action: replace

- job_name: 'blackbox-fundconnect_retail_03'

  scrape_interval: 1020s

  metrics_path: /probe

  honor_timestamps: true

  params:

    module: [http_2xx]

  file_sd_configs:

    - files: ['./dynamic/blackbox/blackbox_retail_FundConnectHealth-03_targets.yml']

  relabel_configs:

  - source_labels: [__address__]

    separator: ;

    regex: (.*)

    target_label: __param_target

    replacement: $1

    action: replace

  - source_labels: [__param_target]

    separator: ;

    regex: (.*)

    target_label: instance

    replacement: $1

    action: replace

  - separator: ;

    regex: (.*)

    target_label: __address__

    replacement: {{ XXXXX}}:9122

    action: replace

Do we have limitations in prometheus jobs.? 

I would like to add more prometheus jobs to scrape the metrics.

Please find attached screenshot which shows jobs in prometheus UI

Prometheus version: prometheus:v2.27.1

Could you please help me on this?

Thanks,

Venkatraman N

[Brian Candler]

Sorry, I don't know what you mean by "not showing all the jobs".  You have only shown a small portion of the targets page.  Are you saying that job blackbox-fundconnect_retail does not appear there, but blackbox-fundconnect_retail_03 does? Is it possible that blackbox-fundconnect_retail has zero targets configured?

The PromQL query "up" will show you all the targets. "count by (job) (up)" will show you the jobs, with the number of targets for each. Those won't show a job with zero targets though.

Note that in the above case, both scrape jobs look to be identical, in which case you could have one job:

  file_sd_configs:

    - files:

- './dynamic/blackbox/blackbox_retail_FundConnect_targets.yml'

- './dynamic/blackbox/blackbox_retail_FundConnectHealth-03_targets.yml'

(You can use target labels to distinguish them, if you wish)

> Do we have limitations in prometheus jobs.? 

No.

> Prometheus version: prometheus:v2.27.1

That's pretty old (May 2021). Current LTS version is v2.45.2

[Venkatraman Natarajan]

Hi Brian,

Thanks for the response.

Are you saying that job blackbox-fundconnect_retail does not appear there, but blackbox-fundconnect_retail_03 does? 

Yes, Correct. We have 100 jobs in prometheus. Now we have added 5 jobs additionally, in that it is showing only 4 jobs. 

blackbox-fundconnect_retail_03,

blackbox-fundconnect_retail_04

blackbox-fundconnect_retail_05

blackbox-fundconnect_retail_06

blackbox-fundconnect_retail - This one is not showing in prometheus UI.

Is it possible that blackbox-fundconnect_retail has zero targets configured?

No, It has 48 targets configured. 

Note: We need to configure different scrape intervals for those jobs that's why configuring different jobs instead of a single job.

Thanks,

Venkatraman N

[Brian Candler]

I can't see the image link you posted: it gives a 404.

There's probably some config error in your prometheus.yml or your targets file, but seeing only snippets means I can't spot them.

Try these commands (adjust paths as necessary):

/path/to/promtool check service-discovery /path/to/prometheus.yml blackbox-fundconnect_retail

/path/to/promtool check config /path/to/prometheus.yml

And what does the query

up{job="blackbox-fundconnect_retail"}

show?

Apart from that, I don't think there's much more I can do to help you. There's a problem with the configs on your system that you'll need to diagnose locally.

[Venkatraman Natarajan]

Hi Brian,

Thanks for commands for troubleshooting. Finally, single quotes are not added in some of the targets. I have added it and works fine.

These commands will be helpful in troubleshooting in future. 

Regards, 

Venkatraman N

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c5aa0bb1-e86e-4842-90d2-e1aab4b76d18n%40googlegroups.com.