Selectors in kubernetes_sd_configs don't restrict services based on labels

[Shubham Shrivastav]

Hello,

I'm trying to scrape all services that have labels app: core. However, after adding the below section to kubernetes_sd_configs, I'm not getting limited services in the Prometheus targets list. In fact, the configuration file does not save saying not able to parse the YAML config file.

After I remove the below section it works fine. I'm placing it in the config file as mentioned below:

kubernetes_sd_configs:

- role: service

namespaces:

names:

- ns-1

selectors:

- role: service

label: "app=core"

Can anyone help me out with this?

[Shubham Shrivastav]

Guyys?

[b.ca...@pobox.com]

> In fact, the configuration file does not save saying not able to parse the YAML config file.

1. Can you show the actual YAML you are using, with correct formatting?  The example you posted is invalid because it doesn't have the spaces to perform YAML alignment.

2. Can you show the actual error message?

[Shubham Shrivastav]

CONFIG FILE:

    ---

# my global config

global:

scrape_interval: 15s # By default, scrape targets every 15 seconds.

evaluation_interval: 15s # By default, scrape targets every 15 seconds.

# scrape_timeout is set to the global default (10s).

# Attach these labels to any time series or alerts when communicating with

# external systems (federation, remote storage, Alertmanager).

external_labels:

monitor: 'Vocera'

# Load and evaluate rules in this file every 'evaluation_interval' seconds.

rule_files:

- /etc/prometheus/alertmanagerconf.yml

# - "first.rules"

# - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

- job_name: 'Node-FileDiscovery'

file_sd_configs:

- files:

- 'target/target.json'

- job_name: 'xxxxxxx-xxxxxx-poc-jmx'

static_configs:

- targets:

- '192.168.43.3:9876'

labels:

tag_product: xxxxxx-tooling

tag_environment: dev

tag_role: xxxx

tag_platform: linux

instance: xxxxxxxxxx

- targets :

- '192.168.43.5:9876'

labels:

tag_product: xxxxxx-tooling

tag_environment: dev

tag_role: xxxxx

tag_platform: linux

instance: xxxxx

- job_name: 'xxxxx-Microservices'

scrape_interval: 5s

metrics_path: /api/prometheus

scheme: https

tls_config:

insecure_skip_verify: true

kubernetes_sd_configs:

- role: service

namespaces:

names:

- xxxxx-xxxxx-dev

selectors:

- role: service

label: "app=core"

relabel_configs:

- source_labels: [__address__]

target_label: __param_target

replacement: $1

action: replace

- source_labels: [__param_target]

target_label: instance

replacement: $1

action: replace

- source_labels: [__meta_kubernetes_namespace]

target_label: kubernetes_namespace

replacement: $1

action: replace

- source_labels: [__meta_kubernetes_service_name]

target_label: kubernetes_name

replacement: $1

action: replace

- source_labels: [__meta_kubernetes_service_label_app]

target_label: app

replacement: $1

action: replace

alerting:

alertmanagers:

- static_configs:

- targets:

- alertmanager:9093

ERROR:

level=error ts=2020-11-30T03:40:47.020Z caller=main.go:723 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n  line 54: field selectors not found in type kubernetes.plain"

[b.ca...@pobox.com]

The YAML formatting is still wrong.  If you can't paste it accurately into Google Groups, try using a service like pastebin to give an accurate copy of your YAML file.

[Shubham Shrivastav]

Sure,

https://pastebin.com/0jHphwpJ

[b.ca...@pobox.com]

What version of prometheus are you running?  What do you get from "promtool check config /etc/prometheus/prometheus.yml" ?

I tried adding this snippet of your config to my prometheus.yml:

  - job_name: 'xxxxxx-Microservices'

    scrape_interval: 5s

    metrics_path: /api/prometheus

    scheme: https

    tls_config:

      insecure_skip_verify: true

    kubernetes_sd_configs:

    - role: service

      namespaces:

        names:

        - operations-tools-dev

      selectors:

        - role: service

          label: "app=core"

and it was accepted without issue.

I also tried putting your entire config into a file, and the only change I had to made was to point rule_files to a different filename.  After that it parsed just fine:

# /opt/prometheus/promtool check config /etc/prometheus/z.yml

Checking /etc/prometheus/z.yml

  WARNING: file "/etc/prometheus/target/target.json" for file_sd in scrape job "Node-FileDiscovery" does not exist

  SUCCESS: 0 rule files found

# 

This is with version 2.22.2.

[Shubham Shrivastav]

Mine is  2.10.0. That probably is the problem let me upgrade it

[Julien Pivotto]

On 30 Nov 11:46, Shubham Shrivastav wrote:
> Mine is 2.10.0. That probably is the problem let me upgrade it

That feature was added in 2.17.0. I recommend you take Prometheus 2.23.0.

> >>>>>>> I'm trying to scrape all services that have labels *app: core. *However,

> >>>>>>> after adding the below section to kubernetes_sd_configs, I'm not getting
> >>>>>>> limited services in the Prometheus targets list. In fact, the configuration
> >>>>>>> file does not save saying not able to parse the YAML config file.
> >>>>>>>
> >>>>>>> After I remove the below section it works fine. I'm placing it in
> >>>>>>> the config file as mentioned below:
> >>>>>>>

> >>>>>>> *kubernetes_sd_configs:*
> >>>>>>> * - role: service*
> >>>>>>> * namespaces:*
> >>>>>>> * names:*
> >>>>>>> * - ns-1*
> >>>>>>> *selectors:*
> >>>>>>> * - role: service*
> >>>>>>> * label: "app=core"*

> >>>>>>>
> >>>>>>> Can anyone help me out with this?
> >>>>>>>
> >>>>>>
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f31dc296-096c-4a04-b32a-ed0efe5b110cn%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

[Shubham Shrivastav]

2.23 worked for me thank you soo much guys