Discover from Kubernetes limited to a namespace

2,228 views
Skip to first unread message

L M

unread,
Sep 18, 2019, 10:51:42 AM9/18/19
to Prometheus Users
I'm trying to discover targets dynamically from kubernetes, but my logs are filled up with these types of messages:

```
level=error ts=2019-09-18T14:17:17.487Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:264: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:ude-837:helm-prometheus-dev-server\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2019-09-18T14:17:17.522Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:301: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:ude-837:helm-prometheus-dev-server\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2019-09-18T14:17:17.523Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:265: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:ude-837:helm-prometheus-dev-server\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2019-09-18T14:17:17.621Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:263: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:ude-837:helm-prometheus-dev-server\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
```  

I think I need to set the namespace so that it works correctly, but I get syntax errors when I try anything.  Here's my current configs more or less:


```
- job_name: 'kubernetes-service-endpoints'

        kubernetes_sd_configs:
          - role: endpoints

        # namespaces:
        #   names:
        #   - ude_837

        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
            action: replace
            target_label: __scheme__
            regex: (https?)
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
            action: replace
            target_label: __address__
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            action: replace
            target_label: kubernetes_name
          - source_labels: [__meta_kubernetes_pod_node_name]
            action: replace
            target_label: kubernetes_node
```

I was told by a good source that I should add that commented out namespace section into my configs, but it seems to cause yml parse errors so I'd appreciate any additional tips on this subject.  

Simon Pasquier

unread,
Sep 18, 2019, 11:36:18 AM9/18/19
to L M, Prometheus Users
This should work:

- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- ude_837
...

If not, you probably have an indent problem...
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/34d02006-678b-422c-a6eb-ded5244b8f0c%40googlegroups.com.

L M

unread,
Sep 18, 2019, 1:49:19 PM9/18/19
to Prometheus Users
Thank you, Simon using your indentation/ nesting fixed the problem entirely!  

I'm getting a followup error indicating I need to use a more priveledged service account to get access the the endpoints needed.  

> Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:ude-837:helm-prometheus-dev-server\" cannot list resource \"pods\" in API group \"\" in the namespace \"ude_837\”"

Do you know of a way to specify which service account and tokens to use for dynamic target discovery?


On Wednesday, September 18, 2019 at 10:36:18 AM UTC-5, Simon Pasquier wrote:
This should work:

- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - ude_837
...

If not, you probably have an indent problem...

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

Simon Pasquier

unread,
Sep 19, 2019, 8:16:16 AM9/19/19
to L M, Prometheus Users
You can specify bearer_token_file/bearer_token parameters in
kubernetes_sd_configs:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5084c24f-a655-41df-9ea0-1e80696e475c%40googlegroups.com.

L M

unread,
Sep 19, 2019, 10:55:37 AM9/19/19
to Prometheus Users
I had been to those docs but never scrolled down that far to find the example snippet, that's great stuff, thanks!


On Thursday, September 19, 2019 at 7:16:16 AM UTC-5, Simon Pasquier wrote:
You can specify bearer_token_file/bearer_token parameters in
kubernetes_sd_configs:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

Reply all
Reply to author
Forward
0 new messages