k8s service discovery is failing
Steve
Hi
I have been struggling with a RBAC issue and I cannot figure it out.
Help please!
I have node exporter running in my cluster.
As you know, it is a deamonSet and there is a node_exporter pod running on each node.
I also have a Prometheus server also running in the same namespace as the node_exporter deamonSet i.e. the default namespace.
The scrape job for node _exporter is using a SD configuration for pods as follows:
- job_name: prometheus_node_exporter
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
kubernetes_sd_configs:
- role: pod
...
If I setup my Prometheus Server to use a cluster role, the node_exporter targets are properly discovered. So far so good!
Now if I try to reduce the Prometheus Server to use a role instead, then it does not work.
As far as I know if the role includes listing any pods within the same namespace of the Prometheus Server service account, then the API server should grant access.
However, this is not the case. This is the log message I get from Prometheus Server:
level=error ts=2020-03-25T13:57:53.652Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:default:prometheus-server\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
Below is role I used for the Prometheus Server service account:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
creationTimestamp: "2020-03-25T13:40:13Z"
labels:
app: prometheus
component: server
heritage: Helm
release: my-server
name: prometheus-server
namespace: default
resourceVersion: "1943"
selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/default/roles/prometheus-server
uid: 28d3c869-894d-4797-9146-6137f60c7232
rules:
- apiGroups:
- ""
resources:
- pods
- configmaps
verbs:
- get
- list
- watch
Below is the role binding I used for Prometheus Server service account:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
creationTimestamp: "2020-03-25T13:40:13Z"
labels:
app: prometheus
chart: prometheus-10.5.1-steve-server-12
component: server
heritage: Helm
release: my-server
name: prometheus-server
namespace: default
resourceVersion: "1946"
selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/default/rolebindings/prometheus-server
uid: d581c497-52d6-4080-8ade-e33008c019fd
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: prometheus-server
subjects:
- kind: ServiceAccount
name: prometheus-server
namespace: default
Thank you!
Regards
Steve B
Steve
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2b7bd5cf-4fb8-4b5c-991a-f755aaf86106%40googlegroups.com.