kubernetes-cadvisor giving no token found #4038

[yaki...@gmail.com]

Hello,

I have a Kubernetes Cluster in Google Cloud trying install Prometheus on it. But I hit a strange issue. The kubernetes-cadvisor component is showing as down for all nodes. I am not sure what I am missing here or if it actually is a bug. Any suggestion would be great. You can find my yaml configs below :

Kubernetes Cluster in Google CLoud v1.9.3-gke.0

Here is my configmap configuration for it :

      - job_name: 'kubernetes-cadvisor'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}:4194/proxy/metrics/cadvisor

I did a lot of searching and tried many variations (with and without port on the replacement: field, with and without TLS, etc) but it seems like a bug unless I am missing something.

When I remove the port from - replacement: /api/v1/nodes/${1}:4194/proxy/metrics/cadvisor I get 404 not found. With the port I get token not found.

I've also used the follow RBAC configuration :

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  labels:
    app: prometheus
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
  labels:
    app: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring

And my deployment yaml :

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: monitoring
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
          image: prom/prometheus:v2.1.0
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf
        - name: prometheus-storage-volume
          emptyDir: {}

Service :

---

apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/scrape: 'true'
labels:
name: prometheus-service
kubernetes.io/name: "Prometheus"
name: prometheus-service
spec:
selector:
app: prometheus
type: LoadBalancer
ports:
- port: 8080
targetPort: 9090
protocol: TCP
loadBalancerSourceRanges:
- 0.0.0.0/32 (anonymized)
- 0.0.0.0/23 (anonymized)

[Simon Pasquier]

The log message tells you that the content returned by the cAdvisor endpoint [1] doesn't comply with the Prometheus exposition format.

You can get a copy of the cAdvisor output and use promtool to check where it breaks.

[1] https://kubernetes.default.svc:443/api/v1/nodes/gke-test-pool-2-849432b0-84p6:4194/proxy/metrics/cadvisor

On Wed, Apr 4, 2018 at 8:58 AM, <yaki...@gmail.com> wrote:

Also, here is the output of the logs from the POD itself, we get no token found there as well :

level=warn ts=2018-04-04T06:57:37.29590612Z caller=scrape.go:686 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://kubernetes.default.svc:443/api/v1/nodes/gke-test-pool-2-849432b0-84p6:4194/proxy/metrics/cadvisor msg="append failed" err="no token found"
level=warn ts=2018-04-04T06:57:37.574333424Z caller=scrape.go:686 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://kubernetes.default.svc:443/api/v1/nodes/gke-test-pool-2-849432b0-xs7g:4194/proxy/metrics/cadvisor msg="append failed" err="no token found"
level=warn ts=2018-04-04T06:57:37.589663004Z caller=scrape.go:686 component="scrape manager" scrape_pool=kubernetes-cadvisor target=https://kubernetes.default.svc:443/api/v1/nodes/gke-test-pool-2-849432b0-m0tb:4194/proxy/metrics/cadvisor msg="append failed" err="no token found"

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/48fe5f97-c002-4950-8f69-37f47adf8351%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.