Disable Authorization for /metrics endpoint

[Junaid Subhani]

I am currently running Prometheus inside a container in my Kubernetes cluster. My config states ::

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus
data:
  prometheus.yml: |-
    global:
      scrape_interval: 5s
    scrape_configs:


    - job_name: 'kubernetes_apiserver'
      tls_config:
        insecure_skip_verify: true
      kubernetes_sd_configs:
      - api_servers:
        - http://172.29.219.102:8080
        role: apiserver


      relabel_configs:
      - source_labels: [__meta_kubernetes_role]
        action: keep
        regex: (?:apiserver)


     ###################### Kubernetes Pods ##########################




    - job_name: 'haproxy'
      static_configs:
        - targets:
          - 172.29.219.110:9101


    - job_name: 'prometheus'
      static_configs:
         - targets:
           - 172.29.219.110:9090


    - job_name: 'kubernetes_pods'
      tls_config:
        insecure_skip_verify: true
      kubernetes_sd_configs:
      - api_servers:
        - http://172.29.219.102:8080
        role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

As you can see, I have disables tls authentication for now by setting "insecure_skip_verify: true"

Once I bring up my pods and their relevant microservices, I get the ERROR ::

server returned HTTP status 401 Unauthorized

Please note I am using annotation to change /metrics to /websocket/metrics.

Now one way to get get rid of this is using certificates which I know is the most secure solution but what I am wondering is if I can disable the Authorization feature  completely ONLY for /metrics ? Is there any such configuration ? 

[Ben Kochie]

The Authorization problem is from the Kubernetes side, Prometheus can't do anything about that.  It looks like you're missing the CA and bearer token file configs.  You can see how to use those in the example config.

https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/b46cf99f-3f95-489a-a2ad-0bff96323638%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johannes Ziemke]

Hi Junaid,

that's probably due to recent introduction of role based access control to kubernetes. You need to apply this manifest (and might need to adjust the selectors where needed): https://github.com/prometheus/prometheus/blob/master/documentation/examples/rbac-setup.yml

When I tried that though on GKE, I got a permission issue:

Error from server (Forbidden): error when creating "prometheus/prometheus-rbac-role.yml": clusterroles.rbac.authorization.k8s.io "prometheus" is forbidden: attempt to grant extra privileges ....

Took me almost two hours to figure that out: I needed this ClusterRoleBinding to grant my user full access:

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1beta1

metadata:

  name: full-access

  namespace: default

subjects:

- kind: User

  name: my-gcp-email-address

  apiGroup: rbac.authorization.k8s.io

roleRef:

  kind: ClusterRole

  name: cluster-admin

  apiGroup: rbac.authorization.k8s.io

On Wed, May 24, 2017 at 5:45 PM, Junaid Subhani <ijunaid...@gmail.com> wrote:

[Junaid Subhani]

Thanks for your suggestion Johannes.

I am trying to use this manifest but get the ERROR :

error: error validating "test.yaml": error validating data: API version "rbac.authorization.k8s.io/v1beta1" isn't supported, only supports API versions ["authorization.k8s.io/v1beta1" "certificates.k8s.io/v1alpha1" "imagepolicy.k8s.io/v1alpha1" "v1" "apps/v1alpha1" "autoscaling/v1" "batch/v1" "batch/v2alpha1" "rbac.authorization.k8s.io/v1alpha1" "authentication.k8s.io/v1beta1" "policy/v1alpha1" "storage.k8s.io/v1beta1" "federation/v1beta1" "extensions/v1beta1" "componentconfig/v1alpha1"]; if you choose to ignore these errors, turn validation off with --validate=false

Is this a feature in the latest version of Kubernetes ? 

To post to this group, send email to prometheus...@googlegroups.com.

[Johannes Ziemke]

Hi,

On Thu, May 25, 2017 at 2:47 PM, Junaid Subhani <ijunaid...@gmail.com> wrote:

Thanks for your suggestion Johannes.

I am trying to use this manifest but get the ERROR :

error: error validating "test.yaml": error validating data: API version "rbac.authorization.k8s.io/v1beta1" isn't supported, only supports API versions ["authorization.k8s.io/v1beta1" "certificates.k8s.io/v1alpha1" "imagepolicy.k8s.io/v1alpha1" "v1" "apps/v1alpha1" "autoscaling/v1" "batch/v1" "batch/v2alpha1" "rbac.authorization.k8s.io/v1alpha1" "authentication.k8s.io/v1beta1" "policy/v1alpha1" "storage.k8s.io/v1beta1" "federation/v1beta1" "extensions/v1beta1" "componentconfig/v1alpha1"]; if you choose to ignore these errors, turn validation off with --validate=false

Is this a feature in the latest version of Kubernetes ?

Yes, it's fairly new. But if it's not supported on your version, your permission problem might not be related to rbac after all. Did you specify a correct tls config like this?

https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L31