"service account unauthorized" when logging into Vault using Kubernetes Auth method

shub...@gmail.com

unread,

Jan 4, 2019, 4:40:03 PM1/4/19

to Vault

I am trying to get the Kubernetes Auth method working with Vault.

Here are the steps I did.

Apply the CRB and SAs.

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

namespace: default

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

subjects:

- kind: ServiceAccount

namespace: default

---

apiVersion: v1

kind: ServiceAccount

metadata:

---

apiVersion: v1

kind: ServiceAccount

metadata:

Get the vault-tokenreview jwt, and configure the kubernetes auth (already enabled the kubernetes auth method)

VAULT_TOKENREVIEW=$(kubectl get secret $(kubectl get serviceaccount vault-tokenreview -o jsonpath={.secrets[0].name}) -o jsonpath={.data.token} | base64 --decode -)

vault write auth/kubernetes/config \

token_reviewer_jwt=$VAULT_TOKENREVIEW \

kubernetes_host=https://<k8s-host>:6443 \

kubernetes_ca_cert=@cert-chain.pem

Create a policy file and apply it:

policy.hcl:

path "secret/data/demo" {

capabilities = ["read"]

}

vault write sys/policy/demo-policy policy=@policy.hcl

Create the role

vault write auth/kubernetes/role/demo-role \

    bound_service_account_names=vault-auth \

    bound_service_account_namespaces=default \

    policies=demo-policy \

    ttl=1h

Add the service account info to a deployment

serviceAccount: vault-auth

serviceAccountName: vault-auth

Exec into the container, and try to log in

# KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

# curl --insecure --request POST --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "demo-role"}' https://vault-dev:8200/v1/auth/kubernetes/login

{"errors":["lookup failed: service account unauthorized; this could mean it has been deleted"]}

I have followed several different guides, and they all return the JSON response that has the token, but for some reason, mine does not work.

I found https://github.com/hashicorp/vault-plugin-auth-kubernetes/blob/master/token_review.go, where the error is displayed, and it has the comment:

case kubeerrors.IsUnauthorized(err):

// If the err is unauthorized that means the token has since been deleted

return nil, errors.New("lookup failed: service account unauthorized; this could mean it has been deleted")

Why would the token have been deleted? I verified that the KUBE_TOKEN value is the same as what I get if I inspect the SA secret.

Thank you in advance for any assistance.

Vasilev Vjacheslav

unread,

Jan 5, 2019, 12:29:23 PM1/5/19

to Vault

Hello,

The last line with curl command, are you sure that KUBE_TOKEN env. variable have been expanded to actual value?

shub...@gmail.com

unread,

Jan 7, 2019, 11:17:47 AM1/7/19

to Vault

Yes, they are the same.

# kubectl get secret $(kubectl get serviceaccount vault-auth -o jsonpath={.secrets[0].name}) -o jsonpath={.data.token} | base64 --decode -

eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LWF1dGgtdG9rZW4tZ2xsbGMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtYXV0aCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjlhNmZlNDE5LTBmYjMtMTFlOS1hNTc3LTAwNTA1Njk4MmFhNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnZhdWx0LWF1dGgifQ.pXG5zp9b0hSAph_vhwNxHVbSfwPsQ73HV9dIac3QwMHOHyxOhDXwxdjJbni7qXqL7X0K8KmkYD_19-vjebuu8dyIF03JJyh1IQMOlDJODecAnefbUAFl71mgeeD64HFXa1Pth_s_Pq7R65c0LfG_alVVjqFaPMUUpEkGhjf0Bv3U1venKsiqYoP2qNxJ8OEg9iLNt9Vxth9YICmasV9h9QE8aLaTKbKxt9xEIoUJz6Bag4TCkQQUIP-OMSy6BGtvYwI-SpNca11YQd-hoYf6OlCnmrCAWvVQu1WYKwvEj7LFAugL9g7JuvX_qoWRV7xyEKtRfw2q5-WDw9SL9cEmsQ

The token in the container:

# cat /var/run/secrets/kubernetes.io/serviceaccount/token

eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LWF1dGgtdG9rZW4tZ2xsbGMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtYXV0aCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjlhNmZlNDE5LTBmYjMtMTFlOS1hNTc3LTAwNTA1Njk4MmFhNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnZhdWx0LWF1dGgifQ.pXG5zp9b0hSAph_vhwNxHVbSfwPsQ73HV9dIac3QwMHOHyxOhDXwxdjJbni7qXqL7X0K8KmkYD_19-vjebuu8dyIF03JJyh1IQMOlDJODecAnefbUAFl71mgeeD64HFXa1Pth_s_Pq7R65c0LfG_alVVjqFaPMUUpEkGhjf0Bv3U1venKsiqYoP2qNxJ8OEg9iLNt9Vxth9YICmasV9h9QE8aLaTKbKxt9xEIoUJz6Bag4TCkQQUIP-OMSy6BGtvYwI-SpNca11YQd-hoYf6OlCnmrCAWvVQu1WYKwvEj7LFAugL9g7JuvX_qoWRV7xyEKtRfw2q5-WDw9SL9cEmsQ

If I create another pod and assign it a different service account, I get a different method when it tries to auth:

{"errors":["service account name not authorized"]}

Vasilev Vjacheslav

unread,

Jan 7, 2019, 11:22:38 AM1/7/19

to Vault

Please check whether environment variable got expanded in your curl command. For example '$KUBE_TOKEN' (single quotes) won't expand to a value, but "$KUBE_TOKEN" (double quotes) does.

shub...@gmail.com

unread,

Jan 7, 2019, 12:12:44 PM1/7/19

to Vault

If I use just "$KUBE_TOKEN", I get a different error:

# curl --insecure --request POST --data '{"jwt": "$KUBE_TOKEN", "role": "demo-role"}' https://vault-dev:8200/v1/auth/kubernetes/login

{"errors":["not a compact JWS"]}

I tried pasting the exact string into the command, and I get the same error:

# curl --insecure --request POST --data '{"jwt": "eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LWF1dGgtdG9rZW4tZ2xsbGMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtYXV0aCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjlhNmZlNDE5LTBmYjMtMTFlOS1hNTc3LTAwNTA1Njk4MmFhNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnZhdWx0LWF1dGgifQ.pXG5zp9b0hSAph_vhwNxHVbSfwPsQ73HV9dIac3QwMHOHyxOhDXwxdjJbni7qXqL7X0K8KmkYD_19-vjebuu8dyIF03JJyh1IQMOlDJODecAnefbUAFl71mgeeD64HFXa1Pth_s_Pq7R65c0LfG_alVVjqFaPMUUpEkGhjf0Bv3U1venKsiqYoP2qNxJ8OEg9iLNt9Vxth9YICmasV9h9QE8aLaTKbKxt9xEIoUJz6Bag4TCkQQUIP-OMSy6BGtvYwI-SpNca11YQd-hoYf6OlCnmrCAWvVQu1WYKwvEj7LFAugL9g7JuvX_qoWRV7xyEKtRfw2q5-WDw9SL9cEmsQ", "role": "demo-role"}' https://vault-dev:8200/v1/auth/kubernetes/login

{"errors":["lookup failed: service account unauthorized; this could mean it has been deleted"]}

shub...@gmail.com

unread,

Jan 10, 2019, 2:59:02 PM1/10/19

to Vault

So, in doing more debugging, I had a thought. The cluster that I have been using is one that I created using Rancher's RKE tool, and it has been imported into a Rancher server. Rancher installs RBAC so that it can control the access to each cluster.

As a test, I created a standalone Kubernetes node using minikube, and a standalone Vault dev instance, both on my laptop, and the Vault Kubernetes auth work fine. So the problem seems to be with the Rancher setup.

I looked at all of the ClusterRoleBindings, and I found that Rancher creates one of its own:

Name: metrics-server:system:auth-delegator

Labels: <none>

Annotations: kubectl.kubernetes.io/last-applied-configuration:

{"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"metrics-server:system:a...

Role:

Kind: ClusterRole

Name: system:auth-delegator

Subjects:

Kind Name Namespace

---- ---- ---------

ServiceAccount metrics-server kube-system

So now, the question is: Can you have multiple ClusterRoleBindings both with the role of system:auth-delegator? The Rancher-created one is in the kube-system namespace, and the one I created is in the default namespace.

shub...@gmail.com

unread,

Jan 16, 2019, 11:31:01 AM1/16/19

to Vault

In further troubleshooting, I found that the problem appears to be with Rancher. Rancher does RBAC, and they appear to be intercepting the Kubernetes API calls and it does not know about the service account. If I point Vault directly to the Kubernetes hosts API, then it works.

Arun Prasath

unread,

Dec 19, 2019, 2:37:31 PM12/19/19

to Vault

Hello everyone,

I am trying to create the approle using kubernetes auth method for openshift cluster in minishift.

I am facing "service account name not authorized" when login to kubernetes auth method. My cluster is Openshift cluster.

It is clear that I am missing some permissions/role issues, but do not know how to fix this.

In the openshift side, executed the below commands to get the service accounts.

oc login -u system:admin
oc new-project vault-demo
oc projects
oc create sa vault-auth# Create Cluster role binding for vault-auth
oc adm policy add-cluster-role-to-user \
  system:auth-delegator system:serviceaccount:vault-demo:vault-authoc serviceaccounts get-token vault-auth > reviewer_sa_jwt.txt# Lets create two more serviceaccounts for applications
oc create sa app1
oc create sa app2

desktop-e470:~/hashitvault$ curl --request POST --data "{\"jwt\": \"eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJ2YXVsdC1kZW1vIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LWF1dGgtdG9rZW4taHd4NjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtYXV0aCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjFjNDIxNWQyLTIyN2MtMTFlYS05YjZmLTUyNTQwMDk4YWMzOSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDp2YXVsdC1kZW1vOnZhdWx0LWF1dGgifQ.XzvbWRi2DUKnNzYoZYyJfKqHgQdxv8jg_75nHhmqTHAiYuz4-ABaqJokUTlrQGwsvw41V4rqEmc0CVF3MK_jgyUZzmpGnCNMySkRyYQw9TChhHUmOQDH9AKj6OOFcmAV811sQu9-qvVav4QlJPIW4cm6dHe-XHSNxuzqJ7OWScezqVDYaiWXBkcFpzEEisV6puXA7o5Npg-so2u0lW9bGEe9UP363ZyR3AYZ_rlZoRB-Gq7exGlN2TII0xUZDaBwbf9vDE_i3Zs_HFdNSBGsVFsG3-Xlw_iUTPTGTehDkSX7koYTT8GzjS9KR94TMVZdPLGH6txF4QfaRnWAvKgvOg\", \"role\": \"app1-role\"}" $VAULT_ADDR/v1/auth/ocp/login | jq

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 945 100 51 100 894 408 7152 --:--:-- --:--:-- --:--:-- 7560

{

"errors": [

"service account name not authorized"

]

}

How to fix this error ?? please suggest.

Message has been deleted

Vasilyev Viacheslav

unread,

Dec 19, 2019, 3:04:00 PM12/19/19

to Vault

It is not clear where you've got this long jwt token from, a reviewer token must be provided as token_reviewer_jwt parameter when you've been configuring k8s secrets mount, and jwt token for the login command must be extracted from /var/run/ secrets/kubernetes.io/serviceaccount/token file inside a pod

Arun Prasath

unread,

Dec 19, 2019, 3:41:43 PM12/19/19

to Vault

Hi Vasilyev,

Thanks for checking the issue.

I followed the below steps to get the reviewer token from the service account.

# Create Cluster role binding for vault-auth

oc adm policy add-cluster-role-to-user \

system:auth-delegator system:serviceaccount:vault-demo:vault-auth

oc serviceaccounts get-token vault-auth > reviewer_sa_jwt.txt

reviewer_jwt="$(cat reviewer_sa_jwt.txt)"

I followed the blog https://medium.com/hashicorp-engineering/vault-kubernetes-auth-method-for-openshift-9b9155590a6d?

I have not yet used the service account in the deployment spec, but before that I wanted to test the connection by logging in to the kubernets auth method from comamnd line which is failing.

I executed below commands in the Openshift side and in vault side. What am I missing in the below steps or doing wrong?

# Create OC project and token reviewer JWT:

oc login -u system:admin

oc new-project vault-demo

oc projects

oc create sa vault-auth

# Create Cluster role binding for vault-auth

oc adm policy add-cluster-role-to-user \

system:auth-delegator system:serviceaccount:vault-demo:vault-auth

oc serviceaccounts get-token vault-auth > reviewer_sa_jwt.txt

# Lets create two more serviceaccounts for applications

oc create sa app1

oc create sa app2

# Create Cluster role binding for vault-auth

oc adm policy add-cluster-role-to-user \

system:auth-delegator system:serviceaccount:vault-demo:vault-auth

oc serviceaccounts get-token vault-auth > reviewer_sa_jwt.txt

reviewer_jwt="$(cat reviewer_sa_jwt.txt)"

------------------------

desktop-e470:hashitvault# cat app1-policy.hcl

path "secret/app1" {