lookup failed: service account unauthorized; this could mean it has been deleted

[Buri]

Hi, I've been using Vault with Kubernetes auth plugin and today the process stopped working.

When trying to log in, i receive 

lookup failed: service account unauthorized; this could mean it has been deleted

What does this mean? That vault can no longer validate tokens? Or that the token of the authenticating service is no longer valid?

Any advice is much appreciated,

Buri

[Nick Cabatoff]

Hi Buri,

I haven't used the k8s auth plugin before, but I looked at that error in the code and it seems to occur when calling the API apis/authentication.k8s.io/v1/tokenreviews and getting back an "Unauthorized" error.  There is a comment there (https://github.com/hashicorp/vault-plugin-auth-kubernetes/blob/master/token_review.go):

// If the err is unauthorized that means the token has since been deleted return nil, errors.New("lookup failed: service account unauthorized; this could mean it has been deleted")

Can you try to kubectl describe the associated secret?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/a548201c-a8b2-4820-bfa3-cf9f402ff521%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Buri]

kubectl describe sa vault-auth     

Name:                vault-auth

Namespace:           default

Labels:              <none>

Annotations:         kubectl.kubernetes.io/last-applied-configuration:

                       {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"vault-auth","namespace":"default"}}

Image pull secrets:  <none>

Mountable secrets:   vault-auth-token-hz96v

Tokens:              vault-auth-token-hz96v

Events:              <none>

kubectl describe secret vault-auth-token-hz96v

Name:         vault-auth-token-hz96v

Namespace:    default

Labels:       <none>

Annotations:  kubernetes.io/service-account.name: vault-auth

              kubernetes.io/service-account.uid: b61ff1c8-5064-11e9-a721-0a0526458cc8

Type:  kubernetes.io/service-account-token

Data

====

token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LWF1dGgtdG9rZW4taHo5NnYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtYXV0aCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImI2MWZmMWM4LTUwNjQtMTFlOS1hNzIxLTBhMDUyNjQ1OGNjOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnZhdWx0LWF1dGgifQ.RXni__QdLNd7KzkaL51POxGl3IXX8Wtwon8DDM_1n98Jb0P2v0AyYmai4bbk4RmcASMBfkf7vBYXfB5jrychXinElhYUr7PXV_wd6EBLsqeUtps4vUzK5ovlQvoAoKeyO6GQLKmR4l2csB7_s0L42YnZPuqKfRfBXmdxEJA8W9BzswHYm1JZMn1LfqRMWX2yFFkDdWoXYw-YGyl8hhvKvmDTcMA3eR7A8Ht0gZn8ufObTAZ-ToKHxCE5Mhu8KNGVd49W2q51C0O5zNvwZFV9tuXLfIuNK6FQcOAB5X3ILbOPCFrpM9y0kNzeIBpPClfhH0w5rI4wuKLVAmco84xTnQ

ca.crt:     1025 bytes

namespace:  7 bytes

kubectl describe clusterrolebinding role-tokenreview-binding

Name:         role-tokenreview-binding

Labels:       <none>

Annotations:  kubectl.kubernetes.io/last-applied-configuration:

                {"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"role-tokenreview-bindin...

Role:

  Kind:  ClusterRole

  Name:  system:auth-delegator

Subjects:

  Kind            Name        Namespace

  ----            ----        ---------

  ServiceAccount  vault-auth  default

I found the same link, but wasn't much wiser regardless :(


Dne středa 27. března 2019 12:56:45 UTC+1 Nick Cabatoff napsal(a):

[Nick Cabatoff]

What are you providing when calling v1/auth/kubernetes/login?

Can I see the config shown by GET $VAULT_ADDR/v1/auth/kubernetes/config, and the role definition ($VAULT_ADDR/v1/auth/kubernetes/role/:name)?

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/d560d296-5d1f-481f-836e-a71fb72acc4a%40googlegroups.com.

[Buri]

Sure, here it goes:

GET /v1/auth/kubernetes/config

{

    "request_id": "65eeee44-e884-5336-674c-1b694fcefe9b",

    "lease_id": "",

    "renewable": false,

    "lease_duration": 0,

    "data": {

        "kubernetes_ca_cert": "-----BEGIN CERTIFICATE-----\nMIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl\ncm5ldGVzMB4XDTE5MDMxODA3MjIxNFoXDTI5MDMxNTA3MjIxNFowFTETMBEGA1UE\nAxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJoY\ntHfNAQfsbXklQWEn91sffDbLuvotfrh/vm7vbw5xoKVOxywQzJJsf+I97sEAr2yf\ntitvNuLaR40zTIVQ51Yn1O2k16njK6vmdjx81aT/vmqRHhhm6a9d539Z9DaKSRiG\nJ6PXqZp2vrJktUXT9dgdKui/GsPCjg3XZAtaatx174nOsJPQLSzSDnRVk/KREAt8\npfsw8DrZYljJslPro7JX1doIcnqt7nlE1l+JZiZorRpnP+z7PDSxx+B9Lf4uowff\nGRi3k4B+8C5Xu6JJfNWYgY0YLVScl4UlfOsgELbNrdQTmLt4F2D+cOsWn7aL8WNZ\nwrpCfrKNfO/cNd/QDYUCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB\n/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAD3gyD5G71abPK0iafSZ6jPBPvi/\nWzaPg0Yai8kbqv5I30Ai7j+5r8dXqrIo818bruOSEZuLJnBFpivff6pyhs5yX+qt\nrUR+oRiiXnKjAMB4N7sLKpvikYfHcHrePh8RD3BQoJRSCGT07Krch6yRhGuHwihD\n/4Q2uMIS3dI3Oa9C0d1MgTJl/kMzruVsM0oax/58VLjTCP3zv/r293HCN8OyJ3cX\ncUF5kOuCNZDUFp8IV3PK1CR1hyBx83RwDLrgKNHPQDUqORGH4BBrxbmlmsK9IH7/\nc2I5YRU+SuYk5QSKYlhTSbISROVC7xQXLiRrFOsqO1NA92LOWVK+8CUaV2o=\n-----END CERTIFICATE-----\n",

        "kubernetes_host": "https://kubernetes.default:443",

        "pem_keys": []

    },

    "wrap_info": null,

    "warnings": null,

    "auth": null

}

GET v1/auth/kubernetes/role/abc

{

    "errors": []

}

Dne středa 27. března 2019 13:43:30 UTC+1 Nick Cabatoff napsal(a):

[Nick Cabatoff]

I think you missed one of my questions: What are you providing when calling v1/auth/kubernetes/login?

And if 'abc' is not the role name that you're providing in your call to the login api, I'd like to see the output from v1/auth/kubernetes/role/:role for the role in question.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/8a6bb115-456e-479f-8dbb-9a683740ceab%40googlegroups.com.

[Buri]

Sorry, my bad

POST https://.../v1/auth/kubernetes/login

{ jwt:

  node-vault    'eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzbS1jb3JlLWF3cy1hcGktYXV0aGVudGljYXRpb24iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiZGVmYXVsdC10b2tlbi1obHYyNCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiZGIzNTcwMzktNGIxZi0xMWU5LTk5YWMtMDY3NDlkMTc4YzQwIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnNtLWNvcmUtYXdzLWFwaS1hdXRoZW50aWNhdGlvbjpkZWZhdWx0In0.gmAqH108iRRPHk8Ud0kaJShY0BEq7Xbkfh7eqMZZM_2Id-yLgs2D4HNupQmN0c3qHJi0ATAKj2GFFhv6qxoQTzhCZL6LYXbYPaF8O1IDCpJF8MqkwStZsC-2k9hTonx9cHUYDtxPtVvwzQHE2s1Sw4fy4axT0GER5iDHDWHlqajVdeo19QtG8J_A-Eq8Kv7GQNwO7LRYHzH_RvHTV8BC5XV2Rvb_MmaVsbdb8FCxtT6Ti3IpOJZiHTT5C0--cc-fccFXYbWIpmCkuvGeUldj9BnPKlZR6T3bqdsIF5lpL-x7jARLBWpewHFodBxvoqbCtLf7AXKGWdTkW-qUkbr-Qg',

  node-vault   role: 'sm-core-aws-api-authentication' }

Role definition

{"request_id":"c305cdc4-5fc1-0363-5878-8db8c13b717a","lease_id":"","renewable":false,"lease_duration":0,"data":{"bound_cidrs":[],"bound_service_account_names":["*"],"bound_service_account_namespaces":["sm-core-aws-api-authentication"],"max_ttl":0,"num_uses":0,"period":0,"policies":null,"ttl":3600},"wrap_info":null,"warnings":null,"auth":null}

Dne středa 27. března 2019 13:51:45 UTC+1 Nick Cabatoff napsal(a):

[Nick Cabatoff]

Again, I'm new to this plugin, but my reading suggests that the jwt value you provide in your call to the vault login api should be the token associated with the service account.  Going to https://jwt.io, we can see that the token you got from kubectl describe secret vault-auth-token-hz96v translates to:

{

  "iss": "kubernetes/serviceaccount",

  "kubernetes.io/serviceaccount/namespace": "default",

  "kubernetes.io/serviceaccount/secret.name": "vault-auth-token-hz96v",

  "kubernetes.io/serviceaccount/service-account.name": "vault-auth",

  "kubernetes.io/serviceaccount/service-account.uid": "b61ff1c8-5064-11e9-a721-0a0526458cc8",

  "sub": "system:serviceaccount:default:vault-auth"

}

Whereas the one you're posting to the login API is:

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "sm-core-aws-api-authentication",
  "kubernetes.io/serviceaccount/secret.name": "default-token-hlv24",
  "kubernetes.io/serviceaccount/service-account.name": "default",
  "kubernetes.io/serviceaccount/service-account.uid": "db357039-4b1f-11e9-99ac-06749d178c40",
  "sub": "system:serviceaccount:sm-core-aws-api-authentication:default"
}

Now that you've posted it publicly you should of course rotate the token anyway.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/d3f65352-30d6-4e7d-89d6-59be46518185%40googlegroups.com.

[Buri]

My understanding was that vault has to have role system:auth-delegator to verify other roles, meaning that the first serviceaccount belongs to the vault itself, but the second one belongs to the application that is trying to authenticate against the vault itself. Maybe I was understanding this wrong?

Dne středa 27. března 2019 14:51:10 UTC+1 Nick Cabatoff napsal(a):

[Nick Cabatoff]

That makes sense. In that case what do you get running:

  kubectl --namespace sm-core-aws-api-authentication describe sa default
  kubectl --namespace sm-core-aws-api-authentication describe secret default-token-hlv24

?

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/da1fae1f-0d85-4d6b-948d-50af72d91768%40googlegroups.com.

[Buri]

Sure, here it goes:

kubectl --namespace sm-core-aws-api-authentication describe sa default

Name:                default

Namespace:           sm-core-aws-api-authentication

Labels:              <none>

Annotations:         <none>

Image pull secrets:  <none>

Mountable secrets:   default-token-jh6vw

Tokens:              default-token-jh6vw

Events:              <none>

kubectl --namespace sm-core-aws-api-authentication describe secret default-token-jh6vw

Name:         default-token-jh6vw

Namespace:    sm-core-aws-api-authentication

Labels:       <none>

Annotations:  kubernetes.io/service-account.name: default

              kubernetes.io/service-account.uid: 2622c16c-5072-11e9-99ac-06749d178c40

Type:  kubernetes.io/service-account-token

Data

====

ca.crt:     1025 bytes

namespace:  30 bytes

token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzbS1jb3JlLWF3cy1hcGktYXV0aGVudGljYXRpb24iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiZGVmYXVsdC10b2tlbi1qaDZ2dyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMjYyMmMxNmMtNTA3Mi0xMWU5LTk5YWMtMDY3NDlkMTc4YzQwIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnNtLWNvcmUtYXdzLWFwaS1hdXRoZW50aWNhdGlvbjpkZWZhdWx0In0.l5mdTfHXO5-q0W8e5RD8nLpfj543uUmUaSk4-s8AKY1UDJS7Zw3wHFH4M-pUo1WVN2XhDXw1QFGr57yktpKMKzcg7_aXvJ1-Vw5IC2SSmwXozQJzvtShY-8fFSbSBo-pkL3loK3P4WXvRQcEtR6ZON06vJypmj-wiPuLCnaju04lg1YxnRexNGMp6_qMTpjR1rhFTiOecG28YzbD0EwXmkcZGQuG6KG6XYPyYuBlI5KtMKYuSZymBvKyTyWJ6AB3XQUXCvFdIf2k0fBB71ZqmkN1HkDmjWCiB7vF04mrA0ch2QGPsCtmR5DSRvN3HLZr95JhW_J8Jr50ZcR8pLwoNw

Dne středa 27. března 2019 15:53:58 UTC+1 Nick Cabatoff napsal(a):

[Nick Cabatoff]

So the token default-token-hlv24 as reported by kubectl translates to:

{

  "iss": "kubernetes/serviceaccount",

  "kubernetes.io/serviceaccount/namespace": "sm-core-aws-api-authentication",

  "kubernetes.io/serviceaccount/secret.name": "default-token-jh6vw",

  "kubernetes.io/serviceaccount/service-account.name": "default",

  "kubernetes.io/serviceaccount/service-account.uid": "2622c16c-5072-11e9-99ac-06749d178c40",

  "sub": "system:serviceaccount:sm-core-aws-api-authentication:default"

}

And the one you're posting to the login API is:

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "sm-core-aws-api-authentication",
  "kubernetes.io/serviceaccount/secret.name": "default-token-hlv24",
  "kubernetes.io/serviceaccount/service-account.name": "default",
  "kubernetes.io/serviceaccount/service-account.uid": "db357039-4b1f-11e9-99ac-06749d178c40",
  "sub": "system:serviceaccount:sm-core-aws-api-authentication:default"
}

Note the different service account uid.  I suspect someone recreated the service account, and hence the token, and hasn't updated the caller to the vault login API.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/9e0de574-4fd0-4c5f-a0cc-20ceea1ce758%40googlegroups.com.

[Buri]

You are right. It seems that when I was upgrading Istio, the serviceaccount got recreated. I've updated Token Reviewer JWT with new jwt for that service and its working now. Thank you very much, you are my hero!

Dne středa 27. března 2019 16:02:30 UTC+1 Nick Cabatoff napsal(a):

[Nick Cabatoff]

Glad I could help, and thanks for helping me learn about the kubernetes auth plugin!

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/3b8fd040-1726-46e8-83d6-13f45fafe583%40googlegroups.com.