Certificate issue while login with k8s auth

[Tene Thomas]

I am using Docker for Desktop in Mac

Engine: 18.06.0-ce

Kubernetes: v1.10.3

I am trying to setup k8s authentication in vault.

While doing this, I am getting error

vault write auth/kubernetes/login role=test-role jwt=$TR_ACCOUNT_TOKEN

Error writing data to auth/kubernetes/login: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/auth/kubernetes/login
Code: 500. Errors:

* Post https://localhost:6443/apis/authentication.k8s.io/v1/tokenreviews: x509: certificate is valid for docker-for-desktop, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, host.docker.internal, not localhost

What am I doing wrong?

[Michel Vocks]

Hi there,

the "kubernetes_host" parameter (https://www.vaultproject.io/api/auth/kubernetes/index.html#kubernetes_host) points to a HTTPS endpoint with a certificate which does not include the localhost hostname.

Ideally, you would change the Kubernetes API certificate to a valid certificate but if this is only for testing purpose, you can workaround this issue by creating a HTTP proxy to the Kubernetes API: https://kubernetes.io/docs/tasks/access-kubernetes-api/http-proxy-access-api/

Afterwards, you should be able to change the "kubernetes_host" parameter to "http://localhost:8080".

Cheers,

Michel 

[Tene Thomas]

Hi Michel

I was able to workaround my problem thanks to your suggestion.

However, I am getting an error while creating secret

vault write auth/kubernetes/login role=ofsc-role jwt=$TR_ACCOUNT_TOKEN
Key                                       Value
---                                       -----
token                                     s.BndaZd4iRs3OGgBHCM3X5oQK
token_accessor                            8esDT64bsYK3WkSQpP8VoW7A
token_duration                            768h
token_renewable                           true
token_policies                            ["default" "test-policy"]
identity_policies                         []
policies                                  ["default" "test-policy"]
token_meta_service_account_namespace      default
token_meta_service_account_secret_name    vault-test-token-rxpvq
token_meta_service_account_uid            b5175d01-7847-11e9-b2dc-025000000001
token_meta_role                           ofsc-role
token_meta_service_account_name           vault-test

export VAULT_TOKEN=s.BndaZd4iRs3OGgBHCM3X5oQK

vault write secret/test/foo value=bar

Error writing data to secret/test/foo: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/secret/test/foo
Code: 404. Errors:

* no handler for route 'secret/test/foo'

Could you please let me know what's wrong here?

Thanks

Tene

[Tene Thomas]

I am able to successfully run vault commands using the token received as part of initializing the vault.

However after setting up kubernetes authentication, the token received from k8s login does not work.

vault secrets list
Error listing secrets engines: Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/mounts
Code: 403. Errors:

* 1 error occurred:
    * permission denied

[Michel Vocks]

Hi there,

it looks like your defined policy is incorrect which is attached to your token.

Cheers,

Michel

[Tene Thomas]

This is the policy attached to the token

cat policy.hcl
path "secret/test/*" {
    capabilities = ["create", "read", "update", "delete", "list"]
}

Thanks

Tene

[Michel Vocks]

"vault secrets list" uses the "sys/mounts" API endpoint to read all mounts. You need provide explicit access to this API if you want to allow your token(s) to access this API endpoint:

path "sys/mounts" {
    capabilities = [ "read" ]
}

Cheers,

Michel

[Tene Thomas]

I have added necessary capabilities, but still no luck.

--cat policy.hcl

path "secret/test/*" {
    capabilities = ["create", "read", "update", "delete", "list"]
}

path "sys/mounts" {
    capabilities = ["read"]
}

path "sys/policies" {
    capabilities = ["read"]
}

curl --request POST --data "{ \"jwt\": \"$TR_ACCOUNT_TOKEN\", \"role\": \"test-role\" }" http://127.0.0.1:8200/v1/auth/kubernetes/login
{"request_id":"b153f5fd-d3c4-87ab-cca2-3f643888f2f4","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.kVMbmeo0G3JxFy72rM5ZXMFJ","accessor":"8Rvn2iNm8vzcryKYmcvcjm9J","policies":["default","test-policy"],"token_policies":["default","test-policy"],"metadata":{"role":"test-role","service_account_name":"vault-test","service_account_namespace":"default","service_account_secret_name":"vault-test-token-rxpvq","service_account_uid":"b5175d01-7847-11e9-b2dc-025000000001"},"lease_duration":2764800,"renewable":true,"entity_id":"9a3ce5e1-94c7-254e-a16a-b63497b4ce5c","token_type":"service","orphan":true}}

tethomas-mac:vault-project tethomas$ export VAULT_TOKEN=s.kVMbmeo0G3JxFy72rM5ZXMFJ
tethomas-mac:vault-project tethomas$ vault policy list
Error listing policies: Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/policies/acl?list=true

Code: 403. Errors:

* 1 error occurred:
    * permission denied

Is there anything else I can check to debug this issue?

Thanks

Tene

[Michel Vocks]

This command is different from the one you mentioned before (policies instead of mounts).

For policies you have to enable the policies API:

path "sys/policies*" {
    capabilities = [ "list" ]
}

Cheers,

Michel

[Tene Thomas]

I have enabled both policies and mounts. None of the api's are working. Looks like the issue is not related to policy

cat policy.hcl
path "secret/test/*" {
    capabilities = ["create", "read", "update", "delete", "list"]
}

path "sys/mounts/*" {
    capabilities = ["read", "list"]
}
path "sys/policies/*" {
    capabilities = ["read", "list"]
}

[Michel Vocks]

Are you able to read/write to "secret/test/*"?

Please double check if the policy is really attached to the token and that the token is actually used and not a old token is still active (e.g. check VAULT_TOKEN variable).