How to access vault api

[Manojit Das]

I have deployed vault on k8s. See below :

========

NAME                         READY     STATUS    RESTARTS   AGE

consul-0                     1/1       Running   0          2h

consul-1                     1/1       Running   0          2h

consul-2                     1/1       Running   0          2h

vault-64bf65fff5-fpbvq       2/2       Running   0          2h

vault-64bf65fff5-mkp4s       2/2       Running   0          2h

NAME                                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                                                                            AGE

consul                              ClusterIP   None         <none>        8500/TCP,8443/TCP,8400/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP   3d

kubernetes                          ClusterIP   10.0.0.1     <none>        443/TCP                                                                            140d                                                                        4d

vault                               ClusterIP   10.0.0.216   <none>        8200/TCP

==============

I am able to run vault cli from my workstation with kubectl port-forward on as below

============

vault write auth/kubernetes/login role=postgres jwt=$KUBE_TOKEN

Handling connection for 8200

Key                                       Value

---                                       -----

token                                     6tuCz3C3qCbFnpwX568lZM1D

token_accessor                            4ViFbHlsgFjLIqmL73ayNe68

token_duration                            24h

token_renewable                           true

token_policies                            ["default" "postgres-policy"]

identity_policies                         []

policies                                  ["default" "postgres-policy"]

token_meta_role                           postgres

token_meta_service_account_name           postgres-vault

token_meta_service_account_namespace      default

token_meta_service_account_secret_name    postgres-vault-token-fc98g

token_meta_service_account_uid            3fbde507-2760-11e9-b514-0050569602bb

===============

But I am not able to call vault api for the same commands. Pls guide me what's is missing ..

===============

curl \

>     --request POST \

>     --data @payload.json \

>     http://127.0.0.1:8200/v1/auth/kubernetes/login

Handling connection for 8200

even with https 

Handling connection for 8200

curl: (60) Peer's certificate has an invalid signature.

More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"

 of Certificate Authority (CA) public keys (CA certs). If the default

 bundle file isn't adequate, you can specify an alternate file

 using the --cacerE0206 14:23:42.924575   12069 portforward.go:316] error copying from local connection to remote stream: read tcp4 127.0.0.1:8200->127.0.0.1:32858: read: connection reset by peer

t option.

If this HTTPS server uses a certificate signed by a CA represented in

 the bundle, the certificate verification probably failed due to a

 problem with the certificate (it might be expired, or the name might

 not match the domain name in the URL).

If you'd like to turn off curl's verification of the certificate, use

 the -k (or --insecure) option.

=================== 

My payload.json contains

{

  "role": "postgres",

  "jwt": "eyJ..."

}

===================

vault status

Handling connection for 8200

Key             Value

---             -----

Seal Type       shamir

Initialized     true

Sealed          false

Total Shares    1

Threshold       1

Version         0.11.5

Cluster Name    vault-cluster-2ac865f1

Cluster ID      36d4728b-b280-decc-efda-409c534d8a8b

HA Enabled      true

HA Cluster      https://127.0.0.1:8201

HA Mode         active

============

Any help will be appreciated ..

Thanks and Regards

Manojit

[Michel Vocks]

Hi Manojit,

Vault's default address is "https://127.0.0.1:8200" so it uses HTTPS by default.

I can see from your curl request via HTTPS that vault answers but the used certificate is not trusted by your local system. You can workaround this problem by adding "--insecure" to your curl request.

This is okay for testing purpose but in production you should add your self-signed certificate to your local trusted certificates.

One additional tip: When you add "-v" to your curl command it will print more information which could be helpful for debugging purpose.

Cheers,

Michel

[Manojit Das]

Thanks Mickel, with the insecure tag I am able to get the response. My next question would be 

1) What's the series of steps for secure api access with curl. I have self-signed certificate in my env. I use to export it as export VAULT_CACERT="certs/ca.pem" before running vault cli. 

2) How to use the vault services which is exposed as ClusterIP rather than localhost by kubectl port-forward. Do we need to expose vault as NodeIP or any kind of ingress? What's the best practice in a Production deployment. 

Regards

Manojit

[Manojit Das]

I am trying to access the vault from another pod running in the cluster using the vault service.. But somehow the request is being rejected..

From inside the pods the commands are shown below:

=============================

[ root@curl-7dfbc69769-9nk28:/ ]$ nslookup vault

Server:    10.0.0.10

Address 1: 10.0.0.10 kube-dns.kube-system.svc.gtslabs.ibm.com

Name:      vault

Address 1: 10.0.0.216 vault.default.svc.gtslabs.ibm.com

-------------------

KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

curl -k --request POST --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "postgres"}' https://vault:8

200/v1/auth/kubernetes/login)

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to vault port 8200: Connection refused

[ root@curl-7dfbc69769-9nk28:/ ]$

=====================

Regards

Manojit

[mic...@hashicorp.com]

1) What's the series of steps for secure api access with curl. I have self-signed certificate in my env. I use to export it as export VAULT_CACERT="certs/ca.pem" before running vault cli.

The equivalent for "VAULT_CACERT" is the "--cacert" parameter for curl.

2) How to use the vault services which is exposed as ClusterIP rather than localhost by kubectl port-forward. Do we need to expose vault as NodeIP or any kind of ingress? What's the best practice in a Production deployment. 

If you want to make Vault available outside of the Cluster you need to expose the service: https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-intro/

This is not really a concern of Vault, it's more a concern of Kubernetes. You should always consider if you expose Vault (especially through the internet) more people will get access to it and this is what you should try to avoid.

Please have a look the following recommendations: https://learn.hashicorp.com/vault/operations/production-hardening

I am trying to access the vault from another pod running in the cluster using the vault service.. But somehow the request is being rejected..

This can have several reasons but it's most likely related to your Kubernetes setup. Kubernetes provides here a really good documentation: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/ 

Cheers,

Michel

[Manojit Das]

Thanks Michel for your valuable response.

1) --cacert" parameter for curl is NOT working.. the syntax I am using ==> curl --cacert "certs/ca.pem" --header "X-Vault-Token: xxxxxxx" https://127.0.0.1:8200/v1/secret/precious

2) vault services has been exposed as ClusterIP

3) vault services are resolving in kube-dns and pointing to the vault pods properly but it is giving connection refused.. whereas other services like Jenkins-agent which is exposed same was as ClusterIP is responding both from cluster Nodes and from other pods.

But we are able to port-forward to localhost and can access vault. Can you think deeper what's still missing from vault service? 

Regards

Manojit

[mic...@hashicorp.com]

Hi Manojit,

1) Please verify your used certificate. If you used a certificate signed by a CA you need to use the public certificate from the CA.

2&3) It would be helpful to have an insight into the Kubernetes deployment definitions and the vault configuration file.

Cheers,

Michel