How to Fetch secrets from Vault and store kv pair and certificates in the pod - approach
674 views
Skip to first unread message
Arun Prasath
Jan 3, 2020, 12:59:46 PM1/3/20
to Vault
I have a Vault running in the Minishift(OC cluster) and then operator init and unsealed. Then enabled the Kubernetes auth method and configured it to test the auth method login from the OC deployment.
In the pod deployment logs, I could see the secrets fetched from vault and read the SECRET vaules in the output.
vault-deploy$ oc logs pod/basic-example-6f7bffbcbb-9h8nr
2020/01/02 20:16:55 ==> WARNING: Don't ever write secrets to logs.
2020/01/02 20:16:55 ==> This is for demonstration only.
2020/01/02 20:16:55 s.YbnyYqHSNhgFZuu5RSw2ivrd
2020/01/02 20:16:55 secret secret/data/app1 -> &{393e2abe-5a3f-7d18-af12-489fb6b79dc0 0 false map[data:map[username:app1 password:supasecr3t] metadata:map[created_time:2020-01-02T14:47:17.904134031Z deletion_time: destroyed:false version:1]] [] <nil> <nil>}
2020/01/02 20:16:55 Starting renewal loop
2020/01/02 20:16:55 Successfully renewed: &api.RenewOutput{RenewedAt:time.Time{wall:0x12f518ab, ext:63713593015, loc:(*time.Location)(nil)}, Secret:(*api.Secret)(0xc00004c3c0)}
Question is,
I think of below use-cases.
For key/value secrets,
1. How to store the secrets in the pods as Environment Variables or in a file in the pod or what is the right approach to store and use the secrets inside the pod.
For key/certificates,
2. How to fetch and store the key and crt files in the pod for Nginx deployment
I know this could be the basic thing in vault and kubernetes, but I am not clear on how VAULT Expose the secret values to pod in a right way.
Any documentation or tutorial or blog or github page or any suggestion is appriciated.
Vasilyev Viacheslav
Jan 4, 2020, 12:03:19 PM1/4/20
to Vault
Arun Prasath
Jan 6, 2020, 7:46:37 AM1/6/20
to Vault
Hi Vasilyev - Thanks for the URL. I will check it out and get back to you.
Arun Prasath
Jan 7, 2020, 9:27:47 AM1/7/20
to Vault
Hi Vasilyev,
I have to deploy in Openshift, so manually deployed vault and sidecar and then applied patch and it applied successfully. But the secrets are NOT available in the path /vault/secrets/ inside the pod.
What is missing and there is no error in the pod logs and no clue where to dig.
I am expecting secrets at below path. The secrets are not fetched using annotations
k8s$ oc exec -it app2-6d6f8cbd7b-l4j59 -- ls -l /vault/secrets/app2
ls: /vault/secrets/app2: No such file or directory
command terminated with exit code 1
Below is the config.
k8s$ cat app2-deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: app2
namespace: vault-demo
spec:
selector:
matchLabels:
app: vault-agent-demo
replicas: 1
template:
metadata:
annotations:
labels:
app: vault-agent-demo
spec:
serviceAccountName: app2
containers:
- name: app2
image: "jweissig/app:0.0.1"
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: app2
labels:
app: vault-agent-demo
k8s$ cat patch-annotations.yaml
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-secret-app2: "secret/app2"
vault.hashicorp.com/role: "app2"
Secret is available at below path,
k8s$ vault kv get secret/app2
====== Metadata ======
Key Value
--- -----
created_time 2020-01-06T20:59:42.892728923Z
deletion_time n/a
destroyed false
version 1
====== Data ======
Key Value
--- -----
password supasecr3t
username app2
After patch, the annotations are available in app2 deployment,
k8s$ oc get deployment app2 -o yaml|grep vault
app: vault-agent-demo
namespace: vault-demo
selfLink: /apis/extensions/v1beta1/namespaces/vault-demo/deployments/app2
app: vault-agent-demo
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-secret-app2: secret/app2
vault.hashicorp.com/role: app2
app: vault-agent-demo
Jason O'Donnell
Jan 7, 2020, 9:42:14 AM1/7/20
to Vault
Arun,
It doesn't appear that the sidecar container was injected. I wonder if Minishift does not have admission controllers enabled by default. I think you can enable it using:
$ minishift addon apply dynamic-admission-controllers
If admission controllers are enabled, did you check the logs of the injector to see if any requests were made? Is the injector running and healthy?
$ oc get mutatingwebhookconfigurations
$ oc describe mutatingwebhookconfigurations vault-agent-injector-cfg
$ oc logs <name of the injector pod>
Arun Prasath
Jan 7, 2020, 10:34:18 AM1/7/20
to Vault
Jason, I have not enabled any addon in minishift. The dynamic-admission-controllers are not installed and I have to install and enable. I see some errors in the mutatingwebhook after installing the patch. Could you please tell whether this error is related to addon dynamic-admission-controllers? However, sidecare is running and is okay.
In addition, I also added the pvc to mount the /vault/secrets directory in the pod. Is this also required? Kindly please review these config.
k8s$ minishift addon apply dynamic-admission-controllers
No add-on with the name 'dynamic-admission-controllers' is installed.
k8s$ minishift addon install dynamic-admission-controllers
Add-on installation failed with the error: The source of a addon needs to be a directory. 'dynamic-admission-controllers' is not
k8s$ oc get all |grep inject
pod/vault-injector-7d4bb88bd7-74l8p 1/1 Running 0 18h
service/vault-agent-injector-svc ClusterIP 172.30.29.109 <none> 443/TCP 18h
deployment.apps/vault-injector 1 1 1 1 18h
replicaset.apps/vault-injector-7d4bb88bd7 1 1 1 18h
route.route.openshift.io/vault-agent-injector-svc vault-agent-injector-svc-vault-demo.192.168.42.233.nip.io vault-agent-injector-svc <all> None
k8s$ oc logs vault-injector-7d4bb88bd7-74l8p
Listening on ":8080"...
2020-01-06T20:55:36.367Z [INFO] handler: Starting handler..
Updated certificate bundle received. Updating certs...
Error updating MutatingWebhookConfiguration: the server was unable to return a response in the time allotted, but may still be processing the request (patch mutatingwebhookconfigurations.admissionregistration.k8s.io vault-agent-injector-cfg)
k8s$ oc get mutatingwebhookconfigurations
NAME CREATED AT
vault-agent-injector-cfg 2020-01-06T20:55:25Z
k8s$ oc describe mutatingwebhookconfigurations vault-agent-injector-cfg
Name: vault-agent-injector-cfg
Namespace:
Labels: app=vault-injector
Annotations: <none>
API Version: admissionregistration.k8s.io/v1beta1
Kind: MutatingWebhookConfiguration
Metadata:
Creation Timestamp: 2020-01-06T20:55:25Z
Generation: 2
Resource Version: 95347
Self Link: /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/vault-agent-injector-cfg
UID: dc9ce2e4-30c6-11ea-b5bc-52540068b345
Webhooks:
Client Config:
Ca Bundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNSekNDQWU2Z0F3SUJBZ0lVRldxbzJMYU5nU0crbHpnSkhzSWJ3djA5c0g4d0NnWUlLb1pJemowRUF3SXcKR2pFWU1CWUdBMVVFQXhNUFFXZGxiblFnU1c1cVpXTjBJRU5CTUI0WERUSXdNREV3TmpJd05UUXpObG9YRFRNdwpNREV3TXpJd05UVXpObG93R2pFWU1CWUdBMVVFQXhNUFFXZGxiblFnU1c1cVpXTjBJRU5CTUZrd0V3WUhLb1pJCnpqMENBUVlJS29aSXpqMERBUWNEUWdBRTRIdE1YUG9LK2FmeVFsTTRyTzJEZEpNYVRBRThNNlRQYXBRVUk3QUsKMTBMc1ZKRzk4OS80VU1YRmlLbGhaU1djK1NVakNIVFdOVGJWalVDNk5QOGdUcU9DQVJBd2dnRU1NQTRHQTFVZApEd0VCL3dRRUF3SUNoREFUQmdOVkhTVUVEREFLQmdnckJnRUZCUWNEQVRBUEJnTlZIUk1CQWY4RUJUQURBUUgvCk1HZ0dBMVVkRGdSaEJGODNaVG8wTURvek5Eb3lNVG8zWWpvM1pUb3hOanBtWVRvMVpqcGxNenBrWmpwak1qb3gKWlRvMk5EcGxOem81TXpwa05UbzBZem95TmpvMk9EbzRaVHBrTURvMFlqbzJOenBrTWpwaU9UcGxOenBtTWpvdwpaam8wWXpvd1lqbzRNREJxQmdOVkhTTUVZekJoZ0Y4M1pUbzBNRG96TkRveU1UbzNZam8zWlRveE5qcG1ZVG8xClpqcGxNenBrWmpwak1qb3haVG8yTkRwbE56bzVNenBrTlRvMFl6b3lOam8yT0RvNFpUcGtNRG8wWWpvMk56cGsKTWpwaU9UcGxOenBtTWpvd1pqbzBZem93WWpvNE1EQUtCZ2dxaGtqT1BRUURBZ05IQURCRUFpQlRnVURGdEpVeAoyN1Z3R2V4cUMxbUE3MTQzUTU5SWlYY3dZYnpXSXBEM0RRSWdHdXBHOVpiQmVzYlZDQUhHbU9zMlVPbFdwcXlMCmF2bnV1cU9Xb2l0bW4vYz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
Service:
Name: vault-agent-injector-svc
Namespace: vault-demo
Path: /mutate
Failure Policy: Ignore
Name: vault.hashicorp.com
Namespace Selector:
Rules:
API Groups:
API Versions:
v1
Operations:
CREATE
UPDATE
Resources:
deployments
jobs
pods
statefulsets
Events: <none>
Deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: app2
namespace: vault-demo
spec:
selector:
matchLabels:
app: vault-agent-demo
replicas: 1
template:
metadata:
annotations:
labels:
app: vault-agent-demo
spec:
serviceAccountName: app2
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: mypvc
containers:
- name: app2
image: "jweissig/app:0.0.1"
volumeMounts:
- name: mypvc
mountPath: /vault/secrets
Arun Prasath
Jan 7, 2020, 10:55:12 AM1/7/20
to Vault
Got the connection refuced error after enabling the dynamic-admission-controllers. what is this error and if this related to permission, how to fix this.
k8s$ oc logs pod/vault-injector-7d4bb88bd7-82c2m
2020-01-07T15:40:06.204Z [INFO] handler: Starting handler..
Updated certificate bundle received. Updating certs...
Listening on ":8080"...
Error updating MutatingWebhookConfiguration: Patch https://172.30.0.1:443/apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/vault-agent-injector-cfg: dial tcp 172.30.0.1:443: connect: connection refused
Jason O'Donnell
Jan 7, 2020, 11:36:55 AM1/7/20
to Vault
Arun,
Message has been deleted
Vasilyev Viacheslav
Jan 7, 2020, 3:59:28 PM1/7/20
to Vault
You may install dynamic admission controllers add-on with
git clone https://github.com/minishift/minishift-addons.git
cd minishift-addons/
minishift addons install add-ons/dynamic-admission-controllers/
minishift addons apply dynamic-admission-controllers
Arun Prasath
Jan 8, 2020, 9:40:24 AM1/8/20
to Vault
It seems to be openshift API server running. why do we need to specify explicitly to sidecar injector. The openshift or Kubernetes should take care of API server internally if kubernetes/minishift is running and service pods.
The below are the pods in kube-system and describe output of pods. How to get the URL of the OC API server and how to test the connectivity manually. Do I need to manually configure the OC API anywhere to make this to work.
I believe, if Kubernetes components are installed manually then I need to configure the API server manually to integrate the parts to work. But minishift is single entity which should takecare of all the components and it is already running and service pods. correct me if my understanding is wrong so that I can check this with minishift forum.
$ oc get pods -n kube-system
NAME READY STATUS RESTARTS AGE
kube-controller-manager-localhost 1/1 Running 3 4d
kube-scheduler-localhost 1/1 Running 3 4d
master-api-localhost 1/1 Running 6 4d
master-etcd-localhost 1/1 Running 3 4d
$ oc describe pods master-api-localhost -n kube-system
Name: master-api-localhost
Namespace: kube-system
Priority: 0
PriorityClassName: <none>
Node: localhost/192.168.122.252
Start Time: Fri, 03 Jan 2020 16:46:06 -0300
Labels: openshift.io/component=api
Status: Running
IP: 192.168.122.252
Containers:
api:
Container ID: docker://498106f601cd8466f378bce6c5460d7cc78784a3076ef957e5b159a7b2a35a33
Image: openshift/origin-hypershift:v3.11.0
Image ID: docker-pullable://docker.io/openshift/origin-hypershift@sha256:bc8306c2c34ddf1dc25209d61dcf6e38b800b09d95b8c8e4be5276cdf5b3b039
Port: <none>
Host Port: <none>
Command:
/bin/bash
-c
Args:
#!/bin/bash
set -euo pipefail
if [[ -f /etc/origin/master/master.env ]]; then
set -o allexport
source /etc/origin/master/master.env
fi
exec hypershift openshift-kube-apiserver --config=/etc/origin/master/master-config.yaml
State: Running
Started: Tue, 07 Jan 2020 12:46:41 -0300
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Mon, 06 Jan 2020 14:14:48 -0300
Finished: Tue, 07 Jan 2020 12:46:36 -0300
Ready: True
Restart Count: 6
Liveness: http-get https://:8443/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/origin/cloudprovider/ from master-cloud-provider (rw)
/etc/origin/master/ from master-config (rw)
/var/lib/origin/ from master-data (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
master-config:
Type: HostPath (bare host directory volume)
Path: /var/lib/minishift/base/kube-apiserver
HostPathType:
master-cloud-provider:
Type: HostPath (bare host directory volume)
Path: /etc/origin/cloudprovider
HostPathType:
master-data:
Type: HostPath (bare host directory volume)
Path: /var/lib/origin
HostPathType:
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: :NoExecute
Events: <none>
Jason O'Donnell
Jan 8, 2020, 9:53:36 AM1/8/20
to Vault
Hi Arun,
There is no need to specify the apiserver to the sidecar injector, the client-go package configures this for the injector automatically, that's why this error is interesting.
The sidecar injector wasn't tested on OpenShift/Minishift, it currently only officially supports Kubernetes. I've opened an issue on the Vault K8s project to test on OpenShift, you can find it here: https://github.com/hashicorp/vault-k8s/issues/38.
Arun Prasath
Jan 8, 2020, 10:30:31 AM1/8/20
to Vault
Hi Jason, I will keep an eye on the github issue and will come back. Thanks for creating the issue.
Meanwhile, I can try if there is an workaround on the openshift side. What are your thoughts?
Vasilyev Viacheslav
Jan 8, 2020, 11:55:01 AM1/8/20
to Vault
Minishift has stripped down version of Kubernetes, hence advanced stuff must be installed separately and not fact they will work. This also true for "admissions-webhook" and "dynamic-admission-controller" add-ons
Arun Prasath
Jan 8, 2020, 5:53:01 PM1/8/20
to Vault
yeah i agree. openshift is behind the latest kubernetes version and advanced features may need to be explicitly deployed as an addon. deploying vault via helm in kubernetes deploys the sidecare along with vault. but in oc, i had to deploy vault and injector seperately and there is no native support and all the required stuff has to be done manually to make this work. moreover, i am a learner and have difficulty understanding the integraion of the moving parts in vault, vault-injector and mutations etc. so I had to reach out to experts to seeks help.
I also did a fresh minishift start and copied the minishfit logs to below github repo. just incase you need for debugging.
But this time, I didn't see the error but secrets are not mounted at /vault/secrets/app1 in the pods. I will keep you posted if I found anything interesting. thanks once again.
Reply all
Reply to author
Forward
0 new messages