Integrating the webhook api - questions

[Alexander Gallego]

Hi

I'm trying to integrate w/ the kubevirt hooks api. First, to not run a forked version of kubevirt, i need to inject initContainers.

I'm running into an odd behavior w/ minikube, k8s 1.10.4 and kubevirt 0.7.

First given a successful registration of an admission mutating webhooks, k8s sends me an API request w/ this: 

```

I0829 22:38:29.185938 1 webhook.go:174] INCOMING body: {"kind":"AdmissionReview","apiVersion":"admission.k8s.io/v1beta1","request":{"uid":"4020e946-abdc-11e8-b678-78b6c5ec689c","kind":{"group":"","version":"v1","kind":"Pod"},"resource":{"group":"","version":"v1","resource":"pods"},"name":"virt-launcher-kvirtvm3-n5xs6","namespace":"default","operation":"UPDATE","userInfo":{"username":"system:serviceaccount:kube-system:kubevirt-controller","uid":"374672d5-abdb-11e8-b678-78b6c5ec689c","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"object":{"metadata":{"name":"virt-launcher-kvirtvm3-n5xs6","generateName":"virt-launcher-kvirtvm3-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/virt-launcher-kvirtvm3-n5xs6","uid":"18270752-abdc-11e8-b678-78b6c5ec689c","resourceVersion":"1418","creationTimestamp":"2018-08-29T22:37:22Z","labels":{"guest":"testvm","kubevirt.io":"virt-launcher","kubevirt.io/domain":"kvirtvm3","nui.akamai.com/size":"small"},"annotations":{"kubevirt.io/created-by":"1781d34d-abdc-11e8-b678-78b6c5ec689c","kubevirt.io/owned-by":"virt-handler"}},"spec":{"volumes":[{"name":"virt-share-dir","hostPath":{"path":"/var/run/kubevirt","type":""}},{"name":"libvirt-runtime","emptyDir":{}},{"name":"default-token-gng9l","secret":{"secretName":"default-token-gng9l","defaultMode":420}}],"containers":[{"name":"volumeregistryvolume","image":"docker.io/senior7515/fedora27qcow2base:latest","command":["/entry-point.sh"],"env":[{"name":"COPY_PATH","value":"/var/run/libvirt/kubevirt-ephemeral-disk/registry-disk-data/default/kvirtvm3/disk_registryvolume/disk-image"}],"resources":{},"volumeMounts":[{"name":"libvirt-runtime","mountPath":"/var/run/libvirt"},{"name":"default-token-gng9l","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"readinessProbe":{"exec":{"command":["cat","/tmp/healthy"]},"initialDelaySeconds":2,"timeoutSeconds":5,"periodSeconds":5,"successThreshold":2,"failureThreshold":5},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"compute","image":"docker.io/kubevirt/virt-launcher:v0.7.0","command":["/usr/share/kubevirt/virt-launcher/entrypoint.sh","--qemu-timeout","5m","--name","kvirtvm3","--namespace","default","--kubevirt-share-dir","/var/run/kubevirt","--readiness-file","/tmp/healthy","--grace-period-seconds","45"],"resources":{"limits":{"devices.kubevirt.io/kvm":"1","devices.kubevirt.io/tun":"1"},"requests":{"devices.kubevirt.io/kvm":"1","devices.kubevirt.io/tun":"1","memory":"1099507557"}},"volumeMounts":[{"name":"virt-share-dir","mountPath":"/var/run/kubevirt"},{"name":"libvirt-runtime","mountPath":"/var/run/libvirt"},{"name":"default-token-gng9l","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"readinessProbe":{"exec":{"command":["cat","/tmp/healthy"]},"initialDelaySeconds":2,"timeoutSeconds":5,"periodSeconds":2,"successThreshold":1,"failureThreshold":5},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false,"runAsUser":0}}],"restartPolicy":"Never","terminationGracePeriodSeconds":60,"dnsPolicy":"ClusterFirst","nodeSelector":{"kubevirt.io/schedulable":"true"},"serviceAccountName":"default","serviceAccount":"default","nodeName":"minikube","securityContext":{"seLinuxOptions":{"type":"spc_t"},"runAsUser":0},"hostname":"kvirtvm3","schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}]},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-29T22:37:22Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-29T22:38:29Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-29T22:37:22Z"}],"hostIP":"192.168.122.108","podIP":"172.17.0.11","startTime":"2018-08-29T22:37:22Z","containerStatuses":[{"name":"compute","state":{"running":{"startedAt":"2018-08-29T22:38:17Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kubevirt/virt-launcher:v0.7.0","imageID":"docker-pullable://kubevirt/virt-launcher@sha256:82e87628f9969407d41cf9940d13010b7327cc9d7e9e13f3d666e88063e9ce9a","containerID":"docker://f9cfc3eb9abb10d151e5f710fd6271870c275216748520cb9bd445e1e5c06677"},{"name":"volumeregistryvolume","state":{"running":{"startedAt":"2018-08-29T22:37:55Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"senior7515/fedora27qcow2base:latest","imageID":"docker-pullable://senior7515/fedora27qcow2base@sha256:44da00120e83eb90e31ab6c173c2e30fd89c1dfab403546e2970d8536742a458","containerID":"docker://2544381e2615fb749aa3e828c49bff821f91be64a3f73a18ddc953b9acd1d64a"}],"qosClass":"Burstable"}},"oldObject":{"metadata":{"name":"virt-launcher-kvirtvm3-n5xs6","generateName":"virt-launcher-kvirtvm3-","namespace":"default","uid":"18270752-abdc-11e8-b678-78b6c5ec689c","resourceVersion":"1418","creationTimestamp":"2018-08-29T22:37:22Z","labels":{"guest":"testvm","kubevirt.io":"virt-launcher","kubevirt.io/domain":"kvirtvm3","nui.akamai.com/size":"small"},"annotations":{"kubevirt.io/created-by":"1781d34d-abdc-11e8-b678-78b6c5ec689c","kubevirt.io/owned-by":"virt-controller"}},"spec":{"volumes":[{"name":"virt-share-dir","hostPath":{"path":"/var/run/kubevirt","type":""}},{"name":"libvirt-runtime","emptyDir":{}},{"name":"default-token-gng9l","secret":{"secretName":"default-token-gng9l","defaultMode":420}}],"containers":[{"name":"volumeregistryvolume","image":"docker.io/senior7515/fedora27qcow2base:latest","command":["/entry-point.sh"],"env":[{"name":"COPY_PATH","value":"/var/run/libvirt/kubevirt-ephemeral-disk/registry-disk-data/default/kvirtvm3/disk_registryvolume/disk-image"}],"resources":{},"volumeMounts":[{"name":"libvirt-runtime","mountPath":"/var/run/libvirt"},{"name":"default-token-gng9l","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"readinessProbe":{"exec":{"command":["cat","/tmp/healthy"]},"initialDelaySeconds":2,"timeoutSeconds":5,"periodSeconds":5,"successThreshold":2,"failureThreshold":5},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"compute","image":"docker.io/kubevirt/virt-launcher:v0.7.0","command":["/usr/share/kubevirt/virt-launcher/entrypoint.sh","--qemu-timeout","5m","--name","kvirtvm3","--namespace","default","--kubevirt-share-dir","/var/run/kubevirt","--readiness-file","/tmp/healthy","--grace-period-seconds","45"],"resources":{"limits":{"devices.kubevirt.io/kvm":"1","devices.kubevirt.io/tun":"1"},"requests":{"devices.kubevirt.io/kvm":"1","devices.kubevirt.io/tun":"1","memory":"1099507557"}},"volumeMounts":[{"name":"virt-share-dir","mountPath":"/var/run/kubevirt"},{"name":"libvirt-runtime","mountPath":"/var/run/libvirt"},{"name":"default-token-gng9l","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"readinessProbe":{"exec":{"command":["cat","/tmp/healthy"]},"initialDelaySeconds":2,"timeoutSeconds":5,"periodSeconds":2,"successThreshold":1,"failureThreshold":5},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false,"runAsUser":0}}],"restartPolicy":"Never","terminationGracePeriodSeconds":60,"dnsPolicy":"ClusterFirst","nodeSelector":{"kubevirt.io/schedulable":"true"},"serviceAccountName":"default","serviceAccount":"default","nodeName":"minikube","securityContext":{"seLinuxOptions":{"type":"spc_t"},"runAsUser":0},"hostname":"kvirtvm3","schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}]},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-29T22:37:22Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-29T22:38:29Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-08-29T22:37:22Z"}],"hostIP":"192.168.122.108","podIP":"172.17.0.11","startTime":"2018-08-29T22:37:22Z","containerStatuses":[{"name":"compute","state":{"running":{"startedAt":"2018-08-29T22:38:17Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kubevirt/virt-launcher:v0.7.0","imageID":"docker-pullable://kubevirt/virt-launcher@sha256:82e87628f9969407d41cf9940d13010b7327cc9d7e9e13f3d666e88063e9ce9a","containerID":"docker://f9cfc3eb9abb10d151e5f710fd6271870c275216748520cb9bd445e1e5c06677"},{"name":"volumeregistryvolume","state":{"running":{"startedAt":"2018-08-29T22:37:55Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"senior7515/fedora27qcow2base:latest","imageID":"docker-pullable://senior7515/fedora27qcow2base@sha256:44da00120e83eb90e31ab6c173c2e30fd89c1dfab403546e2970d8536742a458","containerID":"docker://2544381e2615fb749aa3e828c49bff821f91be64a3f73a18ddc953b9acd1d64a"}],"qosClass":"Burstable"}}}}

```

In turn i send a valid JSONPatch response to the API server

```

I0829 22:38:29.187289 1 webhook.go:143] JSON Patch: [{"op":"add","path":"/spec/initContainers","value":[{"name":"init-1-kvirthook-injected","image":"docker.io/ubuntu","command":["/bin/bash"],"args":["-c","echo \"${STARTUP_SCRIPT}\" | bash"],"env":[{"name":"PODNAME"},{"name":"HELLO_WORLD","value":"'cruel world!'"},{"name":"HOST_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.hostIP"}}},{"name":"DOMAIN_FILE","value":"/var/run/libvirt/domain.xml"},{"name":"STARTUP_SCRIPT","value":"#!/bin/bash\necho 'wow, such empty!'\n"}],"resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always","securityContext":{"privileged":true}}]},{"op":"add","path":"/metadata/annotations","value":{"nui.akamai.com/vm":"2018-08-29 22:38:29.187153596 +0000 UTC m=+66.461473594"}}]

I0829 22:38:29.187437 1 webhook.go:204] Writing reponse as: {"response":{"uid":"4020e946-abdc-11e8-b678-78b6c5ec689c","allowed":true,"patch":"W3sib3AiOiJhZGQiLCJwYXRoIjoiL3NwZWMvaW5pdENvbnRhaW5lcnMiLCJ2YWx1ZSI6W3sibmFtZSI6ImluaXQtMS1rdmlydGhvb2staW5qZWN0ZWQiLCJpbWFnZSI6ImRvY2tlci5pby91YnVudHUiLCJjb21tYW5kIjpbIi9iaW4vYmFzaCJdLCJhcmdzIjpbIi1jIiwiZWNobyBcIiR7U1RBUlRVUF9TQ1JJUFR9XCIgfCBiYXNoIl0sImVudiI6W3sibmFtZSI6IlBPRE5BTUUifSx7Im5hbWUiOiJIRUxMT19XT1JMRCIsInZhbHVlIjoiJ2NydWVsIHdvcmxkIScifSx7Im5hbWUiOiJIT1NUX0lQIiwidmFsdWVGcm9tIjp7ImZpZWxkUmVmIjp7ImFwaVZlcnNpb24iOiJ2MSIsImZpZWxkUGF0aCI6InN0YXR1cy5ob3N0SVAifX19LHsibmFtZSI6IkRPTUFJTl9GSUxFIiwidmFsdWUiOiIvdmFyL3J1bi9saWJ2aXJ0L2RvbWFpbi54bWwifSx7Im5hbWUiOiJTVEFSVFVQX1NDUklQVCIsInZhbHVlIjoiIyEvYmluL2Jhc2hcbmVjaG8gJ3dvdywgc3VjaCBlbXB0eSEnXG4ifV0sInJlc291cmNlcyI6e30sInRlcm1pbmF0aW9uTWVzc2FnZVBhdGgiOiIvZGV2L3Rlcm1pbmF0aW9uLWxvZyIsInRlcm1pbmF0aW9uTWVzc2FnZVBvbGljeSI6IkZpbGUiLCJpbWFnZVB1bGxQb2xpY3kiOiJBbHdheXMiLCJzZWN1cml0eUNvbnRleHQiOnsicHJpdmlsZWdlZCI6dHJ1ZX19XX0seyJvcCI6ImFkZCIsInBhdGgiOiIvbWV0YWRhdGEvYW5ub3RhdGlvbnMiLCJ2YWx1ZSI6eyJudWkuYWthbWFpLmNvbS92bSI6IjIwMTgtMDgtMjkgMjI6Mzg6MjkuMTg3MTUzNTk2ICswMDAwIFVUQyBtPSs2Ni40NjE0NzM1OTQifX1d","patchType":"JSONPatch"}}

```

I've validated the JSONPatch that is

a) Valid JSON

b) Valid JSONPatch RFC format.

However, when I do 

`kubectl get pods/vm....` and get the pod definition of the VM, initContainers are not injected.

I tried to subscribe to multiple resources with no luck. 

Things I've tried:

1) Register for kubevirt API via a yaml

```

apiVersion: admissionregistration.k8s.io/v1beta1

kind: MutatingWebhookConfiguration

metadata:

  name: kvirthook-mutatingwebhookconfig

  labels:

    app: kvirthook

webhooks:

  - name: nui.akamai.com

    clientConfig:

      service:

        name: kvirthook-svc

        namespace: default

        path: "/mutate"

      caBundle: ${CA_BUNDLE}

    rules:

      - operations:

          - "CREATE"

          - "UPDATE"

        apiGroups:

          - "kubevirt.io"

        apiVersions:

          - "v1alpha2"

        resources:

          - "*/*"

          # - VirtualMachineInstance

          # - VirtualMachine

```

2)  Mutating just the Pods themselves

```

apiVersion: admissionregistration.k8s.io/v1beta1

kind: MutatingWebhookConfiguration

metadata:

  name: kvirthook-mutatingwebhookconfig

  labels:

    app: kvirthook

webhooks:

  - name: nui.akamai.com

    clientConfig:

      service:

        name: kvirthook-svc

        namespace: default

        path: "/mutate"

      caBundle: ${CA_BUNDLE}

    rules:

      - operations:

          - "CREATE"

          - "UPDATE"

        apiGroups:

          - ""

        apiVersions:

          - "v1"

        resources:

          - pods

```

On the first option, it sorta make sense that when you create a JSONPatch w/ the path of /spec/initContainers  (for example as specified in k8s upstream https://github.com/kubernetes/kubernetes/blob/master/test/images/webhook/pods.go#L32) it wouldn't apply since the VM definition does not have that field defined.

However, but when I do insert it into a Pod, it still ignores it. 

I'm not sure exactly why this would be the case. Running a hello world w/ simple HTTP services seem to work. 

There seems to be a weird interaction between the kubevirt CRD of VirtualMachine (the resulting pod) and injecting JSONPatch objects into it. 

Does anyone have experience w/ this? 

Thanks!

.alex

[Alexander Gallego]

Hi, so followup :)

The issue wasn't the JSONPatch w/ the CRD. The issue was that on 1.9.4 which was the test I was running before upgrading from kubevirt 0.5 -> 0.7 one could specify priviledge container as true without the root pod itself being priviledge - or it changed in a PR. 

So upgrading the kubevirt root pod to priviledge fixed it. 

Well.. what's weird is that the behavior changed in kubernetes 1.10.4 which i'm testing where the init contianers used to crash the main pod if it failed, but on my test it just doesn't show up :'( 

The issue is that it doesn't even show up on the UI because it fails at latent validation stage which means no one has visibility into it. 

Anyway, if anyone else runs into this issue, they can try that option. 

My init container injection is working w/ kubevirt 0.7

.alex