Add mutator webhook to add virtio-fs container to support IRSA for Kubevirt VMs

[Mohd Aqil Beig]

This is related to KubeVirt Issue #13311. Currently, KubeVirt VMs do not support IAM roles for service accounts when running on EKS clusters.

If you are using a service account as a filesystem with a KubeVirt VM, and that service account has the annotation eks.amazonaws.com/role-arn, the IRSA mutates the virt-launcher pod of the VM with the following volume:

volumes:
  - name: aws-iam-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: token

and also mounts the aws-iam-token to each container of the virt-launcher pod, along with the service account token:

volumeMounts:
  - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: kube-api-access-gbzpp
    readOnly: true
  - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
    name: aws-iam-token
    readOnly: true

However, the aws-iam-token is not available inside the VM because KubeVirt only adds the virtio-fs container to share the service account token at a specific path. It does not add a virtio-fs container to share the aws-iam-token, nor does it include the filesystem for it in the domain XML.

To address this, we plan to leverage a hook sidecar to modify the domain XML and add the filesystem for the aws-iam-token, similar to how KubeVirt adds the service account token. The configuration would look like this:


<filesystem type='mount' accessmode='passthrough'>
  <driver type='virtiofs' queue='1024'/>
  <source dir='' socket='/var/run/kubevirt/virtiofs-containers/aws-iam-token.sock'/>
  <target dir='aws-iam-token'/>
</filesystem>

However, we still need a webhook that modifies the virt-launcher pod to add the virtio-fs container. This container will share the token mounted in the pod with the VM using virtiofsd, similar to what is done for the service account token:

  - args:
    - --socket-path=/var/run/kubevirt/virtiofs-containers/aws-iam-token.sock
    - --shared-dir=/var/run/secrets/eks.amazonaws.com/serviceaccount
    - --sandbox=none
    - --cache=auto
    - --migration-on-error=guest-error
    - --migration-mode=find-paths
    command:
    - /usr/libexec/virtiofsd

We have already created a pull request PR#14568 for the hook sidecar implementation.

Additionally, we would like to introduce a new repository in KubeVirt for managing this webhook. To do so, we are using this mailing list and referring to the PR for the new repo setup.

[vrom...@redhat.com]

Hi,

I apologize for the delay in responding to this thread.
Reading this, the suggested path makes sense to me. I think that KubeVirt should have proper support for Kubernetes-style Projected Volumes.

It would enable flexible mounting of service account tokens with custom audiences. We should follow Kubernetes and extend our Volume API to Kubernetes Projected Volumes

However, to properly address this issue, we should have a VEP explaining the problem, discussing the proposed solution, and the alternatives.

Please file an issue and create a corresponding enhancement in the https://github.com/kubevirt/enhancements repo.

Thank you,

Vladik