Why virt-handler and macvtap containers are running as root?

[Xinyan Zhao]

Hello everyone,

We know it is always safer to run containers as non-root users. I found in Kubevirt, most components are already rootless, except for virt-handler, virt-launcher and macvtap.

Wondering does these three containers must have the root privileges? Maybe it is because they need some privileged access to certain resources? 

# macvtap-cni container: 

securityContext: 

    privileged:  true

# virt-luancher container:

securityContext: 

    privileged:  true

# virt-handler container: 

securityContext: 

    privileged:  true

    seLinuxOptions: 

        level: s0

Thanks for your help!

[Roman Mohr]

Hi

On Wed, Nov 10, 2021 at 3:07 AM 'Xinyan Zhao' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hello everyone,

We know it is always safer to run containers as non-root users. I found in Kubevirt, most components are already rootless, except for virt-handler, virt-launcher and macvtap.

Let me respond on a per component basis:

 * virt-handler has to be root. It is our single privileged component by design which has to do sensitive operations.
 * virt-launcher still runs as root by default but this is a temporary state. In fact Lubo (CCed) already introduced the non-root code and we have a feature gate for it (https://github.com/kubevirt/kubevirt/pull/6041, the feature gate in the latest kubevirt release is NonRootExperimental).

 * For macvtap I CCed @Miguel Duarte de Mora Barroso to keep me honest, but like other network-wiring system-components it probably needs extended privileges. It would very likely be able to also work on listing explicit privileges needed instead of a blanko "privileged" boolean.

Best regards,

Roman

 

 

Wondering does these three containers must have the root privileges? Maybe it is because they need some privileged access to certain resources? 

# macvtap-cni container: 

securityContext: 

    privileged:  true

# virt-luancher container:

securityContext: 

    privileged:  true

# virt-handler container: 

securityContext: 

    privileged:  true

    seLinuxOptions: 

        level: s0

Thanks for your help!

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/471b81d4-2412-48ee-bccd-e1cb87950ea3n%40googlegroups.com.

[Luboslav Pivarc]

Hi

On Wed, Nov 10, 2021 at 8:59 AM Roman Mohr <rm...@redhat.com> wrote:

Hi

On Wed, Nov 10, 2021 at 3:07 AM 'Xinyan Zhao' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hello everyone,

We know it is always safer to run containers as non-root users. I found in Kubevirt, most components are already rootless, except for virt-handler, virt-launcher and macvtap.

Let me respond on a per component basis:

 * virt-handler has to be root. It is our single privileged component by design which has to do sensitive operations.
 * virt-launcher still runs as root by default but this is a temporary state. In fact Lubo (CCed) already introduced the non-root code and we have a feature gate for it (https://github.com/kubevirt/kubevirt/pull/6041, the feature gate in the latest kubevirt release is NonRootExperimental).

@Mohr, Roman 

Correct me if I'm wrong but privileged has nothing to do with root/non-root. It doesn't seem to be right to me. I think we need the launcher to be privileged only on ppc64 arch.

[Miguel Duarte de Mora Barroso]

On Wed, Nov 10, 2021 at 10:01 AM Luboslav Pivarc <lpi...@redhat.com> wrote:

Hi

On Wed, Nov 10, 2021 at 8:59 AM Roman Mohr <rm...@redhat.com> wrote:

Hi

On Wed, Nov 10, 2021 at 3:07 AM 'Xinyan Zhao' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hello everyone,

We know it is always safer to run containers as non-root users. I found in Kubevirt, most components are already rootless, except for virt-handler, virt-launcher and macvtap.

Let me respond on a per component basis:

 * virt-handler has to be root. It is our single privileged component by design which has to do sensitive operations.
 * virt-launcher still runs as root by default but this is a temporary state. In fact Lubo (CCed) already introduced the non-root code and we have a feature gate for it (https://github.com/kubevirt/kubevirt/pull/6041, the feature gate in the latest kubevirt release is NonRootExperimental).

@Mohr, Roman 

Correct me if I'm wrong but privileged has nothing to do with root/non-root. It doesn't seem to be right to me. I think we need the launcher to be privileged only on ppc64 arch.

 

 * For macvtap I CCed @Miguel Duarte de Mora Barroso to keep me honest, but like other network-wiring system-components it probably needs extended privileges. It would very likely be able to also work on listing explicit privileges needed instead of a blanko "privileged" boolean.

Macvtap-cni (despite its name) also features an implementation of the device plugin framework, to grant the pod access to the `/dev/tapX` device.

As per [0], "a device plugin must run in a privileged security context". 

[Miguel Duarte de Mora Barroso]

On Wed, Nov 10, 2021 at 10:26 AM Miguel Duarte de Mora Barroso <mdba...@redhat.com> wrote:

On Wed, Nov 10, 2021 at 10:01 AM Luboslav Pivarc <lpi...@redhat.com> wrote:

Hi

On Wed, Nov 10, 2021 at 8:59 AM Roman Mohr <rm...@redhat.com> wrote:

Hi

On Wed, Nov 10, 2021 at 3:07 AM 'Xinyan Zhao' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hello everyone,

We know it is always safer to run containers as non-root users. I found in Kubevirt, most components are already rootless, except for virt-handler, virt-launcher and macvtap.

Let me respond on a per component basis:

 * virt-handler has to be root. It is our single privileged component by design which has to do sensitive operations.
 * virt-launcher still runs as root by default but this is a temporary state. In fact Lubo (CCed) already introduced the non-root code and we have a feature gate for it (https://github.com/kubevirt/kubevirt/pull/6041, the feature gate in the latest kubevirt release is NonRootExperimental).

@Mohr, Roman 

Correct me if I'm wrong but privileged has nothing to do with root/non-root. It doesn't seem to be right to me. I think we need the launcher to be privileged only on ppc64 arch.

 

 * For macvtap I CCed @Miguel Duarte de Mora Barroso to keep me honest, but like other network-wiring system-components it probably needs extended privileges. It would very likely be able to also work on listing explicit privileges needed instead of a blanko "privileged" boolean.

Macvtap-cni (despite its name) also features an implementation of the device plugin framework, to grant the pod access to the `/dev/tapX` device.

As per [0], "a device plugin must run in a privileged security context". 

I forgot to link the document ...

[0] - https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-deployment

[Xinyan Zhao]

Hi,

Thanks all for your clarifications and explanations! That's much clearer to me.

So to summarize, virt-handler and macvtap must have privileges to do some sensitive operations or access certain devices. And for virt-launcher, we could pull the latest non-root effort to have a try (thanks for this effort!).

Although "privileged" and "root" are not related, it doesn't make sense to make the container rootless when it has the overall privilege, right? Like here privileged=true

Best Regards,

Xinyan

[Fabian Deutsch]

On Wed, Nov 10, 2021 at 10:01 AM Luboslav Pivarc <lpi...@redhat.com> wrote:

Hi

On Wed, Nov 10, 2021 at 8:59 AM Roman Mohr <rm...@redhat.com> wrote:

Hi

On Wed, Nov 10, 2021 at 3:07 AM 'Xinyan Zhao' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hello everyone,

We know it is always safer to run containers as non-root users. I found in Kubevirt, most components are already rootless, except for virt-handler, virt-launcher and macvtap.

Let me respond on a per component basis:

 * virt-handler has to be root. It is our single privileged component by design which has to do sensitive operations.
 * virt-launcher still runs as root by default but this is a temporary state. In fact Lubo (CCed) already introduced the non-root code and we have a feature gate for it (https://github.com/kubevirt/kubevirt/pull/6041, the feature gate in the latest kubevirt release is NonRootExperimental).

@Mohr, Roman 

Correct me if I'm wrong but privileged has nothing to do with root/non-root. It doesn't seem to be right to me. I think we need the launcher to be privileged only on ppc64 arch.

Lubo, as you are working on this for a while - do you have any material that could summarize the previous and next steps?

Part of it is, that we are expecting a behavior change at some point.

 

 

 * For macvtap I CCed @Miguel Duarte de Mora Barroso to keep me honest, but like other network-wiring system-components it probably needs extended privileges. It would very likely be able to also work on listing explicit privileges needed instead of a blanko "privileged" boolean.

Best regards,

Roman

 

 

Wondering does these three containers must have the root privileges? Maybe it is because they need some privileged access to certain resources? 

# macvtap-cni container: 

securityContext: 

    privileged:  true

# virt-luancher container:

securityContext: 

    privileged:  true

# virt-handler container: 

securityContext: 

    privileged:  true

    seLinuxOptions: 

        level: s0

Thanks for your help!

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/471b81d4-2412-48ee-bccd-e1cb87950ea3n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAFTR0GXgaNjsqXU4V2A3YGAQ%3Dk%3DCqPvGszT1uHebY24%3DZDW-%2BA%40mail.gmail.com.

[Luboslav Pivarc]

Hey,

On Thu, Nov 11, 2021 at 9:56 AM Fabian Deutsch <fdeu...@redhat.com> wrote:

On Wed, Nov 10, 2021 at 10:01 AM Luboslav Pivarc <lpi...@redhat.com> wrote:

Hi

On Wed, Nov 10, 2021 at 8:59 AM Roman Mohr <rm...@redhat.com> wrote:

Hi

On Wed, Nov 10, 2021 at 3:07 AM 'Xinyan Zhao' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hello everyone,

We know it is always safer to run containers as non-root users. I found in Kubevirt, most components are already rootless, except for virt-handler, virt-launcher and macvtap.

Let me respond on a per component basis:

 * virt-handler has to be root. It is our single privileged component by design which has to do sensitive operations.
 * virt-launcher still runs as root by default but this is a temporary state. In fact Lubo (CCed) already introduced the non-root code and we have a feature gate for it (https://github.com/kubevirt/kubevirt/pull/6041, the feature gate in the latest kubevirt release is NonRootExperimental).

@Mohr, Roman 

Correct me if I'm wrong but privileged has nothing to do with root/non-root. It doesn't seem to be right to me. I think we need the launcher to be privileged only on ppc64 arch.

Lubo, as you are working on this for a while - do you have any material that could summarize the previous and next steps?

Part of it is, that we are expecting a behavior change at some point.

Sure, I shared this document a while ago in a community meeting. It might be sparse but I will try to keep it updated.
https://docs.google.com/document/d/14Wu_-Alo6CJLP6-jJ4ESIhxCaSIPUdOHNtE1My-Cors/edit

[Luboslav Pivarc]

Hey Xinyan,

On Thu, Nov 11, 2021 at 5:52 AM 'Xinyan Zhao' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hi,

Thanks all for your clarifications and explanations! That's much clearer to me.

So to summarize, virt-handler and macvtap must have privileges to do some sensitive operations or access certain devices. And for virt-launcher, we could pull the latest non-root effort to have a try (thanks for this effort!).

Although "privileged" and "root" are not related, it doesn't make sense to make the container rootless when it has the overall privilege, right? Like here privileged=true

IMO it would still be better if the container is running non-root but it doesn't matter because the launcher is not privileged. Can you share how you found out that launcher is privileged?

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/10c6f571-7955-4e53-aade-7a993321c138n%40googlegroups.com.

[Miguel Duarte de Mora Barroso]

On Fri, Nov 19, 2021 at 12:02 PM Luboslav Pivarc <lpi...@redhat.com> wrote:

Hey Xinyan,

On Thu, Nov 11, 2021 at 5:52 AM 'Xinyan Zhao' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hi,

Thanks all for your clarifications and explanations! That's much clearer to me.

So to summarize, virt-handler and macvtap must have privileges to do some sensitive operations or access certain devices. And for virt-launcher, we could pull the latest non-root effort to have a try (thanks for this effort!).

Although "privileged" and "root" are not related, it doesn't make sense to make the container rootless when it has the overall privilege, right? Like here privileged=true

IMO it would still be better if the container is running non-root but it doesn't matter because the launcher is not privileged. Can you share how you found out that launcher is privileged?

The only way (I'm aware of ...) where a launcher pod can be privileged is when it runs on ppc architecture [1].

@Xinyan Zhao Did you mean this scenario ? 

Best Regards,

Xinyan

On Wednesday, November 10, 2021 at 1:27:54 AM UTC-8 Miguel Duarte de Mora Barroso wrote:

On Wed, Nov 10, 2021 at 10:26 AM Miguel Duarte de Mora Barroso <mdba...@redhat.com> wrote:

On Wed, Nov 10, 2021 at 10:01 AM Luboslav Pivarc <lpi...@redhat.com> wrote:

Hi

On Wed, Nov 10, 2021 at 8:59 AM Roman Mohr <rm...@redhat.com> wrote:

Hi

On Wed, Nov 10, 2021 at 3:07 AM 'Xinyan Zhao' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hello everyone,

We know it is always safer to run containers as non-root users. I found in Kubevirt, most components are already rootless, except for virt-handler, virt-launcher and macvtap.

Let me respond on a per component basis:

 * virt-handler has to be root. It is our single privileged component by design which has to do sensitive operations.
 * virt-launcher still runs as root by default but this is a temporary state. In fact Lubo (CCed) already introduced the non-root code and we have a feature gate for it (https://github.com/kubevirt/kubevirt/pull/6041, the feature gate in the latest kubevirt release is NonRootExperimental).

@Mohr, Roman 

Correct me if I'm wrong but privileged has nothing to do with root/non-root. It doesn't seem to be right to me. I think we need the launcher to be privileged only on ppc64 arch.

 

 * For macvtap I CCed @Miguel Duarte de Mora Barroso to keep me honest, but like other network-wiring system-components it probably needs extended privileges. It would very likely be able to also work on listing explicit privileges needed instead of a blanko "privileged" boolean.

Macvtap-cni (despite its name) also features an implementation of the device plugin framework, to grant the pod access to the `/dev/tapX` device.

As per [0], "a device plugin must run in a privileged security context". 

I forgot to link the document ...

[0] - https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-deployment

[1] - https://github.com/kubevirt/kubevirt/blob/ac41c252c42a0770a638dd802e99ab3fde5e999b/pkg/virt-controller/services/template.go#L504

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAFTR0GVrekjc2CBVnOh2EqgMShd_f9qBw0DtPEh2JSEkDVtsnQ%40mail.gmail.com.

[Xinyan Zhao]

Hi Lubo and Miguel,

Thanks for you clarifications! I found it directly form the virt-launcher yaml file:

securityContext:

      capabilities:

        add:

        - NET_BIND_SERVICE

        - NET_ADMIN

        - SYS_NICE

        drop:

        - NET_RAW

      privileged: true

      runAsUser: 0

Here UID=0 and privileged=true. Does it expected?

[Luboslav Pivarc]

Hi,

I see also capability NET_ADMIN. It seems to me you are running a very old version of Kubevirt. It is definitely not expected to have privileged=true.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/e68a53ff-e72b-4f79-ba7d-1c30edbb9393n%40googlegroups.com.

[Xinyan Zhao]

We are using Kubevirt 0.45, I think it is not old... Maybe we made some changes on virt-luancher and lead to this privileged=true?

[Zang Li]

Hi Lubo and Miguel,

Yes we made the change on NET_ADMIN capability and privileged=true locally to get kubevirt 0.45 working for us. Due to the company requirement, we have to use Debian as our base image, so we re-packed the binary on our own.  Do you know why we have to enable those capabilities with the base image change? Or something else caused the issue?

Thanks,

Zang

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/fc8f8234-a547-4734-be17-2f418bdc3cffn%40googlegroups.com.

[Roman Mohr]

On Thu, Dec 2, 2021 at 9:16 PM 'Zang Li' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hi Lubo and Miguel,

Yes we made the change on NET_ADMIN capability and privileged=true locally to get kubevirt 0.45 working for us. Due to the company requirement, we have to use Debian as our base image, so we re-packed the binary on our own.  Do you know why we have to enable those capabilities with the base image change? Or something else caused the issue?

Could you share what you see when it is not set? That may help diagnosing what's missing. Also: If you run the corresponding community build on your Debian nodes, do you see the same issue?

Best regards,

Roman

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAO_S94gG-BrAu-gfhwiiLZvkdRgL1rwz8s19khkiXMu9ehNStA%40mail.gmail.com.

[Luboslav Pivarc]

On Fri, Dec 3, 2021 at 9:42 AM Roman Mohr <rm...@redhat.com> wrote:

On Thu, Dec 2, 2021 at 9:16 PM 'Zang Li' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hi Lubo and Miguel,

Yes we made the change on NET_ADMIN capability and privileged=true locally to get kubevirt 0.45 working for us. Due to the company requirement, we have to use Debian as our base image, so we re-packed the binary on our own.  Do you know why we have to enable those capabilities with the base image change? Or something else caused the issue?

Could you share what you see when it is not set? That may help diagnosing what's missing. Also: If you run the corresponding community build on your Debian nodes, do you see the same issue?

+ at least libvirt version you are building with. There also might be a feature that is requiring privileged for your use case. Kubevirt removed it here https://github.com/kubevirt/kubevirt/commit/69e962dfc7d56674c47b3fadc15d9114b5902063 .  NET_ADMIN was removed here https://github.com/kubevirt/kubevirt/pull/5147/commits/455a083cced8f19bae63e872f5517978fefdb60d

Best regards,

Lubo

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CALDPj7tafKY0vByWtWuzppfm5Ow7yN8Zu-UG8NwQ_x%3DHYRCxqA%40mail.gmail.com.

[Zang Li]

Hi Lubo and Miguel,

Thank you for helping on this! We are using libvirt 7.0.0.  The kubevirt images we built use debian buster as base image, but the host cluster node for our testing is ubuntu 20.04. 

If we change the kubevirt images to the kubevirt oss built ones on the same cluster, the vm creation works. If we add the permissions in our build, the vm creation also works. 

When we moved to kubevirt 0.43.0, if we don't enable NET_ADMIN, we will hit the following errors on vm creation:

Warning SyncFailed 37s (x18 over 6m6s) virt-handler server error. command SyncVMI failed: "LibvirtError(Code=38, Domain=0, Message='Cannot set interface MTU on 'tap0': Operation not permitted')"

Then when we moved to kubevirt 0.45.0, if we don't add the priviledge=true, we hit the following error

{"component":"virt-launcher","level":"info","msg":"Hardware emulation device '/dev/kvm' not present. Using software emulation.","pos":"converter.go:1100","timestamp":"2021-10-12T04:40:04.018745Z"}
{"component":"virt-launcher","level":"info","msg":"In-kernel virtio-net device emulation '/dev/vhost-net' not present. Falling back to QEMU userland emulation.","pos":"converter.go:1114","timestamp":"2021-10-12T04:40:04.018784Z"}
{"component":"virt-launcher","level":"error","msg":"Cannot find 'tc' in path: No such file or directory","pos":"virExec:624","subcomponent":"libvirt","thread":"31","timestamp":"2021-10-12T04:40:04.023000Z"}
{"component":"virt-launcher","level":"error","msg":"Failed to create v1 controller cpu for group: Read-only file system","pos":"virCgroupV1MakeGroup:660","subcomponent":"libvirt","thread":"31","timestamp":"2021-10-12T04:40:04.032000Z"}
{"component":"virt-launcher","level":"info","msg":"Reaped pid 87 with status 0","pos":"virt-launcher.go:549","timestamp":"2021-10-12T04:40:04.033871Z"}
{"component":"virt-launcher","kind":"","level":"error","msg":"Failed to start VirtualMachineInstance with flags 0.","name":"rhel-vm","namespace":"default","pos":"manager.go:814","reason":"virError(Code=38, Domain=54, Message='Failed to create v1 controller cpu for group: Read-only file system')","timestamp":"2021-10-12T04:40:04.234102Z","uid":"8e39c127-da44-4f30-8f1a-51eb8df15b9d"}

Thanks!

Zang

[Miguel Duarte de Mora Barroso]

On Fri, Dec 3, 2021 at 9:29 PM 'Zang Li' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hi Lubo and Miguel,

Thank you for helping on this! We are using libvirt 7.0.0.  The kubevirt images we built use debian buster as base image, but the host cluster node for our testing is ubuntu 20.04. 

If we change the kubevirt images to the kubevirt oss built ones on the same cluster, the vm creation works. If we add the permissions in our build, the vm creation also works. 

When we moved to kubevirt 0.43.0, if we don't enable NET_ADMIN, we will hit the following errors on vm creation:

Warning SyncFailed 37s (x18 over 6m6s) virt-handler server error. command SyncVMI failed: "LibvirtError(Code=38, Domain=0, Message='Cannot set interface MTU on 'tap0': Operation not permitted')"

I think this is bug [2]. 

I didn't see any mention of this bug on debian's changelogs [3] ... I don't know if they backported it into 7.0.0.

[2] - https://bugzilla.redhat.com/show_bug.cgi?id=1905929

[3] - https://metadata.ftp-master.debian.org/changelogs//main/libv/libvirt/libvirt_7.0.0-3_changelog

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAO_S94iwNUemm%3DYHMZdoS%3Dn9M%3D1areCgF2bojb11PpUu%2BaHVTA%40mail.gmail.com.

[Luboslav Pivarc]

For the second one.

You need to make sure you have the correct configuration https://github.com/kubevirt/kubevirt/blob/main/cmd/virt-launcher/qemu.conf#L10 .

Lubo

[Zang Li]

Miguel and Lubo, 

Thank you very much for the pointers! For the first issue, is it a libvirt version issue or a combination of libvirt and debian? 

Thanks,

Zang