initContainers

[Alexander Gallego]

Hi Guys, 

https://gist.github.com/8886c951ff7262f442fe5aca587de210

I was trying to add initContainers to the `kind: VirtualMachine` spec with no success. As in they are ignored, nothing happens - no failure on the command line, no logs... doesn't even show in the k8s dashboard as a container. (trying out kubevirt v0.5.0)

I know with the `kind: Pod` it is really helpful for us to have `initContainers:` and something I didn't think about until now. 

Is there a specific reason the VM spec ignores it . Is there a way to add support for initContainers - maybe a code pointer is enough as well ?

I guess probably in the `virt-handler` code?

Thank you!

For completeness below is the gist:

apiVersion: kubevirt.io/v1alpha1

kind: VirtualMachine

metadata:

  creationTimestamp: null

  name: vm-1

  labels:

    foo: legacy

spec:

  initContainers:

  - name: init-helloworld-ab

    image: fedora

    command: ["/bin/bash"]

    args: ["-c", "echo \"${STARTUP_SCRIPT}\" | bash"]

    securityContext:

      privileged: true

    env:

      - name: HELLO_WORLD

        value: "'cruel world!'"

      - name: HOST_IP

        valueFrom:

          fieldRef:

            fieldPath: status.hostIP

      - name: STARTUP_SCRIPT

        value: |

          #! /bin/bash

          set -v

          set -o errexit

          set -o pipefail

          set -o nounset

          env

          function r() {

            local runner=(nsenter -t 1 -m -u -i -n -p --)

            "${runner[@]}" "$@"

          }

          r bash -c "echo 'On host'"

          # while true; do sleep 1; echo 'alex'; done;

  domain:

    devices:

      disks:

      - disk:

          bus: virtio

        name: registrydisk

        volumeName: registryvolume

      - disk:

          bus: virtio

        name: cloudinitdisk

        volumeName: cloudinitvolume

    machine:

      type: ""

    resources:

      requests:

        memory: 1024M

  terminationGracePeriodSeconds: 0

  volumes:

  - name: registryvolume

    registryDisk:

      image: docker.io/senior7515/fedora27qcow2base:latest

  - cloudInitNoCloud:

      userDataBase64: IyEvYmluL2Jhc2gKc2V0IC1ldngKIyBwYXNzPWJvb20gdmlhIG9wZW5zc2wgcGFzc3dkIC0xIGJvb20KcGFzcz0nJDEkQmI2cnNXNEkkTEVUODd0c3RtRkZ1ODhNaEY5eGNIMCcKdXNlcm5hbWU9ImFnYWxsZWdvIgp1c2VyYWRkIC1wICRwYXNzICR1c2VybmFtZQppZiBbWyAkKGdyZXAgIndoZWVsIiAvZXRjL2dyb3VwKSAhPSAiIiBdXTsgdGhlbgogICAgdXNlcm1vZCAtYUcgd2hlZWwgJHVzZXJuYW1lCmZpCg==

    name: cloudinitvolume

status: {}

[Fabian Deutsch]

Hey Alex,

initContainers are not supported on VMs - yet.

Today we technically copy labels and annotations - we'd need to do this whitelisting for every pod feature we want to expose.

We could do it - technically - but could you explain your use case in order to understand what you try to achieve?

Greetings

- fabian

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubevi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/71e6d823-20e9-43c9-8b56-cdf6c296dc55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander Gallego]

On Tue, May 22, 2018 at 4:16 PM, Fabian Deutsch <fdeu...@redhat.com> wrote:

Hey Alex,

initContainers are not supported on VMs - yet.

Today we technically copy labels and annotations - we'd need to do this whitelisting for every pod feature we want to expose.

fair enough.

 

 

We could do it - technically - but could you explain your use case in order to understand what you try to achieve?

I was trying to hack doing a  work around the macvtap mounting on the VM via:

```

sudo ip link add link eth0 name macvtap0 address ma:cc:ad:dd:rr:ee:ss type macvtap mode bridge

ip link set <interface> netns namespace ${PID} 

```

so i can create a macvtap device and test mount it as a proof of concept. but, i do know other teams use that feature.

What I understand from a different team and their use of it is:

A use case is we mount an encrypted volume. Keys get mounted to the init container. the init container does some decryption + some setup + bridges an old dynamic compute use cases w/ k8s. Then starts the main container the user wanted to use. 

So the gist is currently it is use to bridge a different use case to run on k8s.

[Fabian Deutsch]

On Tue, May 22, 2018 at 10:45 PM, Alexander Gallego <galleg...@gmail.com> wrote:

On Tue, May 22, 2018 at 4:16 PM, Fabian Deutsch <fdeu...@redhat.com> wrote:

Hey Alex,

initContainers are not supported on VMs - yet.

Today we technically copy labels and annotations - we'd need to do this whitelisting for every pod feature we want to expose.

fair enough.

 

 

We could do it - technically - but could you explain your use case in order to understand what you try to achieve?

I was trying to hack doing a  work around the macvtap mounting on the VM via:

```

sudo ip link add link eth0 name macvtap0 address ma:cc:ad:dd:rr:ee:ss type macvtap mode bridge

ip link set <interface> netns namespace ${PID} 

```

so i can create a macvtap device and test mount it as a proof of concept.

Nifty.

For the POC you can probably create a new docker image whcih you derive from the launcher pod.

There you can add a script to perform the necessary steps.

 Does that work?

but, i do know other teams use that feature.

What I understand from a different team and their use of it is:

A use case is we mount an encrypted volume. Keys get mounted to the init container. the init container does some decryption + some setup + bridges an old dynamic compute use cases w/ k8s. Then starts the main container the user wanted to use. 

So the gist is currently it is use to bridge a different use case to run on k8s.

Yeah, I see that point.
In general it looks like a practical hack (initContainers).

- fabian

[Itamar Heim]

When i saw this i was thinking of custom hooks (well, some of):

https://www.ovirt.org/documentation/admin-guide/appe-VDSM_and_Hooks/

Greetings

- fabian

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To post to this group, send email to kubevi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/71e6d823-20e9-43c9-8b56-cdf6c296dc55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To post to this group, send email to kubevi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAL%2BiW28sPzCSFjguB7jsduk%3DBB9N5rP2bzHQn0tMV%3Dek1CcwSw%40mail.gmail.com.

[Fabian Deutsch]

Yep.

- fabian

Greetings

- fabian

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.

To post to this group, send email to kubevi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/71e6d823-20e9-43c9-8b56-cdf6c296dc55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.

To post to this group, send email to kubevi...@googlegroups.com.

[kfox11...@gmail.com]

On Tuesday, May 22, 2018 at 1:49:31 PM UTC-7, Fabian Deutsch wrote:

<SNIP>

 

Yeah, I see that point.

In general it looks like a practical hack (initContainers).

- fabian

Though some of us would want to use vm's in a multitenant environment, so ensuring the user couldn't inject something bad for the cluster would be important.

Thanks,
Kevin

 

 

Greetings

- fabian

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

[Fabian Deutsch]

On Tue, May 22, 2018 at 10:55 PM, <kfox11...@gmail.com> wrote:

On Tuesday, May 22, 2018 at 1:49:31 PM UTC-7, Fabian Deutsch wrote:

<SNIP>

 

Yeah, I see that point.

In general it looks like a practical hack (initContainers).

- fabian

Though some of us would want to use vm's in a multitenant environment, so ensuring the user couldn't inject something bad for the cluster would be important.

Good point.

Somebody volunteering to write a nice initContainers support proposal? :)

- fabian

 

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.

To post to this group, send email to kubevi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/ce2d18b3-ce5e-4d99-a482-a27968a641f5%40googlegroups.com.

[Alexander Gallego]

On Tue, May 22, 2018 at 5:00 PM, Fabian Deutsch <fdeu...@redhat.com> wrote:

On Tue, May 22, 2018 at 10:55 PM, <kfox11...@gmail.com> wrote:

On Tuesday, May 22, 2018 at 1:49:31 PM UTC-7, Fabian Deutsch wrote:

<SNIP>

 

Yeah, I see that point.

In general it looks like a practical hack (initContainers).

- fabian

Though some of us would want to use vm's in a multitenant environment, so ensuring the user couldn't inject something bad for the cluster would be important.

Good point.

You should be able to control this via https://kubernetes.io/docs/admin/admission-controllers/#securitycontextdeny 

Plus if you can launch a VM you can launch a Pod and if you can launch a Pod you can launch a different security context if no admission-controller is set. 

so AFAIK you'll have to have that anyway. 

 

--
You received this message because you are subscribed to a topic in the Google Groups "kubevirt-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubevirt-dev/HZ5dbmhwpsg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubevirt-dev+unsubscribe@googlegroups.com.

To post to this group, send email to kubevi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CA%2BPVUaT%3DvcFL-EgRdrFLBJjTjoVFd_SWKWBfg0OdfnKekjJAgw%40mail.gmail.com.

[Itamar Heim]

On 05/22/2018 05:04 PM, Alexander Gallego wrote:

On Tue, May 22, 2018 at 5:00 PM, Fabian Deutsch <fdeu...@redhat.com> wrote:

On Tue, May 22, 2018 at 10:55 PM, <kfox11...@gmail.com> wrote:

On Tuesday, May 22, 2018 at 1:49:31 PM UTC-7, Fabian Deutsch wrote:

<SNIP>

 

Yeah, I see that point.

In general it looks like a practical hack (initContainers).

- fabian

Though some of us would want to use vm's in a multitenant environment, so ensuring the user couldn't inject something bad for the cluster would be important.

Good point.

You should be able to control this via https://kubernetes.io/docs/admin/admission-controllers/#securitycontextdeny 

Plus if you can launch a VM you can launch a Pod and if you can launch a Pod you can launch a different security context if no admission-controller is set. 

so AFAIK you'll have to have that anyway.

I believe OpenShift handles aspects of the multi tenancy, but of course can be done of the box.
I think we're assuming the user shouldn't be able to do more than they could by launching a container, with some extra capabilities we do expose for VMs, but those shouldn't allow crossing tenancy, if implemented.

Thanks,
   Itamar

[Alexander Gallego]

so far the only use case we've found is to do 'bootstrap-y' type of things for initContainers. - i.e.: bridge older use cases onto k8s. This means usually the securityContext: object is escalated to priviledge: true.

 

Thanks,
   Itamar

[Alexander Gallego]

On Tuesday, May 22, 2018 at 4:49:31 PM UTC-4, Fabian Deutsch wrote:

On Tue, May 22, 2018 at 10:45 PM, Alexander Gallego <galleg...@gmail.com> wrote:

On Tue, May 22, 2018 at 4:16 PM, Fabian Deutsch <fdeu...@redhat.com> wrote:

Hey Alex,

initContainers are not supported on VMs - yet.

Today we technically copy labels and annotations - we'd need to do this whitelisting for every pod feature we want to expose.

fair enough.

 

 

We could do it - technically - but could you explain your use case in order to understand what you try to achieve?

I was trying to hack doing a  work around the macvtap mounting on the VM via:

```

sudo ip link add link eth0 name macvtap0 address ma:cc:ad:dd:rr:ee:ss type macvtap mode bridge

ip link set <interface> netns namespace ${PID} 

```

so i can create a macvtap device and test mount it as a proof of concept.

Nifty.

For the POC you can probably create a new docker image whcih you derive from the launcher pod.

There you can add a script to perform the necessary steps.

 Does that work?

Do you know where I can override this image on the vm.yml file. 

it is not immediately obvious to me what the name of the virt-launcher image is called where to override it - so i can push this to my registry for example

 

 

Greetings

- fabian

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

[Alexander Gallego]

On Tuesday, May 22, 2018 at 9:19:57 PM UTC-4, Alexander Gallego wrote:

On Tuesday, May 22, 2018 at 4:49:31 PM UTC-4, Fabian Deutsch wrote:

On Tue, May 22, 2018 at 10:45 PM, Alexander Gallego <galleg...@gmail.com> wrote:

On Tue, May 22, 2018 at 4:16 PM, Fabian Deutsch <fdeu...@redhat.com> wrote:

Hey Alex,

initContainers are not supported on VMs - yet.

Today we technically copy labels and annotations - we'd need to do this whitelisting for every pod feature we want to expose.

fair enough.

 

 

We could do it - technically - but could you explain your use case in order to understand what you try to achieve?

I was trying to hack doing a  work around the macvtap mounting on the VM via:

```

sudo ip link add link eth0 name macvtap0 address ma:cc:ad:dd:rr:ee:ss type macvtap mode bridge

ip link set <interface> netns namespace ${PID} 

```

so i can create a macvtap device and test mount it as a proof of concept.

Nifty.

For the POC you can probably create a new docker image whcih you derive from the launcher pod.

There you can add a script to perform the necessary steps.

 Does that work?

Do you know where I can override this image on the vm.yml file. 

it is not immediately obvious to me what the name of the virt-launcher image is called where to override it - so i can push this to my registry for example

I guess is this: https://hub.docker.com/r/kubevirt/virt-launcher/

but not sure how to override it.

[Alexander Gallego]

On Tuesday, May 22, 2018 at 9:23:32 PM UTC-4, Alexander Gallego wrote:

On Tuesday, May 22, 2018 at 9:19:57 PM UTC-4, Alexander Gallego wrote:

On Tuesday, May 22, 2018 at 4:49:31 PM UTC-4, Fabian Deutsch wrote:

On Tue, May 22, 2018 at 10:45 PM, Alexander Gallego <galleg...@gmail.com> wrote:

On Tue, May 22, 2018 at 4:16 PM, Fabian Deutsch <fdeu...@redhat.com> wrote:

Hey Alex,

initContainers are not supported on VMs - yet.

Today we technically copy labels and annotations - we'd need to do this whitelisting for every pod feature we want to expose.

fair enough.

 

 

We could do it - technically - but could you explain your use case in order to understand what you try to achieve?

I was trying to hack doing a  work around the macvtap mounting on the VM via:

```

sudo ip link add link eth0 name macvtap0 address ma:cc:ad:dd:rr:ee:ss type macvtap mode bridge

ip link set <interface> netns namespace ${PID} 

```

so i can create a macvtap device and test mount it as a proof of concept.

Nifty.

For the POC you can probably create a new docker image whcih you derive from the launcher pod.

There you can add a script to perform the necessary steps.

 Does that work?

Do you know where I can override this image on the vm.yml file. 

it is not immediately obvious to me what the name of the virt-launcher image is called where to override it - so i can push this to my registry for example

I guess is this: https://hub.docker.com/r/kubevirt/virt-launcher/

but not sure how to override it.

 

I apologize for the spam, but I think i got it:

1) build kubevirt locally

2) call build-manifests but before

3) change kubevirt.yml.in and change the --launcher-image to <my_own_launcher_extension>

4) push my own images

5) launch my images via kube apply -f of the templates generated by manifest-templator.go

more or less? 

[Itamar Heim]

On 05/22/2018 07:59 PM, Itamar Heim wrote:
> On 05/22/2018 05:04 PM, Alexander Gallego wrote:
>>
>>
>> On Tue, May 22, 2018 at 5:00 PM, Fabian Deutsch <fdeu...@redhat.com
>> <mailto:fdeu...@redhat.com>> wrote:
>>
>>
>>
>> On Tue, May 22, 2018 at 10:55 PM, <kfox11...@gmail.com
>> <mailto:kfox11...@gmail.com>> wrote:
>>
>>
>> On Tuesday, May 22, 2018 at 1:49:31 PM UTC-7, Fabian Deutsch
>> wrote:
>>
>>
>>
>> <SNIP>
>>
>> Yeah, I see that point.
>> In general it looks like a practical hack (initContainers).
>>
>> - fabian
>>
>>
>> Though some of us would want to use vm's in a multitenant
>> environment, so ensuring the user couldn't inject something
>> bad for the cluster would be important.
>>
>>
>> Good point.
>>
>>
>> You should be able to control this via
>> https://kubernetes.io/docs/admin/admission-controllers/#securitycontextdeny
>>
>>
>> Plus if you can launch a VM you can launch a Pod and if you can launch
>> a Pod you can launch a different security context if no
>> admission-controller is set.
>>
>> so AFAIK you'll have to have that anyway.
>

> I believe OpenShift <https://www.openshift.org/> handles aspects of the

> multi tenancy, but of course can be done of the box.
> I think we're assuming the user shouldn't be able to do more than they
> could by launching a container, with some extra capabilities we do
> expose for VMs, but those shouldn't allow crossing tenancy, if implemented.

Actually, reflecting more on this, custom payload/hooks/etc is something
we would like to enable, but we need to consider what is allowed for the
admin to configure, vs. a regular tenant/user.

[Alexander Gallego]

I think the multi tenancy is being worked on too https://docs.google.com/document/d/1mNL5oCIqtVwXI9piTPMuGArdZH8CA2UFaxHtM5Myp6M/edit#heading=h.kf7ax09rwryf

by a working group. to some extend it would be nice to just plug into the k8s multi tenant setup - wherever that working group goes vs. re-impl'ing it here as well. 

but I agree. 

 

[Fabian Deutsch]

yep - That's it more or less.

You effectively specify the launcher template as an arg to the controller.

You can also change this in the release manifests (github release kubevirt.yaml).

The flow I thought of:
1. You could build your own docker image based on the docker.io/kubevirt/virt-launcher image
2. add your workarounds to the image
3. push it to your registry

4. change the launcher tempalte in the controller flags in kubevirt.yaml to use your custom image in the custom registry

But building from scratch works as well - nice that it worked :)

- fabian

 

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.

To post to this group, send email to kubevi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/a540a426-356b-481b-8424-a295a583fb5d%40googlegroups.com.

[Alexander Gallego]

So it doesn't because they have to run *before* the virt-launcher container launches in an elevated security context i.e.: securityContext.priviledge: true

I put up a quick PR to start some discussion https://github.com/kubevirt/kubevirt/pull/1070 - just trying to play around w/ the source to see if i can get something up and running ha. 

Turned out that initContainers are used quite a bit - for legacy reasons to set unique strings and the like on other use cases. 

Let me know if this would be an avenue that is worth exploring at the code level. 

Thanks!

 

- fabian

 

[Itamar Heim]

On 05/23/2018 04:34 PM, Alexander Gallego wrote:
...

>
> The flow I thought of:
> 1. You could build your own docker image based on the
> docker.io/kubevirt/virt-launcher

> <http://docker.io/kubevirt/virt-launcher> image

> 2. add your workarounds to the image
> 3. push it to your registry
> 4. change the launcher tempalte in the controller flags in
> kubevirt.yaml to use your custom image in the custom registry
>
> But building from scratch works as well - nice that it worked :)
>
>
> So it doesn't because they have to run *before* the virt-launcher
> container launches in an elevated security context i.e.:
> securityContext.priviledge: true
>
> I put up a quick PR to start some
> discussion https://github.com/kubevirt/kubevirt/pull/1070 - just trying
> to play around w/ the source to see if i can get something up and
> running ha.
>
> Turned out that initContainers are used quite a bit - for legacy reasons
> to set unique strings and the like on other use cases.
>
> Let me know if this would be an avenue that is worth exploring at the
> code level.

I think providing ways to add functionality without changing the
code/rebuilding[1], I think worth while.
Would be good though to describe:
- what can/can't be done with it
- security context it is running on / ability to influence the system,
as this is a payload injected by the end user
- if security is not "its just a non privileged container", potentially
needed to be disabled by default and admin needs to enable via config

makes sense?

Thanks,
Itamar

[1] I'm aware its actually your use case, but assuming this can be used
for less complex customization as well.

[Alexander Gallego]

Thanks Itamar, let me write a doc and answer your questions with enough details tom. 

I think it is worth exploring the tradeoffs as well. Will post link here as well.

[Alexander Gallego]

For completeness here is the link to the google doc: https://docs.google.com/document/d/1xwRs9X92ogFB5MaKZo-yFt0i55IYf4kaHx0w4xyEUfM/edit#

@fabian and @rohmr - should we use the github pull request to go forward?

[Alexander Gallego]

typo! i meant to write @rmohr

[Itamar Heim]

On 05/24/2018 11:10 AM, Alexander Gallego wrote:
>
>
> On Wednesday, May 23, 2018 at 8:10:55 PM UTC-4, Alexander Gallego wrote:
>
>
>
> On Wed, May 23, 2018 at 6:45 PM, Itamar Heim <ih...@redhat.com
> <mailto:ih...@redhat.com>> wrote:
>
> On 05/23/2018 04:34 PM, Alexander Gallego wrote:
> ...
>
>
>     The flow I thought of:
>     1. You could build your own docker image based on the
> docker.io/kubevirt/virt-launcher
> <http://docker.io/kubevirt/virt-launcher>

>     <http://docker.io/kubevirt/virt-launcher

I think the general context is InitContainers are great to do features
not available yet, etc.
The main question for me is if they can provide value without always
needing to be privliged, and does that mean the user gets to inject a
container which is privileged?

Thanks,
Itamar

[Alexander Gallego]

Privilege is something the user sets on the security context.

i.e.:

```

  initContainers:

  - name: init-1

    image: ubuntu

    command: ["/bin/bash"]

    args: ["-c", "echo \"${STARTUP_SCRIPT}\" | bash"]

    securityContext:

      privileged: true

 

```

So it is up to the user to choose that or not. However, we don't have to decide for the user, we just have to enable them.

I haven't used them w/out needing to be privileged but I can envision a configuration_management container that could fetch some configs from git for example 

not needing to be privileged in the execution of the container. i.e.: the secrets are embedded in the container itself but from k8s perspective it is not privileged.

Does that help? 

 

Thanks,
   Itamar

[Itamar Heim]

On 05/24/2018 02:21 PM, Alexander Gallego wrote:
>
>
> On Thu, May 24, 2018 at 1:46 PM, Itamar Heim <ih...@redhat.com
> <mailto:ih...@redhat.com>> wrote:
>
> On 05/24/2018 11:10 AM, Alexander Gallego wrote:
>
>
>
> On Wednesday, May 23, 2018 at 8:10:55 PM UTC-4, Alexander
> Gallego wrote:
>
>
>
>     On Wed, May 23, 2018 at 6:45 PM, Itamar Heim
> <ih...@redhat.com <mailto:ih...@redhat.com>

yes. it just means we trust the user to run privileged code, and they
choose which container they run with what privileged is because they are
security conscious.
I just wonder how many environment allow users to choose to run
privileged containers.

[Alexander Gallego]

I can have a follow-on patch that protects initContainers w/ an 

`if defined in kubevirt.yml: 

    log.warning "Kubevirt installation has disabled initContainers"

    vm.spec.InitContainers = []k8sv1.Container 

Though in general, that's a hack and not really a fix in any form since the user can just run a Pod with constraints to run on the host and securityContext.privilege = true. 

 

I just wonder how many environment allow users to choose to run privileged containers.

Two things:

1) Is there a way to mark features as 'beta' or smth like that - until they graduate - i think it's gna be hard to tell without real user exposure

2) If users have a need, they can ask for a protection and I can put the ` if defined in kubevirt.yml ` type logic in the spec. 

[Marcus]

It's an interesting way to allow administrators to 'tack on' functionality to Kubevirt.  In general, I don't like the idea of allowing users to reach in and manipulate what is going on at the pod level in Kubevirt, it requires some knowledge of kubevirt's launcher. I'd probably opt to hide the VM pod altogether from the user, actually, and have them interact with only the Kubevirt CRDs.

I do understand the temptation to create a 'hook' here, but I would expect external things to break often as Kubevirt is developed on. It will require vigilance on the maintainer of the initContainer.

I agree that admission controllers could be used to block people from running initContainers if desired, and that it could also be a configurable in Kubevirt.

In general I'd like to see the launcher pod's permissions reduced. I've played with removing privileged mode and right now it needs mknod (which is basically full admin capability) and network admin. I think that this may make it harder to reduce privileges down the road if people become accustomed to adding all sorts of ad-hoc admin functionality into the launcher pod. We don't allow users to create privileged containers at all, in general, and there was some OPA tweaking required to allow Kubevirt to do so on a user's behalf.

The encrypted volume thing is definitely a use case of interest, though I think the direction I'm looking at is going to be cluster-wide and automatic to all pods using a certain storage class, Kubevirt or not, rather than user-created. That only works if you own the cluster, though, can't do that in someone else's.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.

To post to this group, send email to kubevi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/fa9bc252-dd00-c559-8aa2-28c7e5743164%40redhat.com.

[Itamar Heim]

I'm kind of "all for hooks", but, "defined by the admin, not the user",
to avoid security/compat issues.
I can see adding support for initContainers, if currently say, disabled
by default, and the admin need to change a config for it to enable it,
as it allows an end user to manipulate the VM flow, it shouldn't be
enabled by default, especially when we are still launching privileged.

> On Thu, May 24, 2018 at 11:46 AM, Itamar Heim <ih...@redhat.com
> <mailto:ih...@redhat.com>> wrote:
>
> On 05/24/2018 11:10 AM, Alexander Gallego wrote:
>
>
>
> On Wednesday, May 23, 2018 at 8:10:55 PM UTC-4, Alexander
> Gallego wrote:
>
>
>
>     On Wed, May 23, 2018 at 6:45 PM, Itamar Heim
> <ih...@redhat.com <mailto:ih...@redhat.com>

> send an email to kubevirt-dev...@googlegroups.com
> <mailto:kubevirt-dev%2Bunsu...@googlegroups.com>.

> To post to this group, send email to kubevi...@googlegroups.com

> <mailto:kubevi...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kubevirt-dev/fa9bc252-dd00-c559-8aa2-28c7e5743164%40redhat.com

> <https://groups.google.com/d/msgid/kubevirt-dev/fa9bc252-dd00-c559-8aa2-28c7e5743164%40redhat.com>.

>
> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.
>
>

[Alexander Gallego]

On Thu, May 24, 2018 at 5:11 PM, Itamar Heim <ih...@redhat.com> wrote:

On 05/24/2018 04:53 PM, Marcus wrote:

It's an interesting way to allow administrators to 'tack on' functionality to Kubevirt.  In general, I don't like the idea of allowing users to reach in and manipulate what is going on at the pod level in Kubevirt, it requires some knowledge of kubevirt's launcher. I'd probably opt to hide the VM pod altogether from the user, actually, and have them interact with only the Kubevirt CRDs.

I do understand the temptation to create a 'hook' here, but I would expect external things to break often as Kubevirt is developed on. It will require vigilance on the maintainer of the initContainer.

I agree that admission controllers could be used to block people from running initContainers if desired, and that it could also be a configurable in Kubevirt.

In general I'd like to see the launcher pod's permissions reduced. I've played with removing privileged mode and right now it needs mknod (which is basically full admin capability) and network admin. I think that this may make it harder to reduce privileges down the road if people become accustomed to adding all sorts of ad-hoc admin functionality into the launcher pod. We don't allow users to create privileged containers at all, in general, and there was some OPA tweaking required to allow Kubevirt to do so on a user's behalf.

The encrypted volume thing is definitely a use case of interest, though I think the direction I'm looking at is going to be cluster-wide and automatic to all pods using a certain storage class, Kubevirt or not, rather than user-created. That only works if you own the cluster, though, can't do that in someone else's.

I'm kind of "all for hooks", but, "defined by the admin, not the user", to avoid security/compat issues.
I can see adding support for initContainers, if currently say, disabled by default, and the admin need to change a config for it to enable it, as it allows an end user to manipulate the VM flow, it shouldn't be enabled by default, especially when we are still launching privileged.

One can have a CRD for 

```

kind: VirtualMachine

```

and one for

```

kind: AdminVM

```

or smth like that. 

Additionally I can have my own operator and proxy the calls to kubevirt - however this would require the ability to patch the Pod definition which I thought I had tried to do, but couldn't figure out how to do for VMs  

Last, we can have a gRPC protocol for extending the Pod definitions, where the default Service included as part of KubeVirt would simply strip away all the containers with securityContext.privilege = true

though it does increase the footprint of the number of services  / adds a lot fo complexity to the project which already has like 4 service layers away from the VM....... 

@itamar, what are you envisioning for 'admin only' 

Do ppl offer the raw k8s api to clients as the default? - i.e.: no proxy of any form / validation of any form?  still new to the project but this seems to me more like an admin tool from the ground up.

 

 

    send an email to kubevirt-dev+unsubscribe@googlegroups.com

    <mailto:kubevirt-dev%2Bunsubscri...@googlegroups.com>.

    To post to this group, send email to kubevi...@googlegroups.com

    <mailto:kubevirt-dev@googlegroups.com>.

[Marcus]

I think a mutating admission controller could patch pods, virt-launcher or not, if you haven't already looked at that.

Regarding admin-only, I think it's the idea that anyone could apply a VM yaml that has an 'initContainers', which calls a container that modifies the behavior of the virt-launcher pod.  It's more palatable if it were something the admin could add to virt-launcher, but the user had no control over. Imagine perhaps various initContainer 'plugins' that could be listed on the virt-controller CLI so it would apply them when it deploys instances of virt-launcher. User has no control but the admin does.

[Alexander Gallego]

On Thu, May 24, 2018 at 6:10 PM, Marcus <shad...@gmail.com> wrote:

I think a mutating admission controller could patch pods, virt-launcher or not, if you haven't already looked at that.

Thanks will do! (but ... kindly, no top posting pls ! :) )

 

Regarding admin-only, I think it's the idea that anyone could apply a VM yaml that has an 'initContainers', which calls a container that modifies the behavior of the virt-launcher pod.  It's more palatable if it were something the admin could add to virt-launcher, but the user had no control over. Imagine perhaps various initContainer 'plugins' that could be listed on the virt-controller CLI so it would apply them when it deploys instances of virt-launcher. User has no control but the admin does.

Maybe. I'd disagree, this would  introduce a new entire failure domain and complicated the comms protocol. Wouldn't you consider this a major architecture change? - note that initContainers plays nicely w/ the rest of k8s 

* what do do on timeout? fail altogether?

* what to do if network partition?

* what to do on invalid certificates - how long do you keep them around / rotate them for the comms - do you put them under the same CA? 

* what to do to enhance change / libvirtd.sh  - currently this is easy just pass a file via /var/run/kubvirt-private/${namespace}/file-arguments for example

just a couple of the top of my head. This would probably be rated on the last resolution.

I like the idea of Jess Frazz and the multi-tenancy group of copying the `clone() ` syscall at the cluster level here. 

https://blog.jessfraz.com/post/hard-multi-tenancy-in-kubernetes/

- Likely in the near future we would be able to attach ourselves to their solutions w.r.t permissioning of elevated security contexts. 

[Itamar Heim]

On 05/24/2018 06:02 PM, Alexander Gallego wrote:
>
>
> On Thu, May 24, 2018 at 5:11 PM, Itamar Heim <ih...@redhat.com

CRD for me is an "admin" feature, not an end-user one (but i could be
wrong)?

>
> ```

> kind: VirtualMachine
> ```
>
> and one for
>
> ```
> kind: AdminVM
> ```
>
> or smth like that.
>
> Additionally I can have my own operator and proxy the calls to kubevirt
> - however this would require the ability to patch the Pod definition
> which I thought I had tried to do, but couldn't figure out how to do for
> VMs
>
> Last, we can have a gRPC protocol for extending the Pod definitions,
> where the default Service included as part of KubeVirt would simply
> strip away all the containers with securityContext.privilege = true
>
> though it does increase the footprint of the number of services  / adds
> a lot fo complexity to the project which already has like 4 service
> layers away from the VM.......
>

this starting to sound complex. I think this can be done simply with
some basic enablement allowing to affect the flow/libvirt xml/etc. (gave
an example in the other thread).

> @itamar, what are you envisioning for 'admin only'

I think marcus answered this same as I would.

>
> Do ppl offer the raw k8s api to clients as the default? - i.e.: no proxy
> of any form / validation of any form?  still new to the project but this
> seems to me more like an admin tool from the ground up.

So, one can use Kubernetes in many ways of course.
When I look at OpenShift, then yes, direct access to the Kubernetes API
for the users.
So if your users are all behind an application/proxy/etc. - you are the
kubernetes admin and you can choose how to configure it.
if your users access kubernetes directly, then they should not be able
to influence flows escalating them to privileged containers.
(and even just manipulating the libvirt xml without extra privileges may
not be wanted to be enabled to an end user, rather only for specific
fields the admin expects).

makes sense?

Thanks,
Itamar

>
>
>
> On Thu, May 24, 2018 at 11:46 AM, Itamar Heim <ih...@redhat.com

>     send an email to kubevirt-dev...@googlegroups.com
> <mailto:kubevirt-dev%2Bunsu...@googlegroups.com>
>     <mailto:kubevirt-dev%2Bunsu...@googlegroups.com
> <mailto:kubevirt-dev%252Buns...@googlegroups.com>>.

>     To post to this group, send email to

> kubevi...@googlegroups.com <mailto:kubevi...@googlegroups.com>
>     <mailto:kubevi...@googlegroups.com
> <mailto:kubevi...@googlegroups.com>>.

[Fabian Deutsch]

On Thu, May 24, 2018 at 10:53 PM, Marcus <shad...@gmail.com> wrote:

It's an interesting way to allow administrators to 'tack on' functionality to Kubevirt.  In general, I don't like the idea of allowing users to reach in and manipulate what is going on at the pod level in Kubevirt, it requires some knowledge of kubevirt's launcher. I'd probably opt to hide the VM pod altogether from the user, actually, and have them interact with only the Kubevirt CRDs.

I do understand the temptation to create a 'hook' here, but I would expect external things to break often as Kubevirt is developed on. It will require vigilance on the maintainer of the initContainer.

I agree that admission controllers could be used to block people from running initContainers if desired, and that it could also be a configurable in Kubevirt.

In general I'd like to see the launcher pod's permissions reduced. I've played with removing privileged mode and right now it needs mknod (which is basically full admin capability) and network admin. I think that this may make it harder to reduce privileges down the road if people become accustomed to adding all sorts of ad-hoc admin functionality into the launcher pod. We don't allow users to create privileged containers at all, in general, and there was some OPA tweaking required to allow Kubevirt to do so on a user's behalf.

Just a side note here - We are also looking at reducing the launcher pod privileges.

A device plugin for KVM should solve the mknod problem, but CAP_NET_ADMIN probably needs to stay for now. (But probably at least not privileged anymore).

 

The encrypted volume thing is definitely a use case of interest, though I think the direction I'm looking at is going to be cluster-wide and automatic to all pods using a certain storage class, Kubevirt or not, rather than user-created. That only works if you own the cluster, though, can't do that in someone else's.

Good thing to look at solving it in the Kube layer.

An alternative thing if you want encrypted volumes is to use encryption of disks within the guest.

Here it could actually help if secrets could be used with VMs in order to provide the encryption key during boot.

- fabian

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CALFpzo5K6G-k87spgx9w2fWs7sxTDoLaFQzVN%3DRN7gATXnZW-A%40mail.gmail.com.

[Marcus]

On Thu, May 24, 2018 at 4:31 PM, Alexander Gallego <galleg...@gmail.com> wrote:

On Thu, May 24, 2018 at 6:10 PM, Marcus <shad...@gmail.com> wrote:

I think a mutating admission controller could patch pods, virt-launcher or not, if you haven't already looked at that.

Thanks will do! (but ... kindly, no top posting pls ! :) )

 

Regarding admin-only, I think it's the idea that anyone could apply a VM yaml that has an 'initContainers', which calls a container that modifies the behavior of the virt-launcher pod.  It's more palatable if it were something the admin could add to virt-launcher, but the user had no control over. Imagine perhaps various initContainer 'plugins' that could be listed on the virt-controller CLI so it would apply them when it deploys instances of virt-launcher. User has no control but the admin does.

Maybe. I'd disagree, this would  introduce a new entire failure domain and complicated the comms protocol. Wouldn't you consider this a major architecture change? - note that initContainers plays nicely w/ the rest of k8s 

* what do do on timeout? fail altogether?

* what to do if network partition?

* what to do on invalid certificates - how long do you keep them around / rotate them for the comms - do you put them under the same CA?  

* what to do to enhance change / libvirtd.sh  - currently this is easy just pass a file via /var/run/kubvirt-private/${namespace}/file-arguments for example

There are various designs for cert management, I won't(and probably can't) go into the details, but it's sort of beside the point in that it isn't a design option we would consider for our clusters. We want to hide the implementation of the VM as much as possible.

I'm only speaking for my own use cases here, of course. Note I'm not necessarily against Kubevirt being capable of providing encryption, or even admin hooks if cluster admins want to tweak how things are done, I just think the internals should not be open to end users. For example, what if Kubevirt decided to stop using Libvirt, or the work from Kata/Hyper/Clear can somehow be leveraged in the future? it's much more difficult to make architectural changes if users are down working in the implementation.