Workaround for KubeVirt on oc cluster

[Fabian Deutsch]

Hey,

some of you are running into issues with KubeVirt on minishift and oc cluster,

this is due to [1].

Now, for oc cluster there is this workaround you can use until the fix is landing in 3.11 [2].

- fabian

---

1: https://github.com/openshift/origin/pull/20351

2: https://github.com/fabiand/traviskube/blob/e2f81c5243f4fda94fd9f21de0722107aa355e51/start-cluster#L37

[Fabian Deutsch]

And the workaround for minishisft:

minishift start --driver=none --iso-url centos
# ONCE after creation
minishift ssh sed "/kubeletArguments/ a\ feature-gates:"
/var/lib/minishift/openshift.local.config/node-localhost/node-config.yaml
minishift ssh sed "/feature-gates/ a\ - DevicePlugins: true"
/var/lib/minishift/openshift.local.config/node-localhost/node-config.yaml
# ON EVERY start of minishift
minishift ssh bash - <<EOF
set -x
KUBELET_ROOTFS=\$(sudo docker inspect \$(sudo docker ps | grep
kubelet | cut -d" " -f1) | grep MergedDir | cut -d ":" -f2 | tr -d '"
,')
sudo mkdir -p /var/lib/kubelet/device-plugins
$KUBELET_ROOTFS/var/lib/kubelet/device-plugins
sudo mount -o bind $KUBELET_ROOTFS/var/lib/kubelet/device-plugins
/var/lib/kubelet/device-plugins
sudo ls /var/lib/kubelet/device-plugins
EOF

- fabian

[Steve Gordon]

Hi folks,

I applied this and I did indeed get past the missing /dev/tun and /dev/kvm devices. I can now start my VMI (yay!) however with my basic Cirros VMI based on the registry disk everything appears up, but when I try virtctl console I get this:

 websocket: close 1006 (abnormal closure): unexpected EOF

virtctl VNC just hangs in remote viewer waiting for display (eventually times out after a minute). The VMI shows phase as Running though and nothing else seems abnormal afaict.

VM describe: https://paste.fedoraproject.org/paste/ev588kddE5K1W-SmOHFXMQ

VMI describe: https://paste.fedoraproject.org/paste/1wAP2yCWvv0iJ0LBkUL0Og

launcher pod: https://paste.fedoraproject.org/paste/IL8R1dm44hkdw~fYwLNOjg

compute container logs: https://paste.fedoraproject.org/paste/SahtQGFaAvKDfzlb6mSBgg

volumeregistryvolume logs: https://paste.fedoraproject.org/paste/Mhyw6bnPDYz0xS~SJ74psA

$ virtctl version

Client Version: version.Info{GitVersion:"v0.8.0", GitCommit:"6165ead7201d8b96946fdfcfb026e44b86d3d980", GitTreeState:"clean", BuildDate:"2018-09-05T15:38:18Z", GoVersion:"go1.10", Compiler:"gc", Platform:"linux/amd64"}

Server Version: &version.Info{GitVersion:"v0.8.0", GitCommit:"6165ead7201d8b96946fdfcfb026e44b86d3d980", GitTreeState:"clean", BuildDate:"2018-09-05T15:38:18Z", GoVersion:"go1.10", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0+b81c8f8", GitCommit:"b81c8f8", GitTreeState:"clean", BuildDate:"2018-09-06T15:07:14Z", GoVersion:"go1.9.7", Compiler:"gc", Platform:"linux/amd64"}

Any ideas?

Thanks,

Steve

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubevi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CA%2BPVUaR6QMVC%3DztpQwhKZyRkbUjimqbvhpr_H1bTNUqL2QiJTw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Stephen Gordon,
Principal Product Manager,
Red Hat

[Fabian Deutsch]

On Fri, Sep 7, 2018 at 7:42 PM Steve Gordon <sgo...@redhat.com> wrote:
>
> Hi folks,
>
> I applied this and I did indeed get past the missing /dev/tun and /dev/kvm devices. I can now start my VMI (yay!) however with my basic Cirros VMI based on the registry disk everything appears up, but when I try virtctl console I get this:
>
> websocket: close 1006 (abnormal closure): unexpected EOF
>
> virtctl VNC just hangs in remote viewer waiting for display (eventually times out after a minute). The VMI shows phase as Running though and nothing else seems abnormal afaict.
>
> VM describe: https://paste.fedoraproject.org/paste/ev588kddE5K1W-SmOHFXMQ
> VMI describe: https://paste.fedoraproject.org/paste/1wAP2yCWvv0iJ0LBkUL0Og
> launcher pod: https://paste.fedoraproject.org/paste/IL8R1dm44hkdw~fYwLNOjg
> compute container logs: https://paste.fedoraproject.org/paste/SahtQGFaAvKDfzlb6mSBgg
> volumeregistryvolume logs: https://paste.fedoraproject.org/paste/Mhyw6bnPDYz0xS~SJ74psA
>
> $ virtctl version
> Client Version: version.Info{GitVersion:"v0.8.0", GitCommit:"6165ead7201d8b96946fdfcfb026e44b86d3d980", GitTreeState:"clean", BuildDate:"2018-09-05T15:38:18Z", GoVersion:"go1.10", Compiler:"gc", Platform:"linux/amd64"}
> Server Version: &version.Info{GitVersion:"v0.8.0", GitCommit:"6165ead7201d8b96946fdfcfb026e44b86d3d980", GitTreeState:"clean", BuildDate:"2018-09-05T15:38:18Z", GoVersion:"go1.10", Compiler:"gc", Platform:"linux/amd64"}
> $ kubectl version
> Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
> Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0+b81c8f8", GitCommit:"b81c8f8", GitTreeState:"clean", BuildDate:"2018-09-06T15:07:14Z", GoVersion:"go1.9.7", Compiler:"gc", Platform:"linux/amd64"}
>
> Any ideas?

Not on a first glance.

Adding David, as it's related to the subresources.
maybe the registration went wrong or something like that.

At least we've got CI in that area, and I wonder why it didn't catch it.

- fabian

>> To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

[Steve Gordon]

On Fri, Sep 7, 2018 at 1:42 PM, Steve Gordon <sgo...@redhat.com> wrote:

Hi folks,

I applied this and I did indeed get past the missing /dev/tun and /dev/kvm devices. I can now start my VMI (yay!) however with my basic Cirros VMI based on the registry disk everything appears up, but when I try virtctl console I get this:

 websocket: close 1006 (abnormal closure): unexpected EOF

OK, spent a fair amount of time spelunking with Stuart and Vladik. It turns out what I was missing for minishift deployment is this:

$ oc adm policy add-scc-to-user privileged system:serviceaccount:kube-system:kubevirt-apiserver

In the getting started guide (which I know Fabian pinned back a version while we sort this out) we listed these oc adm commands:

$ oc adm policy add-scc-to-user privileged system:serviceaccount:kube-system:kubevirt-privileged

$ oc adm policy add-scc-to-user privileged system:serviceaccount:kube-system:kubevirt-controller

$ oc adm policy add-scc-to-user privileged system:serviceaccount:kube-system:kubevirt-infra

Theoretically as I understand it the kubevirt-apiserver line supersedes the kubevirt-infra one shown here. I've filed an issue here:

https://github.com/kubevirt/kubevirt.github.io/issues/98

Thanks,

[Fabian Deutsch]

Awesome debugging!

Please also file the issue on the user-guide.

- fabian

[Itamar Heim]

I could find the 3 commands above (slightly different syntax for some reason) in at least these places. the first is still kubevirt-infra, the latter are already api-server:

https://kubevirt.io/get_kubevirt/

https://github.com/kubevirt/demo

https://github.com/kubevirt/user-guide/tree/master/installation

[Federico Simoncelli]

On Fri, Sep 7, 2018 at 2:30 PM Fabian Deutsch <fdeu...@redhat.com> wrote:

And the workaround for minishisft:

  minishift start --driver=none --iso-url centos
  # ONCE after creation
  minishift ssh sed "/kubeletArguments/ a\  feature-gates:"
/var/lib/minishift/openshift.local.config/node-localhost/node-config.yaml
  minishift ssh sed "/feature-gates/ a\  - DevicePlugins: true"
/var/lib/minishift/openshift.local.config/node-localhost/node-config.yaml
  # ON EVERY start of minishift
  minishift ssh bash - <<EOF
  set -x
  KUBELET_ROOTFS=\$(sudo docker inspect \$(sudo docker ps | grep
kubelet | cut -d" " -f1) | grep MergedDir | cut -d ":" -f2 | tr -d '"
,')
  sudo mkdir -p /var/lib/kubelet/device-plugins
$KUBELET_ROOTFS/var/lib/kubelet/device-plugins
  sudo mount -o bind $KUBELET_ROOTFS/var/lib/kubelet/device-plugins
/var/lib/kubelet/device-plugins
  sudo ls /var/lib/kubelet/device-plugins
EOF

Thanks, that helped me!
Actually using minishift v1.23 and oc v3.10 I just needed to do the following:

$ minishift version

minishift v1.23.0+91235ee

$ oc version

oc v3.10.0+dd10d17

kubernetes v1.10.0+b81c8f8

features: Basic-Auth GSSAPI Kerberos SPNEGO

openshift v3.10.0+e3465d0-44

kubernetes v1.10.0+b81c8f8

$ minishift start

$ eval $(minishift oc-env)

$ oc login -u system:admin

$ minishift ssh -- sudo mkdir -p /var/lib/kubelet/device-plugins

$ minishift ssh -- sudo mount -o bind \$\(docker inspect --format '{{.GraphDriver.Data.MergedDir}}' origin\)/var/lib/kubelet/device-plugins /var/lib/kubelet/device-plugins

$ minishift ssh -- ls /var/lib/kubelet/device-plugins

kubelet.sock

kubelet_internal_checkpoint

Then I continued normally on http://kubevirt.io/get_kubevirt/

Not sure if I'll hit some issues going forward but so far the basic tutorial worked fine.

--

Federico

[Fabian Deutsch]

Good to hear - although the current workaround is a PITA.

In general I'll see that we get better automation for

- automated syncs with releases
- automated checks with guides.

Pretty high goals, but this would help us in order to have up to date
and tested docs.

- fabian

> --
> Federico
>

[David Vossel]

On Fri, Sep 7, 2018 at 4:38 PM, Fabian Deutsch <fdeu...@redhat.com> wrote:

On Fri, Sep 7, 2018 at 7:42 PM Steve Gordon <sgo...@redhat.com> wrote:
>
> Hi folks,
>
> I applied this and I did indeed get past the missing /dev/tun and /dev/kvm devices. I can now start my VMI (yay!) however with my basic Cirros VMI based on the registry disk everything appears up, but when I try virtctl console I get this:
>
>  websocket: close 1006 (abnormal closure): unexpected EOF
>
> virtctl VNC just hangs in remote viewer waiting for display (eventually times out after a minute). The VMI shows phase as Running though and nothing else seems abnormal afaict.
>
> VM describe: https://paste.fedoraproject.org/paste/ev588kddE5K1W-SmOHFXMQ
> VMI describe: https://paste.fedoraproject.org/paste/1wAP2yCWvv0iJ0LBkUL0Og
> launcher pod: https://paste.fedoraproject.org/paste/IL8R1dm44hkdw~fYwLNOjg
> compute container logs: https://paste.fedoraproject.org/paste/SahtQGFaAvKDfzlb6mSBgg
> volumeregistryvolume logs: https://paste.fedoraproject.org/paste/Mhyw6bnPDYz0xS~SJ74psA
>
> $ virtctl version
> Client Version: version.Info{GitVersion:"v0.8.0", GitCommit:"6165ead7201d8b96946fdfcfb026e44b86d3d980", GitTreeState:"clean", BuildDate:"2018-09-05T15:38:18Z", GoVersion:"go1.10", Compiler:"gc", Platform:"linux/amd64"}
> Server Version: &version.Info{GitVersion:"v0.8.0", GitCommit:"6165ead7201d8b96946fdfcfb026e44b86d3d980", GitTreeState:"clean", BuildDate:"2018-09-05T15:38:18Z", GoVersion:"go1.10", Compiler:"gc", Platform:"linux/amd64"}
> $ kubectl version
> Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
> Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0+b81c8f8", GitCommit:"b81c8f8", GitTreeState:"clean", BuildDate:"2018-09-06T15:07:14Z", GoVersion:"go1.9.7", Compiler:"gc", Platform:"linux/amd64"}
>
> Any ideas? 

Not on a first glance.

Adding David, as it's related to the subresources.

 

I'd need virt-api debug logs. 

 

>> To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.

[Stuart Gott]

David, just a heads up that we ran this issue to ground on friday. It turned out that the permissions related workaround we published in the user guide was still referring to virt-infra, but that's been retired. Once we added virt-apiserver to the list of priviledged service accounts, Steve was able to connect on console/vnc.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAPjOJFtfGV-HzVpS8nuwSrxwNxh%2BnJD6gD_%3DvgGauU4Sc-R0sg%40mail.gmail.com.

[David Vossel]

On Mon, Sep 10, 2018 at 11:17 AM, Stuart Gott <sg...@redhat.com> wrote:

David, just a heads up that we ran this issue to ground on friday. It turned out that the permissions related workaround we published in the user guide was still referring to virt-infra, but that's been retired. Once we added virt-apiserver to the list of priviledged service accounts, Steve was able to connect on console/vnc.

perfect, thanks Stu!