no keyboard or mouse in VNC console

[Seth Galitzer]

Running ganeti 2.15.2 on debian 9.3 (stretch) using kvm. I can get remote console via VNC to my instances, but I cannot interact with them. Can somebody tell me how to correct this behavior? I've been looking through documentation and am not seeing anything that addresses this.

Thanks.

Seth

[candlerb]

> I can get remote console via VNC to my instances, but I cannot interact with them.

What exactly does this mean?  Can you *see* a login prompt, or do you just see a blank screen?  Are these instances Linux (what flavour) or Windows or something else?

It always works for me, but I have seen cases where different VNC clients didn't work.  Try using tigerVNC, that always works well.

Cheers,

Brian.

[Seth Galitzer]

I can see the console of the instance, so yes boot messages and login prompt for linux instances and lock screen for windows instances. I've had an old 2.5.2 cluster that I'm finally getting around to replacing and never had problems with it. I've been using vncviewer, reminna, and KRDC from linux to try to connect. It's acting like I'm connecting view-only, but I'm not using these clients any differently than I have in the past.

Thanks.

Seth

[candlerb]

OK, then I don't know what the issue is - I have neither seen it myself nor heard anyone else report such a problem.  Currently I'm on 2.15.2 under Ubuntu 16.04.

I would be inclined to try a tcpdump of the VNC port, at both client and server side simultaneously:

(1) to prove whether keystrokes and mouse movements are actually being sent

(2) to eliminate problems with the TCP session being stuck (e.g. if you see repeated retransmits of the same packet - as possibly could happen with MTU issue - or packets being sent at one side not being received at the other)

Also check the kvm command line with ps, to see if it seems to have the right vnc options.

    qemu-system-x86_64 -enable-kvm ... -usb -usbdevice tablet -vnc :5100 ...

If ganeti is starting kvm with the right options, and the traffic is getting through, then it would seem to be a problem with kvm.  But I'm more inclined to suspect the client or client-to-server communication.

You're connecting directly, not via an ssh tunnel or somesuch?  I set vnc_bind_address to 0.0.0.0 at the cluster level.

# gnt-cluster info | grep vnc

    vnc_bind_address: 0.0.0.0

    vnc_password_file:

    vnc_tls: False

    vnc_x509_path:

    vnc_x509_verify: False

# gnt-instance info <somename> | grep vnc

  console connection: vnc to <node>:11000 (display 5100)

    vnc_bind_address: default (0.0.0.0)

    vnc_password_file: default ()

    vnc_tls: default (False)

    vnc_x509_path: default ()

    vnc_x509_verify: default (False)

Then: vncviewer <node>:11000

Finally: check client options again. "man qemu-system-x86_64" talks about ctrl-alt to toggle keyboard/mouse grab, but I don't think that applies to VNC.  If you run vncviewer without any arguments, and go to the "Input" tab, you can see if you have view only selected.

[Seth Galitzer]

Tried tcpdump on both client and server, no packets dropped, same packet count on both sides. I'm not an expert packet analyzer, but I don't think I saw any retransmits. There are no firewall rules blocking access on client or server hosts.

For reference, here is an invocation of qemu for an instance: https://pastebin.com/UWZPbQ4d

Here is the config for that instance: https://pastebin.com/KvEXEuK1

Here is the config for the cluster: https://pastebin.com/485nhinC

All of these have been slightly sanitized, but you can see everything looks as it should.

Looking at dmesg within the instance shows that the kernel does recognize an attached AT keyboard. Linux instances all have virtio modules loaded: virtio_ring,virtio_pci,virtio_net,virtio_blk,virtio_balloon.

I did discover yesterday that I can actually get mouse control through vnc. I was able to connect to my windows instances and use the on-screen keyboard to login. From there, I started notepad and checked to see if any keys actually worked and discovered that some did. I was able to get some typographic chars, but using the shift key with them had no effect. I was also able to get numeric chars only by holding down the shift key with those keys, but got no typographic chars from those keys. Because of this, I thought maybe the keymap was broken and tried specifying us-en, but that unfortunately didn't help.

I have tested from multiple clients with multiple client vnc applications, all with the same results. View-only is not selected in any of them.

Version of qemu in debian stretch is 2.8, but I have yet to find a bug report on this. My old cluster is on debian wheezy, with qemu 1.2.

So I'm still stuck on a problem that absolutely shouldn't be happening. Serial terminal works fine, but that doesn't help me if I'm trying to interrupt the boot process of the instance, eg for pxe-booting a diagnostic image, and doesn't help me for windows instances.

Any other suggestions are welcome.

Thanks.

Seth

[Ansgar Jazdzewski]

hi,

did you try to run a vm on the host that is not managed by ganeti,
just to make shure it is not a qemu bug.
also make sure you have the tablet mode on.

to enter the the grub via serial consol you have to:
1. configure grub to make use of the serial consol
2. use gnt-instance start --pause NAME && gnt-instance console NAME

hope it helps, a bit,
Ansgar

[Seth Galitzer]

Can libvirt tools be installed happily alongside ganeti tools? I don't want to break my production cluster with testing.

I have enabled tablet mode for usb_mouse and it has not helped.

Thanks for the tip on pausing boot to get to grub, but I need to be able to get to pxe pre-boot environment and also desktop for Windows instances and I don't think I can get those with serial console.

Thanks.

Seth

[Phil Regnauld]

Seth Galitzer (sethgali) writes:
> Can libvirt tools be installed happily alongside ganeti tools? I don't want
> to break my production cluster with testing.

Yes, you can do that without a problem - Ganeti only cares about
the hypervisor instances it knows about (and vice versa for KVM/Libvirt.

[Seth Galitzer]

I was able to create a VM using libvirt directly and had the same result. VNC was active and I could see the console, but keyboard did not work.

Seth

[Seth Galitzer]

I have now tried to stand up a new single-node cluster on centos7 and am seeing the exact same behavior. Version of qemu-kvm on this distro is 1.5.3, and is 2.8 on debian stretch. I am completely out of ideas where to go next on this. Can anybody else at least confirm this issue? Clean install of OS and ganeti 2.15.2. Please advise what I should be looking at to identify the breakdown. At this point is has to be either something really obscure and esoteric or completely stupid simple that is being overlooked, but I'm stumped. This is a deal breaker for me. I've been using ganeti 2.5.2 happily for several years, but this upgrade is killing it for me. I implore you for help here.

Thanks.

Seth

[sascha...@web.de]

Hi Seth,

as I understand, you allready tested with libvirt and got the same result?
So this is not a ganeti problem? Then I agree, it must be something strange:

* some nifty security setting on the server (PaX, apparmor, SElinux)
* a broken vnc client

The simples test-case is: start qemu by hand:

$ qemu-system-x86_64 -m 1024 -display vnc=0.0.0.0:0 -cdrom /path/to/live.iso

(use something like systemrescuecd for live.iso)

and then connect to VNC

$ vncviewer <host>::5900

I use xtightvncviewer (the debian package name for tightvnc).

If you still have no keyboard, we have a test-case for further
investigation. Watch if qemu-system-x86_64 has any output, or take a look
into system logs.

Thansk, Sascha.

[Seth Galitzer]

Thank you for the suggestion. I tried this and got the same result. No messages were logged either on the server or the client and I'm not so far seeing a way to make them more verbose. I agree, I don't think this necessarily a ganeti problem, but the fact is that it happens on two systems with different qemu and kernel versions, and ganeti is the only thing that is the same on both. The host OS installs are fairly vanilla, nothing special that should touch qemu, afaik. On the debian host, I disabled apparmor, and on the centos host, I set SELinux to permissive. Any other debugging suggestions are welcome.

Thanks.

Seth

[candlerb]

You said "ganeti is the only thing that is the same on both", but it sounds to me like you're using the same VNC client device, and probably the same network between the devices.

I can only suggest:

1. Try a completely different PC or laptop as the VNC client device.  Make sure it's running something mainstream (Windows or Ubuntu), make sure it doesn't have any funny keyboard mapping, and use TigerVNC.

If that still doesn't work, then:

2. Bring up a server running Ubuntu 16.04 and standalone qemu, and use VNC to connect to that.

If that still doesn't work, then:

3. Make a point-to-point ethernet connection between the qemu machine and the VNC client (say, statically configure 10.0.0.1 and 10.0.0.2) and try again.  Just a CAT5 cable, no switches or antyhing else between.

This is definitely not a ganeti problem though - and if you want to convince yourself of that, uninstall ganeti and then re-run your qemu/kvm test. So it's worth asking on a qemu mailing list if anyone has seen anything similar.

[Georg Faerber]

Hi Seth,

On 18-03-09 14:48:03, Seth Galitzer wrote:
> Thank you for the suggestion. I tried this and got the same result. No
> messages were logged either on the server or the client and I'm not so far
> seeing a way to make them more verbose. I agree, I don't think this
> necessarily a ganeti problem, but the fact is that it happens on two
> systems with different qemu and kernel versions, and ganeti is the only
> thing that is the same on both. The host OS installs are fairly vanilla,
> nothing special that should touch qemu, afaik. On the debian host, I
> disabled apparmor, and on the centos host, I set SELinux to permissive. Any
> other debugging suggestions are welcome.

Could you please record the packets sent by your VNC client, for example
via tcpdump? I would like to have a look..

Cheers,
Georg

[sascha...@web.de]

Hi Seth,

would you mind to try a "clean" client? I.e. boot systemrescuecd on your
client and then run:

vncviewer <host>::5900

I'm pretty sure, that the debian ganeti package has no side effect on
qemu+vnc. You can even follow candlerb's suggestion and remove ganeti from
the host.

An other way could be to install the qemu-package on your linux client
computer and run the test there.

Thanks, Sascha.