KVM migration issues

[ricardo]

Hello everyone,

I wonder if anyone have successfully run ganeti with kvm and debian
lenny 5.0.4?

For me it seems to work but the live migration. I get the following
error:

Could not pre-migrate instance vm.host: Failed to accept instance:
Failed to start instance vm.host: exited with exit code 1 (qemu:
loading initrd (0x65bad8 bytes) at 0x0000000017994000
Migration failed rc=201

I even added the command output to see if the "incoming" parameter was
being added correctly for my configuration

(cmd:[u'/usr/bin/kvm', u'-name', u'vm.host', u'-m', 384, u'-smp', 1,
u'-pidfile', u'/var/run/ganeti/kvm-hypervisor/pid/vm.host', u'-
daemonize', u'-boot', u'c', u'-drive', u'file=/var/run/ganeti/instance-
disks/vm.host:0,format=raw,if=virtio,boot=on', u'-kernel', u'/boot/
vmlinuz-2.6-kvmU', u'-initrd', u'/boot/initrd.img-2.6-kvmU', u'-
append', u'root=/dev/vda1 ro console=ttyS0,38400', u'-vnc',
u'127.0.0.1:5100', u'-usbdevice', u'tablet', u'-monitor', u'unix:/var/
run/ganeti/kvm-hypervisor/ctrl/vm.host.monitor,server,nowait', u'-
serial', u'unix:/var/run/ganeti/kvm-hypervisor/ctrl/
vm.host.serial,server,nowait', u'-localtime', '-net',
u'nic,vlan=0,macaddr=00:16:3e:d9:76:53,model=virtio', '-net',
'tap,vlan=0,script=/tmp/tmpoBz2kO', '-incoming', u'tcp:
10.1.1.12:8102'])

For me it seems fine.

The documentation from the project is really complete and to be honest
the only change I had to make because kvm wasn't working at all was to
change the default mac address prefix used to "00:16:3e" as the
"00:00:aa" doesn't seem to work with kvm and/or debian, no idea why.

It is frustrating that failover works fine but not migration. So far I
am a bit afraid of adding a 3rd node as I wont be able to migrate an
instance to a new node (for example, instance lives in node A and B
and i can failover between them but not migrate to C for example),
unless I export/import but it is not efficient as some instances are
large.


Thank you

[ricardo]

It's me again.

If someone can point me on how to debug this?
I don't seem to get a useful error and cannot find much information
about this on the internet but a post suggesting upgrading kvm but
then the behavior is freezing.


Thank you.

[Iustin Pop]

On Tue, May 11, 2010 at 12:03:16AM -0700, ricardo wrote:
> It's me again.
>
> If someone can point me on how to debug this?
> I don't seem to get a useful error and cannot find much information
> about this on the internet but a post suggesting upgrading kvm but
> then the behavior is freezing.

Sorry, don't know much about KVM versions/behaviour, but see a note
below.

Hmm, there must be some confusion here. If you have an instance on A/B,
then add a third node C, then you can switch the instance from A/B to
A/c via gnt-instance replace-disks, and then use failover or migration
(if it works) to toggle A/C to C/A and back.

In other words, not migration is what allows you to change the secondary
node, but replace-disks...

regards,
iustin

[Roberto Espinoza]

Hello Iustin,

Thank you for your reply. 

Seems I was confused on how to move them when in DRBD, this was my biggest concern as I want to have this cluster live soon and failures are to be expected and didn't want to have long downtimes while exporting / importing.

It would be nice to change in the documentation where the "migrate" is used to something like "Also failover can be used". (http://ganeti-doc.googlecode.com/svn/ganeti-2.1/html/admin.html#instance-relocation)

"migrate" seems like a good feature to have but not critical at this point.

Now I am checking why I cannot set the vnc password using a file, ssh'ing to the monitor and doing "change vnc password" seems to work, but hopefully I can fix that, but that will be in another mail.

Thanks again.

2010/5/11 Iustin Pop <ius...@google.com>

[Guido Trotter]

On Tue, May 11, 2010 at 7:16 AM, ricardo <ricardo....@gmail.com> wrote:
> Hello everyone,
>
> I wonder if anyone have successfully run ganeti with kvm and debian
> lenny 5.0.4?
>

Yes, it should work. Which version of kvm are you using?

> For me it seems to work but the live migration. I get the following
> error:
>
> Could not pre-migrate instance vm.host: Failed to accept instance:
> Failed to start instance vm.host: exited with exit code 1 (qemu:
> loading initrd (0x65bad8 bytes) at 0x0000000017994000
> Migration failed rc=201
>
> I even added the command output to see if the "incoming" parameter was
> being added correctly for my configuration
>
> (cmd:[u'/usr/bin/kvm', u'-name', u'vm.host', u'-m', 384, u'-smp', 1,
> u'-pidfile', u'/var/run/ganeti/kvm-hypervisor/pid/vm.host', u'-
> daemonize', u'-boot', u'c', u'-drive', u'file=/var/run/ganeti/instance-
> disks/vm.host:0,format=raw,if=virtio,boot=on', u'-kernel', u'/boot/
> vmlinuz-2.6-kvmU', u'-initrd', u'/boot/initrd.img-2.6-kvmU', u'-
> append', u'root=/dev/vda1 ro console=ttyS0,38400', u'-vnc',
> u'127.0.0.1:5100', u'-usbdevice', u'tablet', u'-monitor', u'unix:/var/
> run/ganeti/kvm-hypervisor/ctrl/vm.host.monitor,server,nowait', u'-
> serial', u'unix:/var/run/ganeti/kvm-hypervisor/ctrl/
> vm.host.serial,server,nowait', u'-localtime', '-net',
> u'nic,vlan=0,macaddr=00:16:3e:d9:76:53,model=virtio', '-net',
> 'tap,vlan=0,script=/tmp/tmpoBz2kO', '-incoming', u'tcp:
> 10.1.1.12:8102'])
>

For newer kvm (qemu-kvm-*) maybe you want to try to change the
hypervisor parameter to pass the -enable-kvm flag.
This wasn't needed for kvm-7X and 8X.
gnt-cluster modify -H kvm:kvm_flag=enabled

> For me it seems fine.

It complains about initrd: so is your initrd file at the right place,
on both nodes.
I always tried with monolithic kernels built with no modules support,
for now, so maybe there's a bug with that.
Let's investigate and find out! :)

Thanks,

Guido

[ricardo]

Hello Guido,

>
> Yes, it should work. Which version of kvm are you using?
>

I am using the default kvm package from debian lenny 5.0.4.
QEMU PC emulator version 0.9.1 (kvm-72)
Package Version: 72+dfsg-5~lenny5

Kernel: 2.6.26-2-amd64

These are the lastest packages from debian but I am not using the
backports repository.
I think ganeti support KVM 72 and up so it should be fine.

>
> It complains about initrd: so is your initrd file at the right place,
> on both nodes.
> I always tried with monolithic kernels built with no modules support,
> for now, so maybe there's a bug with that.
> Let's investigate and find out! :)
>

Both initrd and kernel files have the same name

node1:
cluster1-node1:/var/log/ganeti/os# ls -l /boot/
total 9536
-rw-r--r-- 1 root root 85663 2010-03-10 09:34 config-2.6.26-2-amd64
drwxr-xr-x 2 root root 4096 2010-05-10 17:12 grub
-rw-r--r-- 1 root root 6660610 2010-05-10 17:09 initrd.img-2.6.26-2-
amd64
lrwxrwxrwx 1 root root 25 2010-05-10 17:37 initrd.img-2.6-kvmU ->
initrd.img-2.6.26-2-amd64
-rw-r--r-- 1 root root 1226063 2010-03-10 09:34 System.map-2.6.26-2-
amd64
-rw-r--r-- 1 root root 1755760 2010-03-10 09:34 vmlinuz-2.6.26-2-amd64
lrwxrwxrwx 1 root root 22 2010-05-10 17:37 vmlinuz-2.6-kvmU ->
vmlinuz-2.6.26-2-amd64

node2:
cluster1-node2:~$ ls -l /boot/
total 9540
-rw-r--r-- 1 root root 85663 2010-03-10 09:34 config-2.6.26-2-amd64
drwxr-xr-x 2 root root 4096 2010-05-10 12:21 grub
-rw-r--r-- 1 root root 6666968 2010-05-10 12:16 initrd.img-2.6.26-2-
amd64
lrwxrwxrwx 1 root root 25 2010-05-11 10:31 initrd.img-2.6-kvmU ->
initrd.img-2.6.26-2-amd64
-rw-r--r-- 1 root root 1226063 2010-03-10 09:34 System.map-2.6.26-2-
amd64
-rw-r--r-- 1 root root 1755760 2010-03-10 09:34 vmlinuz-2.6.26-2-amd64
lrwxrwxrwx 1 root root 22 2010-05-11 10:31 vmlinuz-2.6-kvmU ->
vmlinuz-2.6.26-2-amd64


I can test out any ideas you have to see if it is a bug or something
is not set correctly on my side.



Ricardo

[Guido Trotter]

On Wed, May 12, 2010 at 3:36 AM, ricardo <ricardo....@gmail.com> wrote:

Hi,

>
> I am using the default kvm package from debian lenny 5.0.4.
> QEMU PC emulator version 0.9.1 (kvm-72)
> Package Version: 72+dfsg-5~lenny5
>

This is a really ancient version of KVM. Even when I implemented
migration originally I was working with some newer backports (79, than
84, IIRC).
Now this is reported to work, with a new kernel (2.6.32 or 33) and
with qemu-kvm 0.12.X (or 0.11.X). I really recommend that you try with
those backports from backports.org and see if you get any difference.

> Kernel: 2.6.26-2-amd64
>
> These are the lastest packages from debian but I am not using the
> backports repository.
> I think ganeti support KVM 72 and up so it should be fine.
>

It does, basically, but we cannot guarantee that all funcionality
works under it.

Thanks,
Guido

[Ricardo Espinoza]

Thanks Guido,

I will try to test out tomorrow. Just quick question, I need to enable the kvm flag when installing qemu-kvm right?

Also changing the kernel means I will have to recompile drbd, right?

Actually I tried installing the newer KVM a few days ago but it froze, but didn't update the kernel so probably that is what happened.

I'll post my findings later.

Ricardo

2010/5/12 Guido Trotter <ultr...@google.com>

[Guido Trotter]

On Wed, May 12, 2010 at 10:58 AM, Ricardo Espinoza
<ricardo....@gmail.com> wrote:
> Thanks Guido,
> I will try to test out tomorrow. Just quick question, I need to enable the
> kvm flag when installing qemu-kvm right?

Yes, you can do that by changing the relevant hypervisor parameter.

> Also changing the kernel means I will have to recompile drbd, right?

Yes. The kernel module needs to be the one for the new kernel. The
good news is that very new kernels (I don't remember if it's 2.6.33
only or even 2.6.32) have drbd in mainline, so it just needs to be
configured in.

> Actually I tried installing the newer KVM a few days ago but it froze, but
> didn't update the kernel so probably that is what happened.
> I'll post my findings later.
>

Yeah, you need a new kernel module for the new kvm. If you don't want
to upgrade your kernel, the kvm develpers release the module as a
separate entity that can be compiled against old kernels.

Thanks,

Guido

[ricardo]

Hello

I installed linux-image-2.6.32-bpo.3-amd64 but complained about the
drbd, "module not found".
There is no /proc/drbd :/
I checked the mailing lists for Debian and they actually are going
remove the drbd8-source as should be built in.

bpo.4 refuses to install (missing dependencies).

Have anyone tried using lenny-backports kernel?

On May 12, 7:16 pm, Guido Trotter <ultrot...@gmail.com> wrote:
> On Wed, May 12, 2010 at 10:58 AM, Ricardo Espinoza
>

[Guido Trotter]

On Wed, May 12, 2010 at 2:26 PM, ricardo <ricardo....@gmail.com> wrote:
> Hello
>
> I installed linux-image-2.6.32-bpo.3-amd64 but complained about the
> drbd, "module not found".
> There is no /proc/drbd :/
> I checked the mailing lists for Debian and they actually are going
> remove the drbd8-source as should be built in.
>
> bpo.4 refuses to install (missing dependencies).
>
> Have anyone tried using lenny-backports kernel?
>

Not really, I mostly rolled my own from vanilla. But perhaps try
emailing the backports maintainer about the missing deps, and/or
compile the module against the one that installs.

Thanks,

Guido

[Coalescence]

On May 12, 2:47 pm, Guido Trotter <ultrot...@google.com> wrote:

> On Wed, May 12, 2010 at 2:26 PM, ricardo <ricardo.espin...@gmail.com> wrote:
> > Hello
>
> > I installed linux-image-2.6.32-bpo.3-amd64 but complained about the
> > drbd, "module not found".
> > There is no /proc/drbd :/
> > I checked the mailing lists for Debian and they actually are going
> > remove the drbd8-source as should be built in.
>
> > bpo.4 refuses to install (missing dependencies).
>
> > Have anyone tried using lenny-backports kernel?
>
> Not really, I mostly rolled my own from vanilla. But perhaps try
> emailing the backports maintainer about the missing deps, and/or
> compile the module against the one that installs.

I've had issues with the later backport kernels.. they always seem to
be in a state of flux and not very useful for production boxes, also
the drbd stuff isn't there.

I use grml

#echo "deb http://deb.grml.org/ grml-stable main" >> /etc/apt/
sources.list

#aptitude install linux-image-2.6.33-grml64

(may want to add the key too)

#update-grub

reboot




>
> Thanks,
>
> Guido

[Roberto Espinoza]

Thanks for the quick reply Guido.

It seems there is no fix coming soon to backports. Should be in testing.

I am using 2.6.30 so I can have the kbuild for that kernel and build drbd

And FINALLY :)

Wed May 12 22:55:44 2010 * starting the instance on the target node

cluster1-node1:/home/roberto# gnt-instance migrate somehost

Instance somehost will be migrated. Note that migration might

impact the instance if anything goes wrong (e.g. due to bugs in the

hypervisor). Continue?

y/[n]/?: y

Wed May 12 22:55:52 2010 Migrating instance somehost.local

Wed May 12 22:55:52 2010 * checking disk consistency between source and target

Wed May 12 22:55:52 2010 * switching node cluster1-node2.local to secondary mode

Wed May 12 22:55:52 2010 * changing into standalone mode

Wed May 12 22:55:52 2010 * changing disks into dual-master mode

Wed May 12 22:55:53 2010 * wait until resync is done

Wed May 12 22:55:53 2010 * preparing cluster1-node2.local to accept the instance

Wed May 12 22:55:53 2010 * migrating instance to cluster1-node2.local

Wed May 12 22:56:17 2010 * switching node cluster1-node1.local to secondary mode

Wed May 12 22:56:17 2010 * wait until resync is done

Wed May 12 22:56:17 2010 * changing into standalone mode

Wed May 12 22:56:17 2010 * changing disks into single-master mode

Wed May 12 22:56:18 2010 * wait until resync is done

Wed May 12 22:56:20 2010 * done

cluster1-node1:/home/roberto#

Thank you very much for your help.

Just for the record these is what I tried if anyone is interested.

linux-image-2.6.30-bpo.2-amd64 is installed

linux-kbuild-2.6.30 to build the drbd module (2.6.32 doesn't exist in backports)

I also installed drbd8-source and utils from backports too. You then need to install drbd8 as usual.

KVM is now 0.12.3+dfsg-4~bpo50+2, in backports is called qemu-kvm and it will automatically uninstall kvm (stable).

The kvm-flag needs to be enabled in the cluster/instances.

So far the instance seems to run fine, failover and migrate (finally) works. Need to test a bit more to see how it behaves.

Ricardo

[Roberto Espinoza]

Hello Coalescence,

I actually plan to use 2.6 for the guests to stay in the stable as there are no special needs.

I have no experience using grml.

Why do you think backports kernel are not ready for production? Did you have a bad experience with them or they just lack features?

In testing it should be the one with DRBD but it is not completely in backports. Some people have installed the kbuild and linux-base from testing and the kernel from backports.

So far like I said in my other posts, migration is the only thing that doesn't work with lenny stable, but failover is fine. Of course I am a bit worried on using backports but then it can't be as bad as compiling mine :P

What do you think?

2010/5/12 Coalescence <joel.m...@gmail.com>