ganeti-instance-image restore segfault

[Daniel DeFreez]

Hi,

We recently built a two-node Ganeti cluster on top of CentOS 6.1 (kmod-drbd83 from elrepo.org) and things are working pretty well. I am deploying CentOS guests with ganeti-instance-image. New instances (--no-install) work fine when the OS is installed via PXE. I can use ganeti-instance-image to successfully take a dump of the instance after it has been installed, but when I attempt to restore that dump to a new instance I get the following error (excerpted from the OS script log):

+ root_dump=/var/cache/ganeti-instance-image/centdesk-x86_64-root.dump

+ cd /tmp/tmp.RJYaPYLXC0

+ restore -r -y -f /var/cache/ganeti-instance-image/centdesk-x86_64-root.dump

Dump tape is compressed.

restore: ./lost+found: File exists

restore: ./boot: File exists

+ '[' -n /dev/mapper/drbd2-1 ']'

+ boot_dump=/var/cache/ganeti-instance-image/centdesk-x86_64-boot.dump

+ '[' '!' -f /var/cache/ganeti-instance-image/centdesk-x86_64-boot.dump ']'

+ cd /tmp/tmp.RJYaPYLXC0/boot

+ restore -r -y -f /var/cache/ganeti-instance-image/centdesk-x86_64-boot.dump

Dump tape is compressed.

restore: ./lost+found: Input/output error

restore: ./grub: Input/output error

restore: ./efi: Input/output error

restore: ./efi/EFI: No such file or directory

restore: ./efi/EFI/redhat: No such file or directory

restore: fopen: Input/output error

cannot create save file ./restoresymtable for symbol table

/usr/share/ganeti/os/image/create: line 120: 19191 Segmentation fault      (core dumped) restore -r -y -f ${boot_dump} > /dev/null

Earlier in the log I see the following sfdisk errors, but the partitions still get created:

BLKRRPART: Invalid argument

Disk /dev/drbd2: cannot get geometry

BLKRRPART: Invalid argument

Here is my config for ganeti-instance-image:

SWAP=yes

FILESYSTEM=ext4

IMAGE_NAME=centdesk

IMAGE_TYPE=dump

IMAGE_DIR=/var/cache/ganeti-instance-image

ARCH=x86_64

IMAGE_DEBUG=1

Any ideas? I can restore the dump to the local filesystem just fine.

Thanks everyone for all of the work that has gone into these tools,

Daniel DeFreez

Network Administrator

Southern Oregon University

[Daniel DeFreez]

I managed to resolve this particular issue, though the workaround is
less than ideal.

The problem had to do with the way /boot was being mounted and
restored by the OS create script. The create script calls mount_disk0
is common.sh, which mounts the root partition (in a temp directory),
creates a boot mount point nested under that and mount the boot
partition. After this, the root partition is restored, followed by the
boot partition. The problem was that my root dumps contained an empty
boot directory, which when restored mashed over the previously created
boot mount point. Subsequently fopen() failed and restore segfaulted.

Has anyone else experienced this? Something peculiar to my setup /
guest?

I solved this by going into my guest and doing a chattr +d /boot, then
adding -h 0 to the make-dump script (ensure no_dump attribute is
honored). That got me up and running, but I don't think it is the best
way to deal with the issue. It seems that either the make-dump script
should strip /boot out of the dump or common.sh should handle the root
and boot mounts separately. I'm not sure if common.sh is supposed to
remain pristine in the OS definitions.

I'm happy to submit a patch if it seems worthwhile.

[Iustin Pop]

This is another example why filesystem-based import/export is bad. We
have discussed many times about switching this to simple block-based
import/export (handled internally in Ganeti, no OS script support
needed), which would remove the need for the node to be able to handle
the specific filesystems of each OS.

Thoughts?

iustin

[Richard Rafalski]

Am 21.12.2011 09:33, schrieb Iustin Pop:
...

> This is another example why filesystem-based import/export is bad. We
> have discussed many times about switching this to simple block-based
> import/export (handled internally in Ganeti, no OS script support
> needed), which would remove the need for the node to be able to handle
> the specific filesystems of each OS.
>
> Thoughts?

Due to this problems I am using dd for export/import. But this is very
time consuming as mentioned earlier in this list.

The use of a tool like partimage should speed up a block-based
export/import because it is only copying blocks of a partition that are
used by the filesystem. Of course only if the filesystem is supported by
partimage.

Maybe the integration of partimage into ganeti for import/export makes
sense? These are only some thoughts.

Ritchi

[Lance Albertson]

Mind making a feature request for it on ganeti-instance-image's issue
tracker? Sorry for the issues with dump/restore.

--
Lance Albertson
Systems Administrator / Architect Open Source Lab
Information Services Oregon State University