grub2 bootloader installation when running kiwi on docker

[Rodrigo Oshiro]

Hi,

We recently started replacing grub to grub2 bootloader in our appliances based on SLES11SP3. Everything was working fine when I was running KIWI after installing the required packages, however when I run KIWI inside a container in docker I get the following error:

Oct-30 15:45:59 <1> : EXEC [/usr/sbin/grub2-bios-setup -f -d /tmp/kiwiboot.d4VlGW/boot/grub2/i386-pc -m /tmp/kiwiboot.d4VlGW/boot/grub2/device.map /image/product/vmx-product/appliance.x86_64-1.3.8.2.101.raw 2>&1]

Oct-30 15:46:00 <3> : Couldn't install grub2 on /image/product/vmx-product/appliance.x86_64-1.3.8.2.101.raw: /usr/sbin/grub2-bios-setup: error: failed to get canonical path of /dev/mapper/docker-254:0-2701730-e72728ed6c87b118424b29c9deea27eaf4c73c187619777366a472c052aa3299.

I copied i386-pc modules and the device.map locally to try and run that "grub2-bios-setup" command myself, but it looks like it stops when it cannot find the mapped container in the /dev/ folder. That folder exists only on the host. Inside the container the mapping is completely different. My environment is a SLES12 runing a SLES11SP3 docker container with the packages for the build. I spend some time investigating it and I had to modify the following file to complete the build:

KIWIBoot.pm:

            # architectures: ix86, x86_64 and arm

            $status = KIWIQX::qxx (

                "pgrep init 2>&1"

            );

            $result = $? >> 8;

            if ($result != 0) {

                $loaderTarget = $this->{loop};

                $grubtool = $locator -> getExecPath ('grub2-install');

                $grubtoolopts = "--grub-mkdevicemap=$dmfile ";

                $grubtoolopts.= "-d $stages ";

                $grubtoolopts.= "--root-directory=/mnt --force --no-nvram ";

                $targetMessage= "On PReP partition";

            } elsif ($chainload) {

                $loaderTarget = readlink ($bootdev);

                $loaderTarget =~ s/\.\./\/dev/;

                $grubtoolopts = "-f -d $stages -m $dmfile";

                $targetMessage= "On partition target";

            } else {

                $loaderTarget = $diskname;

                $grubtoolopts = "-f -d $stages -m $dmfile";

                $targetMessage= "On disk target";

            }

Where the red lines are lines I added myself. What it does is check if I am running KIWI from a container and if I am, then I make a call for grub2-install instead of grub2-bios-setup. Maybe I am wrong, but I think these red lines can replace those in blue. Was there a reason to use grub2-bios-setup and not 'grub2-install' to install the bootloader? Well, at least I know its possible to install grub2 bootloader in a container. Also, this issue happens outside the container when I rename LVM disks to not conflict with the LVM of the appliance I am building. This is needed for building an appliance that can build itself, so maybe it could be a solution for both. For now we have to restart the appliance, so the LVM rename completes and maps the final locations (if I "lvrename" and then "df", the old name would still be there, for example. mtab also lists old names after renames).

Also, for "core.elf", it appears that -v argument is being used for grub2-install, and that's making the tool output the version and may hide errors. Not sure, as I do not need this, but I created the red lines based on "core.elf" code. Below is the additional argument that outputs versions in grub2-install instead of installing the bootloader:

            $grubtoolopts = "--grub-mkdevicemap=$dmfile -v ";

Maybe this is not the best solution, but at least I would like to see a fix for installing grub2 from inside a container. Is it possible, or is there anything else I can do to complete the build without having to change the code? Maybe there is something I can modify in config.xml that I am not aware, or some commands I can run on my container, but I could not find anything specific.

Please, let me know the best approach so I can continue the support of grub2 bootloader in SLES11SP3 by using KIWI.

Thanks,

Rodrigo.

[Marcus Schäfer]

Hi,

> We recently started replacing grub to grub2 bootloader in our
> appliances based on SLES11SP3. Everything was working fine when I was
> running KIWI after installing the required packages, however when I run
> KIWI inside a container in docker I get the following error:
> Oct-30 15:45:59 <1> : EXEC [/usr/sbin/grub2-bios-setup -f -d
> /tmp/kiwiboot.d4VlGW/boot/grub2/i386-pc -m
> /tmp/kiwiboot.d4VlGW/boot/grub2/device.map
> /image/product/vmx-product/appliance.x86_64-1.3.8.2.101.raw 2>&1]
> Oct-30 15:46:00 <3> : Couldn't install grub2 on
> /image/product/vmx-product/appliance.x86_64-1.3.8.2.101.raw:
> /usr/sbin/grub2-bios-setup: error: failed to get canonical path of
> /dev/mapper/docker-254:0-2701730-e72728ed6c87b118424b29c9deea27eaf4c73c
> 187619777366a472c052aa3299.

This is imho a bug in the grub2 (grub-probe) you use inside the docker
container. You should be able to reproduce this with grub-probe directly
also see man grub-probe

grub2 walks through all devices it see and does some device naming
specific matching. Inside a docker container there are mapped devices
like:

/dev/mapper/docker-254:0-2701730-e72728ed6c87b118424b29c9deea27eaf4c73c

which do not strictly follow the naming conventions of device nodes.
Anyway grub should not fall over them

actually this has nothing to do with kiwi :)

you might consider to report this to the grub2 people

Hope that helps

Regards,
Marcus
--
Public Key available
http://pgp.mit.edu/pks/lookup?search=0x94302D78&op=index
-------------------------------------------------------
Marcus Schäfer (Res. & Dev.) SUSE LINUX Products GmbH
Tel: 0911-740 53 0 Maxfeldstrasse 5
FAX: 0911-740 53 479 D-90409 Nürnberg
GF: Jeff Hawn,Jennifer Guild, Felix Imendörffer
HRB: 21284 (AG Nürnberg) Germany
http://www.suse.de
-------------------------------------------------------

[Rodrigo Oshiro]

Hi Marcus,

I am glad I posted here and found a way out to address a similar issue I had in the past... When building images on containers on SLES11 I could bypass the bootloader probing logic by starting udev, but now that I am on SLES12 containers, I cannot use the same approach as the service is too linked with systemd, but I noticed that lvcreate supports --noudevsync on kiwi, so I don't have problems regarding LVM. However, grub2 probing on containers is unreliable and we do not even should have a bootloader installed on them in the first place...

I changed the default values on my lvm.conf to disable udev_sync:

udev_sync = 0

udev_rules = 0

So I used the same lines in red from my first post to resolve the issue and have the bootloader installed on that LVM space. Can we incorporate that change to avoid using grub2 probing that does not work on docker containers?

Here is the log without my changes:

May-14 11:56:08 <1> : Installing grub2:
May-14 11:56:08 <1> : --> On disk target: /root/appliance/vmx-appliance/image.x86_64-12.0.0.1.100.raw
May-14 11:56:08 <1> : EXEC [/usr/sbin/grub2-bios-setup -f -d /tmp/kiwiboot.CM6puH/boot/grub2/i386-pc -m /tmp/kiwiboot.CM6puH/boot/grub2/device.map /root/appliance/vmx-appliance/image.x86_64-12.0.0.1.100.raw 2>&1]
May-14 11:56:08 <3> : Couldn't install grub2 on /root/appliance/vmx-appliance/image.x86_64-12.0.0.1.100.raw: /usr/sbin/grub2-bios-setup: error: failed to get canonical path of `/dev/mapper/docker-254:0-3252534-8c18de990be90700df0b5d61509ff6942d60630fb78e994ebd17edfa48ea3d83'.
May-14 11:56:08 <1> : EXEC [sync]
May-14 11:56:09 <1> : EXEC [sync]
May-14 11:56:09 <1> : EXEC [umount /tmp/kiwiboot.7VILES 2>&1]
May-14 11:56:09 <1> : EXEC [vgchange -an systemVG 2>&1]
May-14 11:56:09 <1> : EXEC [kpartx -sd /dev/loop3 2>&1]
May-14 11:56:10 <1> : EXEC [losetup -d /dev/loop3 2>&1]
May-14 11:56:10 <1> : EXEC [sync]
May-14 11:56:10 <1> : EXEC [sync]
May-14 11:56:10 <1> : EXEC [rm -rf /tmp/kiwiboot.CM6puH 2>&1]
May-14 11:56:10 <1> : EXEC [rm -rf /tmp/kiwiloop.yMMRRV 2>&1]
May-14 11:56:11 <1> : EXEC [rm -rf /tmp/kiwiboot.7VILES 2>&1]
May-14 11:56:11 <1> : EXEC [sync]
May-14 11:56:11 <1> : EXEC [sync]
May-14 11:56:11 <1> : Closing session with ecode: 1
May-14 11:56:11 <3> : KIWI exited with error(s)
May-14 11:56:11 <1> : EXEC [mv /rodrigo/logs/appliance.25.screenrc.log /rodrigo/logs/appliance.log 2>&1]
May-14 11:56:11 <1> : Complete logfile at: /rodrigo/logs/appliance.log
May-14 11:56:11 <1> : EXEC [rm -rf /root/appliance/vmx-appliance/boot-VMX.1JOKFt 2>&1]

And here is the log with them:

May-14 12:15:51 <1> : Installing grub2:
May-14 12:15:51 <1> : --> On PReP partition: /dev/loop3
May-14 12:15:51 <1> : EXEC [/usr/sbin/grub2-install --grub-mkdevicemap=/tmp/kiwiboot.HikTgU/boot/grub2/device.map -d /tmp/kiwiboot.HikTgU/boot/grub2/i386-pc --root-directory=/mnt --force --no-nvram  /dev/loop3 2>&1]
May-14 12:15:59 <1> : EXEC [sync]
May-14 12:15:59 <1> : EXEC [sync]
May-14 12:15:59 <1> : EXEC [umount /mnt 2>&1]
May-14 12:16:00 <1> : EXEC [vgchange -an systemVG 2>&1]
May-14 12:16:00 <1> : EXEC [kpartx -sd /dev/loop3 2>&1]
May-14 12:16:00 <1> : EXEC [losetup -d /dev/loop3 2>&1]
May-14 12:16:00 <1> : EXEC [sync]
May-14 12:16:00 <1> : EXEC [sync]
May-14 12:16:00 <1> : Saving disk label in MBR: 0xb1e7ca73...
May-14 12:16:00 <1> : EXEC [sync]
May-14 12:16:00 <1> : EXEC [sync]
May-14 12:16:00 <1> : EXEC [rm -rf /tmp/kiwiboot.HikTgU]
May-14 12:16:00 <1> : EXEC [sync]
May-14 12:16:00 <1> : EXEC [sync]
May-14 12:16:01 <1> : EXEC [sync]
May-14 12:16:01 <1> : EXEC [sync]
May-14 12:16:01 <1> : EXEC [/usr/bin/uname -m]
May-14 12:16:01 <1> : --> Starting image format conversion...
May-14 12:16:01 <1> : EXEC [parted /root/appliance/vmx-appliance/image.x86_64-12.0.0.1.100.raw print 2>&1]
May-14 12:16:01 <1> : Starting raw => vmdk conversion
May-14 12:16:01 <1> : Creating vmdk image...
May-14 12:16:01 <1> : EXEC [/usr/bin/qemu-img convert -f raw /root/appliance/vmx-appliance/image.x86_64-12.0.0.1.100.raw -O vmdk -o adapter_type=lsilogic /root/appliance/vmx-appliance/image.x86_64-12.0.0.1.100.vmdk 2>&1]
May-14 12:16:55 <1> : Creating vmx image machine configuration
May-14 12:16:55 <1> : Creating image VMware configuration file...
May-14 12:16:56 <1> : EXEC [sync]
May-14 12:16:57 <1> : EXEC [sync]
May-14 12:16:57 <1> : EXEC [rm -rf /tmp/kiwiboot.HikTgU 2>&1]
May-14 12:16:57 <1> : EXEC [rm -rf /tmp/kiwiloop.qRSz87 2>&1]
May-14 12:16:57 <1> : EXEC [sync]
May-14 12:16:57 <1> : EXEC [sync]
May-14 12:16:57 <1> : EXEC [/usr/bin/uname -m]
May-14 12:16:57 <1> : EXEC [mv -f /root/appliance/vmx-appliance/* /root/appliance 2>&1]
May-14 12:16:57 <1> : KIWI exited successfully
May-14 12:16:57 <1> : EXEC [mv /rodrigo/logs/appliance.20831.screenrc.log /rodrigo/logs/appliance.log 2>&1]
May-14 12:16:57 <1> : Complete logfile at: /rodrigo/logs/appliance.log
May-14 12:16:57 <1> : EXEC [rm -rf /root/appliance/vmx-appliance/boot-VMX.mEQq5S 2>&1]

However, it seems that this approach was working fine with the KIWI version installed from SLES 12 media (kiwi-5.06.165-1.7), but did not work with the version I downloaded from openSUSE (kiwi-7.02.42-953.1). I had to install a dependency (gptfdisk), I am not sure if its related. The only environment I was able to succeed was this combination (SLES 12 container + kiwi 5.06 + udev_sync/udev_rules disabled + patch to replace "grub2-bios-setup" with "grub2-install"). If you have a better solution, please let me know.

With the upgrade to kiwi 7, I have this error:

May-14 14:22:02 <1> : Installing grub2:
May-14 14:22:02 <1> : --> On PReP partition: /dev/loop3
May-14 14:22:11 <3> : Couldn't install grub2 on /dev/loop3: Installing for i386-pc platform.
/usr/sbin/grub2-install: error: failed to get canonical path of `/dev/mapper/docker-254:0-3252534-b7b5c7892b1d533a68c006b1ddcd841309bf1b929ce83918f61cba4575be58b2'.
                                                                                                                                                                                            failed
May-14 14:22:14 <1> : Closing session with ecode: 1
May-14 14:22:14 <3> : KIWI exited with error(s)

Thanks,

Rodrigo.

[Marcus Schäfer]

Hi,

> I am glad I posted here and found a way out to address a similar issue
> I had in the past... When building images on containers on SLES11 I
> could bypass the bootloader probing logic by starting udev, but now
> that I am on SLES12 containers, I cannot use the same approach as the
> service is too linked with systemd, but I noticed that lvcreate
> supports --noudevsync on kiwi, so I don't have problems regarding LVM.

yes I recently added that to the kiwi code base here

commit 1eea32b5e345f0465fcc2b12772fa51a1afb17d9
Author: Marcus Schäfer <m...@suse.de>
Date: Wed Apr 29 11:21:02 2015 +0200

Run lvcreate with --noudevsync

This allows to build LVM images in container systems
which doesn't have udev running

> However, grub2 probing on containers is unreliable and we do not even
> should have a bootloader installed on them in the first place...
> I changed the default values on my lvm.conf to disable udev_sync:

For me this is a bug in grub2 not being able to correctly process
storage paths from the device mapper. In docker (lxc) the default
storage backend is based on device mapper.

> /usr/sbin/grub2-bios-setup: error: failed to get canonical path of

I have workarounded this issue by using a btrfs backend in docker
(this also speed things up a little bit if you put it on ram)

qemu-img create /var/lib/docker.btrfs 10g
mkfs.btrfs /var/lib/docker.btrfs
mount /var/lib/docker.btrfs /var/lib/docker

Now start docker with:

/usr/bin/docker -d -s btrfs

and it will work. I use the same approach for my dice container
build system here:

https://github.com/openSUSE/kiwi/wiki/Building-images-in-a-Docker-container

maybe interesting for you as it seems you are building images in a docker
container

Regards,
Marcus
--
Public Key available via: https://keybase.io
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer (Res. & Dev.) SUSE Linux GmbH

Tel: 0911-740 53 0 Maxfeldstrasse 5
FAX: 0911-740 53 479 D-90409 Nürnberg

HRB: 21284 (AG Nürnberg) Germany

GF: Felix Imendörffer, Jane Smithard
GF: Jennifer Guild, Dilip Upmanyu, Graham Norton
http://www.suse.de
-------------------------------------------------------

[Rodrigo Oshiro]

Well, the root-directory is not hardcoded to /mnt on 7 anymore, so I could update that. This dice container is an interesting reference, as kpartx and issues with lvm are a headache. Would this approach using btrfs storage avoid conflicts with device mapper and activation of volume groups on the system as well?

[Marcus Schäfer]

Hi,

> This dice container is an interesting reference, as
> kpartx and issues with lvm are a headache.

right and depending on build host capabilities like rpm, yum
grub, grub2, lvm ...and more could cause a lot of trouble. Also
parallel building of images is possible in a contained and host
independent system. dice is a relatively new project and I have
started it because of the many problems I had with unsatisfied
build host dependencies.

I also wanted to provide a clean way to target an image build
to "somewhere", where somewhere could be a cloud instance of
a public cloud provider, the local system or any other system

imho it should be an interesting approach for all those people
who need a private buildsystem and for some reason can't use
the public open build service, e.g because the image installs
software from a non public repository

> Would this approach using
> btrfs storage avoid conflicts with device mapper and activation of
> volume groups on the system as well?

It will workaround any issues with the device mapper.

However the problem that kiwi should be able to build images even without
udev around is solved in kiwi itself by calling tools like kpartx in sync
mode or lvm tools with "noudevsync".

The tests I have made with container builds did show some of these
problems and I was able to fix them all. I did not have tested
building luks encrypted images which uses the device mapper but
as of today I'm not aware of any issue