Qubes does not boot on Fedora/Qubes dual-boot machine

[Martin Páleník]

My goal is to have a dual-boot of Qubes OS 3.1 and Fedora 23 Workstation (respin). This guy got it working on Ubuntu, but he installed Ubuntu first, while I installed Qubes first. The last resort would be to follow him and reinstall from scratch, but I want to learn something.

What have I tried:

 - installed Qubes OS 3.1 (OK)

 - updated Qubes OS 3.1 dom0 and Fedora23 template (OK)

 - installed the latest Fedora 23 Workstation respin (OK, but reported some GRUB2 bootloader failure; nevertheless Fedora booted fine)

 - updated the latest Fedora 23 Workstation (OK, there were no updates)

 - tried to boot Qubes (fail, no menuentry in GRUB2 boot menu yet)

 - created a menuentry for Qubes (OK, see bellow)

 - tried to boot Qubes (fail, see screenshot)

 - googled this workaround and installed the package

 - I get thgrough the error message now, but after a few seconds the system reboots itself and does that infinitely

How I added the menuitem:

I booted into Fedora and went to the `/etc/grub.d/` folder,  and edited the `40_custom` file. In there i pasted everything between the xen section of the /boot/grub2/grub.cfg file on the qubes boot partition, into the end of that `40_custom` file on Fedora. Then I executed the following command under root

`grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg`

and the qubes menuitems got generated into the `grub.cfg` and became visible in the boot menu. I can provide the contents of grub.cfg later, in the meantime I'd like to know what might got wrong and how to fix it.

Notable information:

 - Qubes boots from GPT on Legacy BIOS (with 2MB biosboot partition)

 - Fedora boots from GPT on UEFI (with EFI partition)

 - each OS has separate `/boot`partition, as well as other partitions

 - there is no extra `/boot`partition

Here is the promised grub.cfg:

    #
    # DO NOT EDIT THIS FILE
    #
    # It is automatically generated by grub2-mkconfig using templates
    # from /etc/grub.d and settings from /etc/default/grub
    #
   
    ### BEGIN /etc/grub.d/00_header ###
    set pager=1
   
    if [ -s $prefix/grubenv ]; then
      load_env
    fi
    if [ "${next_entry}" ] ; then
       set default="${next_entry}"
       set next_entry=
       save_env next_entry
       set boot_once=true
    else
       set default="${saved_entry}"
    fi
   
    if [ x"${feature_menuentry_id}" = xy ]; then
      menuentry_id_option="--id"
    else
      menuentry_id_option=""
    fi
   
    export menuentry_id_option
   
    if [ "${prev_saved_entry}" ]; then
      set saved_entry="${prev_saved_entry}"
      save_env saved_entry
      set prev_saved_entry=
      save_env prev_saved_entry
      set boot_once=true
    fi
   
    function savedefault {
      if [ -z "${boot_once}" ]; then
        saved_entry="${chosen}"
        save_env saved_entry
      fi
    }
   
    function load_video {
      if [ x$feature_all_video_module = xy ]; then
        insmod all_video
      else
        insmod efi_gop
        insmod efi_uga
        insmod ieee1275_fb
        insmod vbe
        insmod vga
        insmod video_bochs
        insmod video_cirrus
      fi
    }
   
    terminal_output console
    if [ x$feature_timeout_style = xy ] ; then
      set timeout_style=menu
      set timeout=5
    # Fallback normal timeout code in case the timeout_style feature is
    # unavailable.
    else
      set timeout=5
    fi
    ### END /etc/grub.d/00_header ###
   
    ### BEGIN /etc/grub.d/01_users ###
    if [ -f ${prefix}/user.cfg ]; then
      source ${prefix}/user.cfg
      if [ -n "${GRUB2_PASSWORD}" ]; then
        set superusers="root"
        export superusers
        password_pbkdf2 root ${GRUB2_PASSWORD}
      fi
    fi
    ### END /etc/grub.d/01_users ###
   
    ### BEGIN /etc/grub.d/10_linux ###
    menuentry 'Fedora (4.4.8-300.fc23.x86_64) 23 (Workstation Edition)' --class fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-4.4.8-300.fc23.x86_64-advanced-d43f46bc-7649-44ca-b02d-7599d115a8e8' {
    load_video
    set gfxpayload=keep
    insmod gzio
    insmod part_gpt
    insmod ext2
    set root='hd0,gpt6'
    if [ x$feature_platform_search_hint = xy ]; then
     search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt6 --hint-efi=hd0,gpt6 --hint-baremetal=ahci0,gpt6  440e2ced-56a2-432f-95e0-c5f1c33941a9
    else
     search --no-floppy --fs-uuid --set=root 440e2ced-56a2-432f-95e0-c5f1c33941a9
    fi
    linuxefi /vmlinuz-4.4.8-300.fc23.x86_64 root=UUID=d43f46bc-7649-44ca-b02d-7599d115a8e8 ro rootflags=subvol=root00 rhgb quiet
    initrdefi /initramfs-4.4.8-300.fc23.x86_64.img
    }
    menuentry 'Fedora (0-rescue-1dccfd914c254de687a4ba1c13e159a9) 23 (Workstation Edition)' --class fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-0-rescue-1dccfd914c254de687a4ba1c13e159a9-advanced-d43f46bc-7649-44ca-b02d-7599d115a8e8' {
    load_video
    insmod gzio
    insmod part_gpt
    insmod ext2
    set root='hd0,gpt6'
    if [ x$feature_platform_search_hint = xy ]; then
     search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt6 --hint-efi=hd0,gpt6 --hint-baremetal=ahci0,gpt6  440e2ced-56a2-432f-95e0-c5f1c33941a9
    else
     search --no-floppy --fs-uuid --set=root 440e2ced-56a2-432f-95e0-c5f1c33941a9
    fi
    linuxefi /vmlinuz-0-rescue-1dccfd914c254de687a4ba1c13e159a9 root=UUID=d43f46bc-7649-44ca-b02d-7599d115a8e8 ro rootflags=subvol=root00 rhgb quiet
    initrdefi /initramfs-0-rescue-1dccfd914c254de687a4ba1c13e159a9.img
    }
    if [ "x$default" = 'Fedora (4.4.8-300.fc23.x86_64) 23 (Workstation Edition)' ]; then default='Advanced options for Fedora>Fedora (4.4.8-300.fc23.x86_64) 23 (Workstation Edition)'; fi;
    ### END /etc/grub.d/10_linux ###
   
    ### BEGIN /etc/grub.d/20_linux_xen ###
   
    ### END /etc/grub.d/20_linux_xen ###
   
    ### BEGIN /etc/grub.d/20_ppc_terminfo ###
    ### END /etc/grub.d/20_ppc_terminfo ###
   
    ### BEGIN /etc/grub.d/30_os-prober ###
   
    ### END /etc/grub.d/30_os-prober ###
   
    ### BEGIN /etc/grub.d/40_custom ###
    # This file provides an easy way to add custom menu entries.  Simply type the
    # menu entries you want to add after this comment.  Be careful not to change
    # the 'exec tail' line above.
    menuentry 'Qubes, with Xen hypervisor' --class qubes --class gnu-linux --class gnu --class os --class xen $menuentry_id_option 'xen-gnulinux-simple-6e934da1-b43d-434b-83b2-878d454fb6bd' {
    insmod part_gpt
    insmod ext2
    set root='hd0,gpt2'
    if [ x$feature_platform_search_hint = xy ]; then
     search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2 --hint='hd0,gpt2'  9c6e43e7-856f-4840-a04f-6936735f5835
    else
     search --no-floppy --fs-uuid --set=root 9c6e43e7-856f-4840-a04f-6936735f5835
    fi
    echo 'Loading Xen 4.6.0 ...'
            if [ "$grub_platform" = "pc" -o "$grub_platform" = "" ]; then
                xen_rm_opts=
            else
                xen_rm_opts="no-real-mode edd=off"
            fi
    multiboot /xen-4.6.0.gz placeholder  console=none dom0_mem=min:1024M dom0_mem=max:4096M ${xen_rm_opts}
    echo 'Loading Linux 4.1.13-9.pvops.qubes.x86_64 ...'
    module /vmlinuz-4.1.13-9.pvops.qubes.x86_64 placeholder root=UUID=6e934da1-b43d-434b-83b2-878d454fb6bd ro rootflags=subvol=root i915.preliminary_hw_support=1 vconsole.font=latarcyrheb-sun16 rhgb quiet
    echo 'Loading initial ramdisk ...'
    module /initramfs-4.1.13-9.pvops.qubes.x86_64.img
    }
    submenu 'Advanced options for Qubes (with Xen hypervisor)' $menuentry_id_option 'gnulinux-advanced-6e934da1-b43d-434b-83b2-878d454fb6bd' {
    submenu 'Xen hypervisor, version 4.6.0' $menuentry_id_option 'xen-hypervisor-4.6.0-6e934da1-b43d-434b-83b2-878d454fb6bd' {
    menuentry 'Qubes, with Xen 4.6.0 and Linux 4.1.13-9.pvops.qubes.x86_64' --class qubes --class gnu-linux --class gnu --class os --class xen $menuentry_id_option 'xen-gnulinux-4.1.13-9.pvops.qubes.x86_64-advanced-6e934da1-b43d-434b-83b2-878d454fb6bd' {
    insmod part_gpt
    insmod ext2
    set root='hd0,gpt2'
    if [ x$feature_platform_search_hint = xy ]; then
     search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2 --hint='hd0,gpt2'  9c6e43e7-856f-4840-a04f-6936735f5835
    else
     search --no-floppy --fs-uuid --set=root 9c6e43e7-856f-4840-a04f-6936735f5835
    fi
    echo 'Loading Xen 4.6.0 ...'
           if [ "$grub_platform" = "pc" -o "$grub_platform" = "" ]; then
               xen_rm_opts=
           else
               xen_rm_opts="no-real-mode edd=off"
           fi
    multiboot /xen-4.6.0.gz placeholder  console=none dom0_mem=min:1024M dom0_mem=max:4096M ${xen_rm_opts}
    echo 'Loading Linux 4.1.13-9.pvops.qubes.x86_64 ...'
    module /vmlinuz-4.1.13-9.pvops.qubes.x86_64 placeholder root=UUID=6e934da1-b43d-434b-83b2-878d454fb6bd ro rootflags=subvol=root i915.preliminary_hw_support=1 vconsole.font=latarcyrheb-sun16 rhgb quiet
    echo 'Loading initial ramdisk ...'
    module /initramfs-4.1.13-9.pvops.qubes.x86_64.img
    }
    }
    }
    ### END /etc/grub.d/40_custom ###
   
    ### BEGIN /etc/grub.d/40_custom.backup ###
    # This file provides an easy way to add custom menu entries.  Simply type the
    # menu entries you want to add after this comment.  Be careful not to change
    # the 'exec tail' line above.
    ### END /etc/grub.d/40_custom.backup ###
   
    ### BEGIN /etc/grub.d/41_custom ###
    if [ -f  ${config_directory}/custom.cfg ]; then
      source ${config_directory}/custom.cfg
    elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
      source $prefix/custom.cfg;
    fi
    ### END /etc/grub.d/41_custom ###

a copy is also on Pastebin.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sat, Apr 30, 2016 at 06:49:37PM -0700, Martin Páleník wrote:
> My goal is to have a dual-boot of Qubes OS 3.1 and Fedora 23 Workstation
> (respin). This guy

> <https://micahflee.com/2014/04/dual-booting-qubes-and-ubuntu-with-encrypted-disks/>

> got it working on Ubuntu, but he installed Ubuntu first, while I installed
> Qubes first. The last resort would be to follow him and reinstall from
> scratch, but I want to learn something.
>
> What have I tried:
>
> - installed Qubes OS 3.1 (OK)
> - updated Qubes OS 3.1 dom0 and Fedora23 template (OK)
> - installed the latest Fedora 23 Workstation respin (OK, but reported some
> GRUB2 bootloader failure; nevertheless Fedora booted fine)
> - updated the latest Fedora 23 Workstation (OK, there were no updates)
> - tried to boot Qubes (fail, no menuentry in GRUB2 boot menu yet)
> - created a menuentry for Qubes (OK, see bellow)
> - tried to boot Qubes (fail, see screenshot

> <https://drive.google.com/open?id=1EWjuC3evDB7Duq4dF20drVO0Fa6m6fSAvA>)
> - googled this workaround
> <https://ask.fedoraproject.org/en/question/79005/xen-on-fedora-23-boot-issue/> and

> installed the package
> - I get thgrough the error message now, but after a few seconds the system
> reboots itself and does that infinitely
>
> How I added the menuitem:
>
> I booted into Fedora and went to the `/etc/grub.d/` folder, and edited the
> `40_custom` file. In there i pasted everything between the xen section of
> the /boot/grub2/grub.cfg file on the qubes boot partition, into the end of
> that `40_custom` file on Fedora. Then I executed the following command
> under root
>
> `grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg`
>
> and the qubes menuitems got generated into the `grub.cfg` and became
> visible in the boot menu. I can provide the contents of grub.cfg later, in
> the meantime I'd like to know what might got wrong and how to fix it.
>
> Notable information:
>
> - Qubes boots from GPT on Legacy BIOS (with 2MB biosboot partition)
> - Fedora boots from GPT on UEFI (with EFI partition)

This is main problem - you're trying to boot legacy system (Qubes here)
from UEFI grub (Fedora). You need to choose one of them - either boot
both in legacy mode, or both in UEFI mode (or change bios settings at
each boot...). So, one option is to setup Fedora in legacy mode - then
putting Qubes legacy boot entry in 40_custom (as you've done) should be
enough.

It is also possible to boot Qubes from UEFI, but it requires different
configuration. And in most cases Grub doesn't make it easier... Anyway
you can try:

1. Locate xen-4.6.0.efi - it should be installed into /boot/efi/EFI/qubes/.
Place it in your EFI System Partition (if /boot/efi isn't there
already).

2. Create xen.cfg in the same directory with a content like this:

[global]
default=4.1.13-8.pvops.qubes.x86_64

[4.1.13-8.pvops.qubes.x86_64]
options=loglvl=all
kernel=vmlinuz-4.1.13-8.pvops.qubes.x86_64 root=/dev/dm-2
vconsole.font=latarcyrheb-sun16 rd.lvm.lv=qubes_dom0/root
ramdisk=initramfs-4.1.13-8.pvops.qubes.x86_64.img

Adjust kernel version, root= and rd.lvm.lv= parameters.

3. Make sure the kernel and initramfs files are in the same directory as
xen.cfg (most likely you'll need to copy them from /boot). It is
critical to have them on EFI System Partition (even if Grub can load
them from other places).

4. Add grub entry:

menuentry 'Qubes UEFI' {
chainloader /EFI/qubes/xen-4.6.0.efi
}

Look here for related troubleshooting:
https://www.qubes-os.org/doc/uefi-troubleshooting/
https://github.com/QubesOS/qubes-issues/issues/794

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXLmtcAAoJENuP0xzK19cseYQH+wfEKT910kp5dOL7gOy23RMf
OUb3sPXks62ARLlzNDQ87INcykcnIZU5qzdgWtYrohI+7nUguGrfSroZHCpwxzlF
BSlMWNOl2/nj5QxYjI5BjHqREJ0R5iO97NP+sqi10S5wI+MmStrdJ+sT9ojnQe9C
6jqAht1+9UbEGXo7+7vB/Zl0LhdEMX65Z3phwRGCsspr9UJEN4yINaKGPPfdVG5C
4SXeJxMWMtmkz4LwvcvB2lwrge6S3JGp3BFwcutk/YjOa53zXTFhVYdsnxXjbLpG
sPV7dN2HobOzVPMgzyM714njKTeqll8suOT18HHWh7RXR1B6SUJtsASNMtBKcZA=
=qbmu
-----END PGP SIGNATURE-----