CentOS Virtualization From Multiple .vmdk
hype
I'm not having success creating a CentOS 7.6 (x86_64) VM from multiple .vmdk files. I've attempted to follow the official Qubes HVM guide and other guides on the Internet.
The files:
CentOS 7.x 64-bit-s001.vmdk
CentOS 7.x 64-bit-s002.vmdk
CentOS 7.x 64-bit-s003.vmdk
CentOS 7.x 64-bit-s004.vmdk
CentOS 7.x 64-bit-s005.vmdk
CentOS 7.x 64-bit-s006.vmdk
CentOS 7.x 64-bit-s007.vmdk
CentOS 7.x 64-bit-s008.vmdk
CentOS 7.x 64-bit-s009.vmdk
CentOS 7.x 64-bit-s010.vmdk
CentOS 7.x 64-bit-s011.vmdk
CentOS 7.x 64-bit.nvram
CentOS 7.x 64-bit.vmdk
CentOS 7.x 64-bit.vmsd
CentOS 7.x 64-bit.vmx
CentOS 7.x 64-bit.vmxf
This runs CentOS with a GUI successfully under VirtualBox on Linux.
I have tried these sections of following guide:
https://www.qubes-os.org/doc/hvm/#converting-virtualbox-vm-to-hvm
https://www.qubes-os.org/doc/hvm/#creating-an-hvm-domain
Here are steps I'm taking now to provide more information:
# Convert vmdk to raw:
qemu-img convert *.vmdk -O raw image.img
# Qube Manager - Create new Qube
# Name and label: centos
# Type: Standalone qube not based on a template
# Advanced: install system from device
# Start CentOS
qvm-start centos --cdrom=TEMP:/home/user/image.img
# centos window opens with the following output:
SeaBIOS (version ...)
Machine UUID ...
Booting from DVD/CD...
Boot failed: Could not read from CDROM (code 0004)
Booting from Hard Disk...
Boot failed: not a bootable disk
Booting from Floppy...
Bootfailed: could not read the boot disk
No bootable device.
# Trying with hddisk as per:
https://groups.google.com/d/msg/qubes-users/84IjinPx_ng/eWFv81S4tnsJ
qvm-start centos --hddisk=TEMP:/home/user/image.img
# F5 and Copying to Dom0 Suggestions also fail:
https://groups.google.com/d/msg/qubes-users/84IjinPx_ng/mYYfbj6mSeIJ
Could something here be the problem?
qemu-img convert *.vmdk -O raw image.img
I don't know what else to try...
Thanks for your help. :)
Drew White
2. use CLI and browse to the directory.
3. Run:
# qemu-img convert {path/to/vmdk}CentOS\ 7.x\ 64-bit.vmdk root.img
This does work. I have done it many times.
hype
Would you mind walking me through this more step-by-step? I've tried many variations of this and I keep failing. I'm using Qubes 4.0. To create the centos qube I do:
Qube Manager->
Qube->
Create new qube->
Name and label: centos red
Type: Standalone qube not based on a template
In a qube named TEST-VMDK, I have the CentOS root image here:
/home/user/root.img
Which I've created using your advice:
qemu-img convert {path/to/vmdk}CentOS\ 7.x\ 64-bit.vmdk root.img
Is this the correct way to create the StandaloneVM I need? Should it be/not be based on a template? Should it be the default (PVH) mode, or HVM? Should the kernel be the default (4.14.18-1) (current)? I don't know how to do this from the terminal using Qubes 4.0.
From dom0 I've:
qvm-run --pass-io TEST-VMDK 'cat /var/lib/qubes/appvm' > /home/user/root.img
I think this is wrong? Why does my PVH/HVM show up in /var/lib/qubes/appvms? Why do none of the qubes in that directory contain root.img? Where should my root.img reside? Is there a command to create the correct VM type and load a root image from a specified qube such as TEST-VMDK?
When I do:
qvm-prefs-centos
There's no line about a root.img so I must be doing something very wrong?
As you can see... a lot of step-by-step hand-holding would be amazing. I really need to understand how to do this. I have a lot of VMs I need to load this way...
Thank you so much for your help.
hype
SeaBIOS (version rel-1.10.2-0-...)
Machine UUID ...
Booting from Hard Disk...
Boot failed: not a bootable disk
Booting from Floppy...
Boot failed: could not read the boot disk
No bootable device.
hype
qvm-create centos --class StandaloneVM --property virt_mode=hvm --property kernel="" --property memory=4096 --property maxmem=4096 --property debug=True --label red
qvm-start centos --cdrom=TEST-VMDK:/home/user/root.img
and
qvm-start centos --hddisk=TEST-VMDK:/home/user/root.img
hype
https://www.turnkeylinux.org/blog/convert-vm-iso
Unfortunately the end of the guide is broken for my case and I'm unable to generate the iso. I've successfully extracted the root filesystem.
Now that I have all the data, how to I turn this into an iso file which will load the vm? Please help, I've been banging my head on this thing for a week and am lost. :(
awokd
Don't have any experience with it, though. Another thing you might try is
to create the HVM, then dd your loop device straight into the new HVM's
root LVM partition (as long as what's in loop is already a bootable
partition with GRUB). Be careful to not dd to the wrong one.
hype
The issue is that this CentOS VM was distributed as a learning environment for a specific course I'm taking with software/services/configurations/etc directly tied to the lessons. I need everything to be intact.
I've reached the limit of my experience in this process and can find no other material online to help me.
Your dd loopback advice exactly fits this step I'm stuck at. I don't know how to make this bootable. Sigh...
I wrote an email to linuxrespin.org yesterday pleading for help which has some information for you I dread to paraphrase/reformat so I'll repost here, because I list commands run and output of file/directory structures... This bends my brain. Apologies and thanks for helping with this mess:
-----------------------------------------------------------------------------------------------------------------------
Hi linuxrespin team!
I'm desperate and need your expertise. For one week now I've been bashing my brain trying to get this to work and I've reached the end of my capabilities. Can you please help me?
1. I have a CentOS virtual machine which runs perfectly in VirtualBox. The VM is in .vmdk format.
*. I run Qubes OS and need to run CentOS as a guest. I need to convert the .vmdk virtual hard drives into a format which Qubes will accept.
2. I've used qemu-img to convert the .vmdk files into a raw image.
3. I've mounted the raw image (LVM) onto a loop device and am able to access the file system.
4. I have zero understanding of how to make an iso / live iso from this.
Can you please help me?
I've used the guide below to aide me, but the final tool from the author's repository does not work for me. Do you know how I can take the CentOS filesystem I have access to and turning it into a bootable iso?
https://www.turnkeylinux.org/blog/convert-vm-iso
-----------------------------------------------------------------------------------------------------------------------
ls -l /media/sf_Linux/CentOS\ 7.x\ 64-bit/
total 54015160
-rwxrwx--- 1 root vboxsf 8684 Jun 7 2017 CentOS 7.x 64-bit.nvram
-rwxrwx--- 1 root vboxsf 3413508096 Jun 7 2017 CentOS 7.x 64-bit-s001.vmdk
-rwxrwx--- 1 root vboxsf 2366242816 Jun 7 2017 CentOS 7.x 64-bit-s002.vmdk
-rwxrwx--- 1 root vboxsf 4088070144 Jun 7 2017 CentOS 7.x 64-bit-s003.vmdk
-rwxrwx--- 1 root vboxsf 2242772992 Jun 7 2017 CentOS 7.x 64-bit-s004.vmdk
-rwxrwx--- 1 root vboxsf 226492416 Jun 7 2017 CentOS 7.x 64-bit-s005.vmdk
-rwxrwx--- 1 root vboxsf 786432 Apr 26 2017 CentOS 7.x 64-bit-s006.vmdk
-rwxrwx--- 1 root vboxsf 720896 Apr 26 2017 CentOS 7.x 64-bit-s007.vmdk
-rwxrwx--- 1 root vboxsf 3080192 Jun 7 2017 CentOS 7.x 64-bit-s008.vmdk
-rwxrwx--- 1 root vboxsf 720896 Apr 26 2017 CentOS 7.x 64-bit-s009.vmdk
-rwxrwx--- 1 root vboxsf 19136512 Jun 7 2017 CentOS 7.x 64-bit-s010.vmdk
-rwxrwx--- 1 root vboxsf 131072 Apr 26 2017 CentOS 7.x 64-bit-s011.vmdk
-rwxrwx--- 1 root vboxsf 1029 Jun 7 2017 CentOS 7.x 64-bit.vmdk
-rwxrwx--- 1 root vboxsf 69 Apr 26 2017 CentOS 7.x 64-bit.vmsd
-rwxrwx--- 1 root vboxsf 2888 Jun 7 2017 CentOS 7.x 64-bit.vmx
-rwxrwx--- 1 root vboxsf 272 Apr 26 2017 CentOS 7.x 64-bit.vmxf
-----------------------------------------------------------------------------------------------------------------------
qemu-img convert -f vmdk turnkey-core.vmdk -O raw turnkey-core.raw
mkdir turnkey-core.mount
loopdev=$(losetup -s -f turnkey-core.raw)
kpartx -a $loopdev
-----------------------------------------------------------------------------------------------------------------------
ls -l /dev/mapper/
total 0
lrwxrwxrwx 1 root root 7 Apr 16 21:06 centos-home -> ../dm-5
lrwxrwxrwx 1 root root 7 Apr 16 21:06 centos-root -> ../dm-2
lrwxrwxrwx 1 root root 7 Apr 16 21:06 centos-swap -> ../dm-4
lrwxrwxrwx 1 root root 7 Apr 16 21:06 centos-var -> ../dm-3
crw------- 1 root root 10, 236 Apr 14 23:39 control
lrwxrwxrwx 1 root root 7 Apr 16 19:03 loop0p2 -> ../dm-1
lrwxrwxrwx 1 root root 7 Apr 16 19:34 loop1p1 -> ../dm-0
lrwxrwxrwx 1 root root 7 Apr 16 19:34 loop1p2 -> ../dm-6
lrwxrwxrwx 1 root root 7 Apr 16 21:06 loop3p1 -> ../dm-7
lrwxrwxrwx 1 root root 7 Apr 16 21:06 loop3p2 -> ../dm-8
-----------------------------------------------------------------------------------------------------------------------
mkdir turnkey-core.mount
mount /dev/mapper/$(basename $loopdev)p1 turnkey-core.mount
^ the above command gives me access to:
-----------------------------------------------------------------------------------------------------------------------
config-3.10.0-514.16.1.el7.x86_64
config-3.10.0-514.2.2.el7.x86_64
extlinux
grub
grub2
initramfs-0-rescue-0a268e711a9e49e6958dee98a83bcef2.img
initramfs-3.10.0-514.16.1.el7.x86_64.img
initramfs-3.10.0-514.16.1.el7.x86_64kdump.img
initramfs-3.10.0-514.2.2.el7.x86_64.img
initramfs-3.10.0-514.2.2.el7.x86_64kdump.img
initrd-plymouth.img
lost+found
symvers-3.10.0-514.16.1.el7.x86_64.gz
symvers-3.10.0-514.2.2.el7.x86_64.gz
System.map-3.10.0-514.16.1.el7.x86_64
System.map-3.10.0-514.2.2.el7.x86_64
vmlinuz-0-rescue-0a268e711a9e49e6958dee98a83bcef2
vmlinuz-3.10.0-514.16.1.el7.x86_64
vmlinuz-3.10.0-514.2.2.el7.x86_64
-----------------------------------------------------------------------------------------------------------------------
mount /dev/mapper/centos-root turnkey-core.mount
^ alternatively, using this gives me access to:
-----------------------------------------------------------------------------------------------------------------------
ls -l turnkey-core.rootfs/
total 76
lrwxrwxrwx 1 root root 7 Jan 17 2017 bin -> usr/bin
drwxr-xr-x 2 root root 4096 May 14 2015 boot
drwxr-xr-x 2 root root 4096 May 14 2015 dev
drwxr-xr-x 148 root root 12288 Jun 7 2017 etc
drwxr-xr-x 2 root root 4096 May 14 2015 home
drwxr-xr-x 2 root root 4096 Apr 16 23:01 isolinux
lrwxrwxrwx 1 root root 7 Jan 17 2017 lib -> usr/lib
lrwxrwxrwx 1 root root 9 Jan 17 2017 lib64 -> usr/lib64
drwx------ 2 root root 4096 May 14 2015 lost+found
drwxr-xr-x 2 root root 4096 Nov 5 2016 media
drwxr-xr-x 3 root root 4096 Nov 5 2016 mnt
drwxr-xr-x 4 root root 4096 Apr 21 2017 opt
drwxr-xr-x 2 root root 4096 May 14 2015 proc
dr-xr-x--- 11 root root 4096 Jun 7 2017 root
drwxr-xr-x 2 root root 4096 May 14 2015 run
lrwxrwxrwx 1 root root 8 Jan 17 2017 sbin -> usr/sbin
drwxr-xr-x 2 root root 4096 Nov 5 2016 srv
drwxr-xr-x 2 root root 4096 May 14 2015 sys
drwxrwxrwt 9 root root 4096 Jun 7 2017 tmp
drwxr-xr-x 13 root root 4096 Jan 17 2017 usr
drwxr-xr-x 2 root root 4096 May 14 2015 var
-----------------------------------------------------------------------------------------------------------------------
mkdir turnkey-core.rootfs
rsync -a -t -r -S -I turnkey-core.mount/ turnkey-core.rootfs
umount -d turnkey-core.mount
kpartx -d $loopdev
losetup -d $loopdev
cat>turnkey-core.rootfs/etc/fstab<<EOF
aufs / aufs rw 0 0
tmpfs /tmp tmpfs nosuid,nodev 0 0
EOF
-----------------------------------------------------------------------------------------------------------------------
I've skipped "GOTCHA 2"
tklpatch-prepare-cdroot turnkey-core.rootfs/
^ this fails with:
-----------------------------------------------------------------------------------------------------------------------
tklpatch-prepare-cdroot turnkey-core.rootfs
# preparing cdroot
basename: missing operand
Try 'basename --help' for more information.
cp: omitting directory 'turnkey-core.rootfs/boot/'
-----------------------------------------------------------------------------------------------------------------------
I figure, I've got this all extracted, can't I just take this stuff and get the boot/kernel stuff write and do some mksquashfs magic to create an iso? I've never used mksquashfs and I actually have no idea what my next step might be. I've scoured the Internet and every blog, forum, mailing list, stack overflow, youtube video, etc. that I could stomach. One week later I'm at my limit.
-----------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------
When I mount centos-root, I get these files:
config-3.10.0-514.16.1.el7.x86_64
config-3.10.0-514.2.2.el7.x86_64
extlinux
grub
grub2
initramfs-0-rescue-0a268e711a9e49e6958dee98a83bcef2.img
initramfs-3.10.0-514.16.1.el7.x86_64.img
initramfs-3.10.0-514.16.1.el7.x86_64kdump.img
initramfs-3.10.0-514.2.2.el7.x86_64.img
initramfs-3.10.0-514.2.2.el7.x86_64kdump.img
initrd-plymouth.img
lost+found
symvers-3.10.0-514.16.1.el7.x86_64.gz
symvers-3.10.0-514.2.2.el7.x86_64.gz
System.map-3.10.0-514.16.1.el7.x86_64
System.map-3.10.0-514.2.2.el7.x86_64
vmlinuz-0-rescue-0a268e711a9e49e6958dee98a83bcef2
vmlinuz-3.10.0-514.16.1.el7.x86_64
vmlinuz-3.10.0-514.2.2.el7.x86_64
There's grub/boot stuff in here, kernal stuff... When I mount the first loopback device from /dev/mapper I get the / file structure... If I *were* to know which commands to run to press and iso, I don't know what directory structure I would use to include both the / structure and that other stuff, which looks to belong in /boot, no? But there's extra stuff in there, like all those image files, lost+found etc...
Then there's the case of modifying stuff in /etc/fstab as was suggested in the tutorial...
Whew...
I wish I could ping Joanna, Marek, Andrew, etc., but that's ridiculous maybe... is it? I have dozens of course VMs like this! :'-(
awokd
> awokd, what a relief to get some additional help here!
>
> The issue is that this CentOS VM was distributed as a learning
> environment for a specific course I'm taking with
> software/services/configurations/etc directly tied to the lessons. I need
> everything to be intact.
Might be able to create a new HVM, boot it with a rescue CD and fdisk it.
Then shut it down, loop mount that new partition, then copy all or tar and
untar from the below mount (seems like this one has everything needed).
Once that's done, boot with your rescue CD again and install GRUB.
Could also try similar with qemu-img converting to raw like
https://askubuntu.com/questions/4398/how-do-i-convert-a-multiple-part-vmdk-disk-image-to-qcow2/4598.
Then dd the raw img to your HVM partition, and use the rescue CD to
install GRUB/make it bootable.