Setting up Cubieboard2 to run kvmarm

[Isa Ansharullah]

Hi All,

Recently I have succeeded running kvm arm on Cubieboard2. Since I haven't seen any guide around yet, I might as well 

make one, hoping that it will be useful for someone.

As you know, Cubieboard2 is equipped with Allwinner20 which has dual core Cortex-A7. Cortex-A7 and Cortex-A15 

supports virtualization via HYP mode, this hardware support has long (since kernel 3.8 *cmiiw) been utilized by kernel-based 

virtual machine, or KVM for ARM. However, compared to Cortex-A15, support for Cortex-A7 in guest emulation counterpart 

(qemu/lkvm) and kvm module counterpart is not as complete yet.

Requirements :

0. Compiling environment

    I use Ubuntu 13.04 (Raring) on x86_64

    Install cross-compiler using

sudo apt-get install gcc-arm-linux-gnueabi

    this will install arm gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1) 

    You also need :

    1. Git 

    2. minicom (to access ttyS0)

 

1. Hardware

    a. Cubieboard2

    b. USB-TTL Cable (No support for HDMI)

    c. LAN Cable (for NFS -- you can try using mmc, Hans's repo has the patch ready)

    d. microSD 

2. Software 

    a. U-boot

        https://github.com/jwrdegoede/u-boot-sunxi.git (Hans Goede's repository)

    b. Linux kernel 

        https://github.com/jwrdegoede/linux-sunxi.git (Hans Goede's repository)

        configuration (for host) : https://gist.github.com/meriororen/7886941

        configuration (for guest) : https://gist.github.com/meriororen/7886948

    c. LKVM (No support for QEMU), this is another kernel tree..

        git://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git

    d. Host Rootfs 

        You can 'debootstrap' it, or you can just download tar file from ubuntu website 

        if you want to use ubuntu. I am comfortable with Ubuntu, so I just downloaded 

        it :

 wget http://cdimage.ubuntu.com/ubuntu-core/releases/13.04/release/ubuntu-core-13.04-core-armhf.tar.gz

    d. Guest Rootfs (you can use any)

        I use "debian-7.1-bare-armhf-2013-08-25.tar.xz" you can google it

    e. Board configuration script (aka script.bin)

        scripts -> https://github.com/linux-sunxi/sunxi-boards

        tools -> https://github.com/linux-sunxi/sunxi-tools

      

Compiling software :

1. U-boot

 Clone the u-boot repo, checkout sunxi-test branch, and then execute make inside the kernel dir :

git clone  https://github.com/jwrdegoede/u-boot-sunxi.git 

git checkout -b sunxi-test origin/sunxi-test

cd u-boot-sunxi 

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- Cubieboard2 -j8 

 if everything goes well, you will get u-boot-sunxi-with-spl.bin, you will use this.

2. Host Kernel

Clone the repo, checkout sunxi-test branch :

git clone https://github.com/jwrdegoede/linux-sunxi.git 

git checkout -b sunxi-test origin/sunxi-test 

cd linux-sunxi  

Copy the config from my gist (for host) to .config file, put it inside kernel tree. And then run

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j8 uImage dtbs LOADADDR=0x40008000

you will need arch/arm/boot/uImage and dtb file, arch/arm/boot/dts/sun7i-a20-cubieboard2.dtb from

this compilation to run the host (cubie2).

3. Guest Kernel

You can use the same kernel tree as the host kernel.

Copy the config from my gist (for guest) to .config file, put it inside kernel tree. And then run

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j8 zImage  

You will need arch/arm/boot/zImage only to run the guest.

4. Script.bin

Clone both script and tools repository, and then run "make" on sunxi-tools directory.

You will get fex2bin tool inside sunxi-tools/. Then you can make your script.bin :

./sunxi-tools/fex2bin ./sunxi-boards/sys_config/a20/cubieboard2.fex > script.bin 

Or you can just take script.bin out of your working Cubian (or else) image.

Preparing microSD :

source : http://linux-sunxi.org/Bootable_SD_card

1. Cleanup

For precaution, lets cleanup first part of your microsd (replace X with drive letter, you can see it with lsblk command) :

sudo dd if=/dev/zero of=/dev/sdX bs=1M count=1

 

2. Partitioning (using single partition)

Using fdisk, you can cleanup your current partition :

sudo fdisk /dev/sdX

And then press 'o' + return, to erase partition table. 

Press 'w' + return to apply your change. Fdisk will close after this.

Start another fdisk to create a new partition table.  

Create a new partition by pressing 'n' for new, 'p' for primary, '1' for partition number (first partition), the rest, you can use default. 

Pressing enter (return) in every command. Apply your change with 'w'.

3. Install your u-boot

Using dd :

sudo dd if=u-boot-sunxi-with-spl.bin of=/dev/sdg bs=1024 seek=8 

4. Formatting filesystem (if you want to use mmc instead of NFS)

sudo mkfs.ext4 /dev/sdX1

Preparing NFS server : 

source : http://linux-sunxi.org/How_to_boot_the_A10_or_A20_over_the_network

1. Install nfs-server on Ubuntu

sudo apt-get install nfs-kernel-server

2. Create a new share directory

mkdir -p /var/nfsexport/arch/

3. Edit /etc/exports to include this directory. Put the following line to the end:

/var/nfsexport *(rw,sync,no_root_squash,no_subtree_check)

4. Restart nfs-server daemon

/etc/init.d/nfs-kernel-server restart

5. Untar Ubuntu cd image to share directory 

tar xzvf ubuntu-core-13.04-core-armhf.tar.gz -C /var/nfsexport/arch/

6. Setup user & password using chroot 

 First, let's prepare the system for chroot, install qemu and required packages :

sudo apt-get install qemu binfmt-support qemu-user-static

 Then, copy qemu-arm-static to rootfs's bin directory :

sudo cp /usr/bin/qemu-arm-static /var/nfsexport/arch/usr/bin/

 Copy /etc/resolv.conf for internet access during chroot :

sudo cp /etc/resolv.conf /var/nfsexport/arch/etc/

 Chroot yourself into the rootfs :

sudo chroot /var/nfsexport/arch /bin/bash

 You can prepare your rootfs here as much as you like, you can set up your root password :

passwd

 Or install a lot of useful things before you start working on the real stuff. You can even add

 files, clone git directory, or install stuff while you are working on your cubieboard.

 I always install my favorite editor, vim beforehand. If you want to access your cubieboard from

 ssh, you can install openssh-server, however, this may not work on the first run (sometimes you

 forget something), it is wise to first boot your linux via serial.

 <IMPORTANT!!> In order to be able to access shell on tty serial, you must set up ttyS0 in your rootfs.

 To do that, see : https://help.ubuntu.com/community/SerialConsoleHowto 

7. Put your uImage, script.bin, and fdt blob (sun7i-a20-cubieboard2.dtb) inside your share dir :

sudo cp <your_kernel_dir>/arch/arm/boot/uImage /var/nfsexport/arch/

sudo cp <script.bin_path>/script.bin /var/nfsexport/arch/

sudo cp <your_kernel_dir>/arch/arm/boot/dts/sun7i-a20-cubieboard2.dtb /var/nfsexport/arch/ 

8. Put your guest kernel image into your root directory inside your rootfs :

sudo cp <your_kernel_dir>/arch/arm/boot/zImage /var/nfsexport/arch/root/

Preparing guest rootfs :

1. Make a 512mb harddisk image using qemu-img, and make ext4 filesystem out of it

qemu-img rootfs.img 512M 

sudo mkfs.ext4 rootfs.img

2. mount somewhere,

sudo mount rootfs.img /mnt

3. Download debian-7.1-bare-armhf-2013-08-25.tar.xz anywhere you find it.

4. Untar it (twice -- there is another tar file inside) into mount point, and you can configure things (you can even use chroot method like we did)

sudo tar xvf armhf-rootfs-debian-wheezy.tar -C /mnt

5. If you want root access inside your guest, just delete 'x' in /mnt/etc/passwd belonging to root

6. You also need to configure ttyS0 here, I haven't figure out how to setup virtio console on lkvm (TODO), you can just uncomment 

    ttyS0 configuration inside /mnt/etc/inittab

7. unmount rootfs.img

sudo umount /mnt

8. Put it inside your NFS share directory

sudo cp rootfs.img /var/nfsexport/arch/root

Booting :

1. Launch minicom, replace /dev/ttyUSB0 with whatever your serial device is, make sure you get the right configuration :

sudo minicom -s -c on /dev/ttyUSB0 

2. Put your microSD into your Cubieboard2. Plug in power. If you see U-boot prompt on minicom, you have configured 

  your u-boot correctly. At first, it will complain for nonexistent of several files like "boot.scr" or "uEnv.txt", since we are

  going to boot linux via NFS, you can ignore it.

3. Configure nfs in u-boot:

  First, specify ethernet mac address, you can use random mac address here, 

setenv ethaddr 5a:fe:b0:07:b0:07

  Disable autoload (it will stop u-boot from loading via tftp when you execute "dhcp") :

setenv autoload no 

  Next, let dhcp configure your ip address :

dhcp

  Set your NFS server ip address (your computer IP) and nfsroot :

setenv serverip <server-ip> 

setenv nfsroot ${serverip}:/var/nfsexport/arch

  Set this to have script.bin, kernel, and fdt blob load into ram later :

setenv nfs_load_self "nfs 0x43000000 ${nfsroot}/script.bin; nfs 0x46000000 ${nfsroot}/uImage; nfs 0x49000000 ${nfsroot}/sun7i-a20-cubieboard2.dtb;"

  Set this to insert ethernet mac address into fdt later :

setenv fdt_add_eth0 'fdt addr 0x60000000; fdt resize; fdt set ethernet0 mac-address "[5a fe b0 07 b0 07]"'

  Set bootcmd & bootargs :

setenv bootcmd "run nfs_load_self; run fdt_add_eth0; bootm 0x46000000 - 0x49000000"
setenv bootargs "console=ttyS0,115200 root=/dev/nfs nfsroot=${nfsroot},tcp ip=dhcp earlyprintk"

  Never forget to save all your configuration to microsd so you don't have to do it again later :

 saveenv

4. Reset your board by typing "reset".

If everything's okay, you will see kernel boot message on your minicom. And then you will see shell prompt at the end. Check your

environment variables to troubleshoot if you fail to boot. 

Setting up guest :

Once you get into your Cubie2 linux, you still have several things to prepare, first, make sure that /dev/kvm is available. 

And then you can start configuring LKVM :

1. Clone Will Deacon's kernel tree which contains kvm tool, you can do this inside your cubie or on your PC using chroot :

git clone git://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git

cd kvmtool 

git checkout -b kvmtoolwill origin/kvmtool/arm 

 Go inside tools/kvm/ directory, and execute make (you will need to install gcc, I will leave that to you)

cd tools/kvm

make 

2. Now that you have lkvm, guest zImage, and guest rootfs, nothing can stop you from running kvm!! (unless you forget something)

lkvm run -k zImage -d rootfs.img -p "console=ttyS0 root=/dev/vda earlyprintk" -c 2 -m 512

Please try the guide, and notify me if I forget something.

Cheers

Isa

 

 

[gkl...@gmail.com]

Hi Isa,

ever got Windows working on this scenario??

[Isa Ansharullah]

Hi gklipel,

As far as I could get, I have only run linux kernel on it (as stated on the tutorial).

AFAIK, the kvm-tools wasn't really intended to support "serious" OS, that's Qemu for you. 

Qemu wasn't supported yet on cortex-A7 processor at the time I did the tutorial.

The kvm-arm (and cubieboard2) supports has been undergoing development since then,

I suggest you find out the updates on kvm-arm mailing list, I'm sure you can get satisfiable

answers to your question there.

Good luck.

--
You received this message because you are subscribed to a topic in the Google Groups "Cubieboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cubieboard/4EGONZMoIAU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cubieboard+...@googlegroups.com.
Visit this group at http://groups.google.com/group/cubieboard.

For more options, visit https://groups.google.com/d/optout.

--
Isa Ansharullah

aLab, inc.

Tokyo, Shibuya-ku, Yoyogi

1-42-10

Brooklyn Court 2F