Re\: \[android\-building\] Re\: android10.0.0_rxx \: lunch qemu_trusty

Felix LeClair

unread,

Jun 29, 2021, 7:32:09 PM6/29/21

to android-...@googlegroups.com, julien....@gmail.com

Hi Julien, I'm trying to replicate your method for building an arm64 build of android that can be run in native QEMU, and have built both AOSP and the 5.10 kernel to your specification.

An issue however is that there seems to be no vendor img produced when done in a clean environment. are other steps needed prior to the ones below to generate vendor.img ? or should I instead be using files inside of some of the other images created by AOSP?

Thanks,

FelixCLC

Hi again,

Here is the AOSP 10 on ARM64 QEMU version of the procedure I follow. Andoid

5.10 arm64 kernel build included, like for x86_64.

Appart from using "vdb" and "vdc" into fstab instead of sdb and sdc, and

removing some "rild.rc" file, the approach is the same. With some

additionnal information at the end

repo init -u https://android.googlesource.com/platform/manifest -b

android-10.0.0_r47

repo sync

source build/envsetup.sh

lunch

m -j $(nproc)

# out/target/product/generic_arm64/system.img

# out/target/product/generic_arm64/vendor.img

Create and ext4 format the partition image files:

    qemu-img create rootfs.img 5G

    qemu-img create userdata.img 1G

    qemu-img create cache.img 500M

    losetup /dev/loop0 rootfs.img

    losetup /dev/loop1 userdata.img

    losetup /dev/loop2 cache.img

    mkfs -t ext4 /dev/loop0

    mkfs -t ext4 /dev/loop1

    mkfs -t ext4 /dev/loop2

    losetup -d /dev/loop1

    losetup -d /dev/loop2

*Mouting the rootfs partition *

    mount -t ext4 /dev/loop0 /where/you/want

cp -a [system.img mount point] [rootfs/mount/point]

The content of system.img may be copied inside a folder which is named like

the mount point. Move everything 1 folder above.

cp -a [vendor.img mount point] [rootfs/mount/point]

If the name of the "vendor.img" mount point is "vendor", everything should

now be inside the "/vendor" folder of the rootfs. If not, you have to move

it to /vendor folder

Delete these files:

    /vendor/lib/hw/camera.ranchu.so

    /vendor/lib/hw64/camera.ranchu.so

    /vendor/lib/hw/camera.ranchu.jpeg.so

    /vendor/lib/hw64/camera.ranchu.jpeg.so

    /vendor/lib/hw/vendor/lib/hw/gralloc.ranchu

    /vendor/lib/hw64/gralloc.ranchu

    /vendor/lib/hw/hwcomposer.ranchu

    /vendor/lib/hw64/hwcomposer.ranchu

    /vendor/lib/hw/vulkan.ranchu.so

    /vendor/lib/hw64/vulkan.ranchu.so

    /vendor/etc/init/rild.rc

Change /vendor/etc/fstab.ranchu, those 2 lines should be enough:

    /dev/block/vdb          /data           ext4

noatime,nosuid,nodev,nomblk_io_submit,errors=panic

wait,check,quota,reservedsize=128M,first_stage_mount

    /dev/block/vdc          /cache          ext4

noatime,nosuid,nodev,nomblk_io_submit,errors=panic

wait,check,quota,reservedsize=128M,first_stage_mount

Change /vendor/etc/init/hw/init.ranchu.rc by changing this line:

    setprop ro.hardware.egl emulation

to

    setprop ro.hardware.egl swiftshader

*Manufacturing an up to date 5.10 ARM64 Android kernel:*

git clone https://android.googlesource.com/kernel/configs

/!\ From configs/android-5.10/android-recommended.config, if you want to

use a mouse, remove the following line : # CONFIG_INPUT_MOUSE is not set

git clone https://android.googlesource.com/kernel/common -b android12-5.10

Start from copying "arch/arm64/configs/defconfig" file to the root of the

kernel tree, name it ".config",  and append to it:

    CONFIG_NETFILTER_ADVANCED=y

    Content of configs/android-5.10/android-base.config

    Content of configs/android-5.10/android-recommended.config

    Content of configs/android-5.10/android-recommended-arm64.config

make ARCH=arm64 menuconfig, exit, save: yes.

By looking with some tools, some of the things included into

x86_64_defconfig, android-base.config, android-recommended.config are not

included into the resulting ".config".

It works anyway, but it would be nice to know why theese option aren't

enabling.

    Still missing from arm64/configs/defconfig

        CONFIG_SYSVIPC=y

        CONFIG_ACPI_APEI_PCIEAER=y

        CONFIG_KSM=y

        CONFIG_IP6_NF_NAT=m

        CONFIG_IP6_NF_TARGET_MASQUERADE=m

        CONFIG_LEGACY_PTY_COUNT=16

        CONFIG_POWER_AVS=y

        CONFIG_BACKLIGHT_GENERIC=m

        CONFIG_QCOM_IOMMU=y

        CONFIG_NFS_FS=y

        CONFIG_NFS_V4=y

        CONFIG_NFS_V4_1=y

        CONFIG_NFS_V4_2=y

        CONFIG_ROOT_NFS=y

    Still missing from android-base.config:

        CONFIG_TRACE_GPU_MEM=y

    Still missing from android-recommended.config:

        CONFIG_BACKLIGHT_LCD_SUPPORT=y

        CONFIG_REFCOUNT_FULL=y

        CONFIG_SDCARD_FS=y

Now, some options that are still disabled, but should be enabled according

to android-base-conditional.xml:

    CONFIG_ARMV8_DEPRECATED=y

    CONFIG_CP15_BARRIER_EMULATION=y

    CONFIG_SETEND_EMULATION=y

    CONFIG_SHADOW_CALL_STACK=y

    CONFIG_SWP_EMULATION=y

    CONFIG_BPF_JIT_ALWAYS_ON=y

    CONFIG_KFENCE=y

    CONFIG_USERFAULTFD=y

CONFIG_AIO and CONFIG_ANDROID_BINDERFS options (from android-base.config)

are causing early reboot without explanations. Would be nice to know why,

to keep them enabled and have something running... Until so, disable them

by adding at the end of .config:

    # CONFIG_AIO is not set

    # CONFIG_ANDROID_BINDERFS is not set

Adding VirtIO things:

    CONFIG_VIRTIO=y

    CONFIG_VIRTIO_PCI=y

    CONFIG_VIRTIO_PCI_LEGACY=y

    CONFIG_VIRTIO_BALLOON=y

    CONFIG_VIRTIO_INPUT=y

    CONFIG_VIRTIO_MMIO=y

    CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y

    CONFIG_VIRTIO_DMA_SHARED_BUFFER=y

    CONFIG_BLK_MQ_VIRTIO=y

    CONFIG_MEMORY_BALLOON=y

    CONFIG_BALLOON_COMPACTION=y

    CONFIG_PAGE_REPORTING=y

    CONFIG_DRM_GEM_SHMEM_HELPER=y

    CONFIG_DRM_VIRTIO_GPU=y

make ARCH=arm64 menuconfig, exit, save: yes.

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j $(nproc)

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

INSTALL_MOD_PATH=/home/user/Desktop/ INSTALL_MOD_STRIP=1 modules_install

*"Installing" the kernel:*

Place the the arch/arm64/boot/Image.gz file at the same place as the

rootfs.img, userdata.img and cache.img files

Place the content of /home/user/Desktop/lib/modules/5.10.37+/ into

/vendor/lib/modules/

*Unmount rootfs.img*

    umount /its/mount/point

    losetup -d /dev/loop0

*Ready to go:*

>From an x86_64 computer:

    qemu-system-aarch64 -M virt -cpu cortex-a72 -accel tcg,thread=multi

-smp 4 -m 2048 -kernel Image.gz -monitor none -parallel none -append

"root=/dev/vda rootfstype=ext4 ro init=/init selinux=1 checkreqprot=1

androidboot.selinux=permissive console=ttyAMA0 androidboot.hardware=ranchu

loglevel=8" -serial mon:stdio -vga std -device ramfb -device nec-usb-xhci

-device usb-kbd -device usb-mouse -bios

/usr/share/qemu-efi-aarch64/QEMU_EFI.fd -drive format=raw,file=rootfs.img

-drive format=raw,file=userdata.img -drive format=raw,file=cache.img

If you are lucky enough to have an arm64 machine with KVM available on it,

and enough RAM, into the above command line, remplace:

    qemu-system-aarch64 -M virt -cpu cortex-a72 -accel tcg,thread=multi

-smp 4 -m 2048

by

    kvm -M virt -cpu host -accel kvm -smp 4 -m 2048

In summary, from an arm64 computer :

    kvm -M virt -cpu host -accel kvm -smp 4 -m 2048 -kernel Image.gz

-monitor none -parallel none -append "root=/dev/vda rootfstype=ext4 ro

init=/init selinux=1 checkreqprot=1 androidboot.selinux=permissive

console=ttyAMA0 androidboot.hardware=ranchu loglevel=8" -serial mon:stdio

-vga std -device ramfb -device nec-usb-xhci -device usb-kbd -device

usb-mouse -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd -drive

format=raw,file=rootfs.img -drive format=raw,file=userdata.img -drive

format=raw,file=cache.img

*Bits of additionnal information:*

Having ARM64 AOSP running on QEMU from x86_64, even with "-accel

tcg,thread=multi" which makes things much better, you still need to wait 30

seconds before having "android" boot logo (on an AMD 3700X) and 350 seconds

(almost 6 minutes) on first boot before having the android main screen with

icons and all stuff (2 minutes at the following boots).

There are some repeated messages about camera 2.4 stuff and radio@1.0. You

may remove the

/vendor/etc/init/android.hardware.camera.provi...@2.4-service.rc, but

messages about "android.hardware.camera.provider@2.4" will still appear, so

you may remove the associated entries (camera and radio) from

/vendor/manifext.xml. Then no more messages about it.

In case of need to debug, you may type "logcat" in the console: this give

way too much informations but you may spot information on what you are

looking for, or few lines above some service crashing (sometimes some

messages contains usefull information to tell you why it is going to crash

just after). Into the "userdata.img" partition you may also find

"/tombstones" folder, containing crash logs of services.

In order to cleanly poweroff your emulated system you can type "reboot -p"

so that the "userdata.img" is kept as clean as possible.

--

--

You received this message because you are subscribed to the "Android Building"

mailing list.

To post to this group, send email to android-...@googlegroups.com

To unsubscribe from this group, send email to

android-building+unsubscr...@googlegroups.com

For more options, visit this group at

http://groups.google.com/group/android-building?hl=en

---

You received this message because you are subscribed to the Google Groups

"Android Building" group.

To unsubscribe from this group and stop receiving emails from it, send an email

to android-building+unsubscr...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/android-building/0dd7a47b-4fdd-48d3-b5bf-fe0af08f8c5en%40googlegroups.com.

Julien Robin

unread,

Jun 29, 2021, 7:32:29 PM6/29/21

to Felix LeClair, android-...@googlegroups.com

Hi Felix,

I took a look again into all my stuff to double check, using the described list of commands* (using Debian 11 x86_64 computer, but was also working on Debian 10) I obtained both "system.img" and "vendor.img" into the output folder.

Commands*:

source build/envsetup.sh
lunch aosp_arm64-eng
m -j $(nproc)

Attached here is a tutorial I made myself about AOSP 11 (arm64) on QEMU: for now it works but not perfectly (only works using and arm64 KVM, unstable, but starting, and allowing to browse the main screen, menus, settings... looks like instability is related to swiftshader, according "tombstones" files - Janne Karhunen messages from few days ago may drive to something interesting to replace swiftshader by something more efficient on qemu!). In case you need to compare your results, here is a link about my today attempts following the attached tutorial: https://pix-server-sorel.luoss.fr/Manual/Android/qemu-kvm-aarch64-android-11.0.0_r38/

There is a kernel (today's 5.10.43 version), its .config file, and prepared filesystem images to boot android 11.0.0_r38 on arm64 qemu-kvm (using a RPi4 for example, at least, that's what I used, 8GB model to have plenty of RAM, but 4GB may be enough). Into the "build-output" sub-folder you'll find the "system.img" and "vendor.img" that landed into my "out/target/product/generic_arm64/" folder (before and after I used "simg2img" command to convert them into standard ext4 partitions images).

Good luck !

Julien

how-to-AOSP-11-on-QEMU-arm64.txt

Janne Karhunen

unread,

Aug 26, 2021, 1:49:14 PM8/26/21

to Android Building

Hi,

I'm getting pretty close with a legit config. The bootup log is almost entirely clean. See this:

https://github.com/jkrh/kvms/blob/master/scripts/run-qemu6-virt-android-super.sh

If run with the CPU emulation it generates some SIGILLs from zygote (even with the QEMU MAX cpu), but overall:

[ 0.856757] [drm] pci: virtio-gpu-pci detected at 0000:00:02.0

[ 0.857639] [drm] features: +virgl +edid

...

08-20 07:08:30.867 345 345 D libEGL : loaded /vendor/lib64/egl/libEGL_mesa.so

08-20 07:08:30.948 345 345 D libEGL : loaded /vendor/lib64/egl/libGLESv1_CM_mesa.so

08-20 07:08:31.133 345 345 D libEGL : loaded /vendor/lib64/egl/libGLESv2_mesa.so

...

08-20 07:08:34.216 345 345 I RenderEngine: OpenGL ES informations:

08-20 07:08:34.216 345 345 I RenderEngine: vendor : Mesa/X.org

08-20 07:08:34.217 345 345 I RenderEngine: renderer : virgl

08-20 07:08:34.217 345 345 I RenderEngine: version : OpenGL ES 3.1 Mesa 21.2.0-devel

I have the kvm support in there as well, so if you have a capable hardware it might actually run.

--

Janne

Janne Karhunen

unread,

Sep 14, 2021, 7:18:33 PM9/14/21

to Android Building

Hi,

Small update although I haven't been able to hack on this for a while. It runs, and stays running, under the QEMU emulation AND under the KVM on a mobile phone. Mesa graphics stack is initialized and the android animation starts to rotate with virtio acceleration, but there is still something wrong with the appserver (probably an android build issue). What happens is that the 32bit variant of the appserver dies to following:

F DEBUG : signal 4 (SIGILL), code 1 (ILL_ILLOPC), fault addr 0x700303da (*pc=0x4000e8bd)

Since that is a 'illegal operation code' and it happens under the kvm as well as under the emulation, the given backtrace and the error code are probably broken. It claims that the error is here: /apex/com.android.art/javalib/arm/boot.oat (art_jni_trampoline+34) which translates to instruction 'add.w r12, sp, #4' that is prefectly fine, no way for that to generate a SIGILL as far as I know. Need to debug some more I guess.

Re\: \[android\-building\] Re\: android10.0.0_rxx \: lunch qemu_trusty_arm64\-userdebug fails

Felix LeClair

Julien Robin

Janne Karhunen

Janne Karhunen