New emulator code base (qemu-android) and "ranchu" virtual board.

[David Turner]

The current Android emulator sources, which are located on [1], are based on a very old fork of the QEMU upstream sources. The fork itself was created in 2006, iirc, with several modifications to make it usable on Windows and OSX, a flurry of other fixes, and plenty of Android-specific features. While over the time, many upstream features have been back-ported manually to the Android codebase, but both source lines have evolved in ways that make them extremely divergent.

To support ARM64 emulation, a new codebase [2], currently named "qemu-android", and based on QEMU 2.0, has been setup (with a _lot_ of support from Linaro, thanks again).

At the moment, qemu-android is a minimally modified version of QEMU 2.0, that includes a few patches required to support Android running on a new virtual board named "ranchu" [3].

It currently lacks many features to be a usable alternative for the SDK. However, we plan to improve it considerably in order to emulate the next major Android release with it.

The old emulator will be maintained and used to emulate system images of previous releases. While it would be nice to have a single binary capable of emulating all platform releases, achieving this may require too much work.

1) Main differences between "ranchu" and "goldfish":

We use the name "ranchu" to identify the new kernel configuration supported by the new codebase. The major difference with "goldfish" is that it tries to rely less on goldfish-specific devices, and more on existing ones provided by the upstream kernel.

For example, it uses virtio-block devices to emulate the NAND and SDCard devices (which require their own virtual devices in the old codebase). Note that this also has the potential of providing much better performance (likely but untested so far).

At the moment "ranchu" is only supported for the "arm64" architecture in the android-goldfish-3.10 branch on https://android.googlesource.com/kernel/goldfish

There are no plans to back-port it to 3.4 or earlier. On the other hand, we need similar configurations for other CPU architectures (32-bit ARM, x86, x86_64, MIPS and possibly MIPS64).

Technically, "ranchu" on ARM64 is based on "mach-virt", an ARM-specific paravirtualized platform. Ideally, I would love if the kernel provided a single paravirtualized platform that could vary only on the emulated CPU, but it seems that something like this doesn't exist, i.e. emulating an x86 device still essentially requires emulating a more or less complete PC.

I would love to discuss with people on this list about the best way to move forward with ranchu. Porting goldfish-specific patches from one kernel branch to the other has always been a hassle, and a source of surprises, so anything that can reduce this load would be welcomed. For example, I know that Linaro has experimented with replacing the goldfish pipe with virtio-char, but early results gave noticeable performance degradation (which might be critical for GPU emulation).

Ranchu is not really documented at the moment, so people will have to look at the sources in both the kernel and qemu-android repositories. Hopefully, I'll come up with a detailed doc like the existing docs/GOLDFISH-VIRTUAL-HARDWARE.TXT

2) Main differences between the old and new codebases

2.1) Source code size:

• The old codebase is about 400K of lines of code, split between 100K of Android-specific code, and 300K of QEMU-specific one (though with some patches not found in usptream).
  
  
• The new codebase is about 1500K of lines of code, with only 8K of Android-specific code (at the moment).
  

The patches for the new codebase add support for the following 'goldfish' virtual devices, which are also part of 'ranchu': framebuffer, audio, battery, pipe. They also add support for the "ADB bridge" (which makes the guest system visible and usable through "adb"), as well as the Android console (more precisely a subset of console commands).

Note that GPU emulation is a work-in-progress, you can look at the "ranchu-opengles" branch on the qemu-android repository if you want to play with it.

2.2) Build:

To build the old codebase, one has to checkout a few repositories from the AOSP source servers (ProTip: use 'repo' with 'repo init -u https://android.googlesource.com/platform/manifest -b ub-tools-master' to get a minimal AOSP checkout that let you build the emulator), then use external/qemu/android-rebuild.sh to rebuild everything and run a minimal unit test suite.

The new codebase has a very different build system. The easiest way is to use the script and dependency packages provided by https://qemu-android.googlesource.com/qemu-ranchu

The scripts/rebuild.sh script there can be used to rebuild a qemu-system-aarch64 executable for all 3 supported host platforms.

At some point, the qemu-android and rebuild scripts will be moved to the AOSP servers as well. I'm currently undecided about how to best package the dependencies (glib, libffi, sdl, etc...)

2.3) Launching:

The Android emulator is typically launched by calling the 'emulator' program, which is a tiny wrapper that will find the AVD's virtual CPU type, then launch the appropriate emulation "engine" (e.g. emulator64-arm) based on it. It also modifies the library search path to ensure the engine finds the GPU emulation libraries (e.g. libOpenglRender.so) and the PC BIOS images (required for x86 emulation).

The emulation engines are built by android-rebuild.sh, they are essentially a shell around the old QEMU fork that adds a custom UI and some Android specific features (e.g. ADB bridge, console) in a single executable.

For ARM64 support, emulator64-arm64 is instead another wrapper that is used to launch the real qemu-system-aarch64 executable (built from the qemu-android sources). Its main purpose is to translate the Android emulator-specific command-line options into the corresponding QEMU command-line one, which is generally very error-prone to setup correctly.

Its source is under external/qemu/android/qemu-launcher/, note that this source currently only supports ARM64, but this will change very soon.

I will soon create a new post detailing how to build the qemu-android binaries and use them with 'emulator'.

3) Code refactoring

Having two different codelines is never easy. At a minimum, it means a different set of features and bugs to care for and maintain. To reduce this burden, we've started refactoring the Android-specific part of the old emulator code base to make it easy to port to the new code base.

Ideally, I'd like to have a single static library (let's call it 'android-emulation') that could be linked to either one of the two code bases, and which could have its own set of unit and regression tests.

Another reason why this is important is to make the Android-specific patches to the new QEMU code base as minimal as possible. This would make the code easier to rebase on top of newer versions of upstream QEMU.

This requires defining a proper library interface, with clearly-defined semantics for object lifetimes, event handling, so expect more about this in a future post.

In the meantime, don't be too surprised to see a lot of changes in the old code base related to this.

4) Testing

We need it, and a lot of it :)

While we're setting up a Google-internal test infrastructure to better detect regressions in the emulator itself, SDK system images and the platform build, we also need to setup a reasonable test suite as part of the open-source tree. This will help third-party contributors and ensure issues are detected more quickly during development.

There is no firm plans on how to achieve this right now, but I've slowly being adding unit tests to the Android-specific parts of the old code base, and will keep doing so.

There is also a new set of AOSP build products named "qemu_$ARCH" (e.g. qemu_arm-userdebug), which can be used to build a minimal version of the Android system (without any Dex runtime or graphics interface). The goal is to use its root shell or adb connection to install/run various test programs to perform overall testing of specific emulation features, at a pretty low level.

This should also be a good way to identify issues related to emulator changes, compared to the one that happen routinely by platform changes that were not sufficiently tested (which are much more routine at this point :-( ).

5) Relationship with upstream

In an ideal world, we would not need a fork, and all code would live on the upstream QEMU git.

In reality, things are different: there is little chance that the upstream team would want to maintain 100K+ lines of code that are completely specific to Android, and for good reason. That's why the refactoring effort is so important, we need to find a way to maintain the Android-specific QEMU patches as small as possible, and push as much stuff into the android-emulation library.

Even with smaller changes, it's crucial to have a good set of tests, that exercise these Android-specific features, added to the QEMU test suite, and clear documentation about the implementation being added. This may require a stub or minimal mock version of android-emulation.

Finally, we may want to dedicate serious engineering resources to better continuous integration of upstream QEMU that would also exercise the Android bits.

Until we reach such a situation, we will have to maintain a separate fork and continue to rebase it on top of recent QEMU changes.

- Digit

[1] https://android.googlesource.com/platform/external/qemu.git

[2] https://qemu-android.googlesource.com/qemu-android/

[3] The Android kernel team uses fish names to identify hardware board configurations, the current emulator codebase provides virtual devices for a configuration named "goldfish". The name "ranchu" was selected as a logical evolution for the new iteration :)

[Christoffer Dall]

Hi Digit,

Thanks for sharing this and summarizing!

On Mon, Nov 3, 2014 at 12:28 PM, David Turner <di...@android.com> wrote:

[...]

>
>
> 1) Main differences between "ranchu" and "goldfish":
>
> We use the name "ranchu" to identify the new kernel configuration supported
> by the new codebase. The major difference with "goldfish" is that it tries
> to rely less on goldfish-specific devices, and more on existing ones
> provided by the upstream kernel.

One clarification: We don't actually introduce a 'ranchu' board in
the kernel side of things, we rely on the machine model being
completely device-tree driven and it just happens to be called ranchu
on the QEMU side and identified as such in the compatible name in the
DT.

>
> For example, it uses virtio-block devices to emulate the NAND and SDCard
> devices (which require their own virtual devices in the old codebase). Note
> that this also has the potential of providing much better performance
> (likely but untested so far).
>
> At the moment "ranchu" is only supported for the "arm64" architecture in the
> android-goldfish-3.10 branch on
> https://android.googlesource.com/kernel/goldfish

The ranchu board will actually bood with virtio devices on newer arm64
kernels as well, but of course the kernel side support for specific
goldfish devices (thus adbd and such) will not be there.

Thanks,
-Christoffer

[Miodrag Dinic]

Hello David,

first of all, I would like to thank you for elaborate introduction to the goals and plans for Android emulator.

The team at IMG/MIPS has been working on the new Ranchu virtual board for MIPS (on the new code base QEMU [2]).
It has a lot in common with the ranchu model Google introduced for ARM.

The both boards have the following devices in common :

- Goldfish Frame buffer
- Goldfish Audio
- Goldfish Events
- Android PIPE
- Virtio MMIO transports support for Virtio devices

The rest of MIPS platform devices are :

- Goldfish PIC
- Goldfish Timer & RTC
- Goldfish TTY

This devices are modified to be independent from the Goldfish Platform Bus, which they were relaying in the old code base [1].
The MIPS ranchu board will be driven by the Goldfish kernel (3.10), updated with device tree support, and all of the above mentioned devices
will be registered using device tree information.

Device tree blob is created on the QEMU side and passed to the kernel, which relief's the kernel of having anything hard-coded.

We would like to start contributing to the new code base QEMU [2] and Kernel [3] and start preparing it for MIPS.
First in line of patches would probably be the changes for the MIPS R6.

Can you provide us with instructions about repositories and branches where you want these changes to be pushed?

[1] - external/qemu -  https://android.googlesource.com/platform/external/qemu.git
[2] - QEMU -  https://qemu-android.googlesource.com/qemu-android/
[3] - Kernel - https://android.googlesource.com/kernel/goldfish

Regards,
Miodrag Dinic

[Alex Bennée]

Christopher Covington <c...@codeaurora.org> writes:
> Hi,
>
> [snip--for full message see
> https://groups.google.com/d/msg/android-emulator-dev/dltBnUW_HzU/2tSZNLaVzmQJ]

>
>> 5) Relationship with upstream
>>
>> In an ideal world, we would not need a fork, and all code would live on
>> the upstream QEMU git.
>>
>> In reality, things are different: there is little chance that the upstream
>> team would want to maintain 100K+ lines of code that are completely
>> specific to Android, and for good reason. That's why the refactoring effort
>> is so important, we need to find a way to maintain the Android-specific
>> QEMU patches as small as possible, and push as much stuff into the
>> android-emulation library.
>

> I'm curious, have there been previous discussions with the QEMU maintainers
> that you can summarize or point me to?

Taking my Linaro hat off and certainly not speaking for Google or the
maintainers:

Whenever new functionality is added to QEMU we rely on having:

* Someone to maintain and support the code
* Interested users that can regularly test and report breakages

Without this functionality can bitrot without being noticed and impose
an additional maintenance burden on the rest of the code-base. The
original authors may well have different priorities (e.g. shipping product!).
Of course not working directly upstream does impose additional costs
in the long term as you either take on the maintenance burden of
backporting fixes to your older tree or fixing conflicts on re-basing
with the upstream.

>> Even with smaller changes, it's crucial to have a good set of tests, that
>> exercise these Android-specific features, added to the QEMU test suite, and
>> clear documentation about the implementation being added. This may require
>> a stub or minimal mock version of android-emulation.
>>
>> Finally, we may want to dedicate serious engineering resources to better
>> continuous integration of upstream QEMU that would also exercise the
>> Android bits.
>>
>> Until we reach such a situation, we will have to maintain a separate fork
>> and continue to rebase it on top of recent QEMU changes.

Having said all that if you look at the current ranchu branch you'll see
the delta has come down a fair bit and is in itself relatively
self-contained. This should reduce the pain of regularly re-basing and
I'm sure Google don't want to go through the major upheaval moving from
the very old QEMU fork that the current emulator to a more modern QEMU
too often.

I'm hopeful we can get to a point where basic Android support is
up-stremable (and defended upstream) and the heavy android specific
stuff becomes a simple mechanical re-basing operations. Of the android
changes (off the top of my head) we have:

* Machine descriptions (fairly self-contained)
* Simple event driver (again self-contained)
* android_pipe services (self-contained but replicates virt-io functionality)
* android console support (provides ADB specific interfaces to QEMU)
* Simple Android Frame Buffer
* OpenGL (out-of-tree, likely very android specific)

I suspect the first 2 or 3 could be up-streamed without too much trouble
but it would be interesting to know if having this basic android emulation
in master would be of any interest to the wider community?


--
Alex Bennée

[Christopher Covington]

Hi,

[snip--for full message see
https://groups.google.com/d/msg/android-emulator-dev/dltBnUW_HzU/2tSZNLaVzmQJ]

> 5) Relationship with upstream
>
> In an ideal world, we would not need a fork, and all code would live on
> the upstream QEMU git.
>
> In reality, things are different: there is little chance that the upstream
> team would want to maintain 100K+ lines of code that are completely
> specific to Android, and for good reason. That's why the refactoring effort
> is so important, we need to find a way to maintain the Android-specific
> QEMU patches as small as possible, and push as much stuff into the
> android-emulation library.

I'm curious, have there been previous discussions with the QEMU maintainers
that you can summarize or point me to?

> Even with smaller changes, it's crucial to have a good set of tests, that
> exercise these Android-specific features, added to the QEMU test suite, and
> clear documentation about the implementation being added. This may require
> a stub or minimal mock version of android-emulation.
>
> Finally, we may want to dedicate serious engineering resources to better
> continuous integration of upstream QEMU that would also exercise the
> Android bits.
>
> Until we reach such a situation, we will have to maintain a separate fork
> and continue to rebase it on top of recent QEMU changes.

Thanks,
Cov

--
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

[Yu Ning]

Hi Digit,

Thank you for all the writing you have done to explain "ranchu" to us :)

We at Intel have been playing with qemu-android and evaluating our options for enabling x86/64 emulation. Our intention is to use ACPI for the enumeration of goldfish devices, which is essentially similar to the device tree-driven approach for ARM64 and MIPS, but more consistent with x86/64 platforms.

We have built a prototype of the new x86/64 emulator. It is able to boot the x86_64 emulator kernel [1] to the shell with a functional goldfish battery device, enumerated to the kernel via ACPI. This works with both BIOS (SeaBIOS) and UEFI (OVMF [2]) as boot firmware.

Now we have the following questions for you:

1. We wonder if there are plans to use a more recent kernel for the Android emulator. There is a patch [3] for the kernel's ACPI driver that would simplify our kernel-side modifications. It has been merged into Linux kernel 3.16 and may not be easily back-ported to 3.10.

2. It seems qemu-android has not been updated lately, or probably there's a lot of work on your side which has not been made public. When do you think the new codebase will be ready for external contributions? Will the move to AOSP happen soon?

Thanks,
Yu

[1] https://android.googlesource.com/kernel/goldfish/+/android-goldfish-3.10
[2] http://tianocore.github.io/ovmf/
[3] https://github.com/torvalds/linux/commit/48459340b92b00ae1a75179f168ef20d3e61f264

[Rishi Ranjan]

Is there a document to start playing around with qemu-android. I was able to compile and start the qemu and launched it with parameters specified at https://www.linaro.org/blog/core-dump/running-64bit-android-l-qemu/. I have replaced parameters with my own compiled ranchu images and kernel with my own compiled kernel for ranchu. But when I start the emulator, I don't see anything in the emulator window. 

[David Turner]

Yes, look at section V of external/qemu/docs/SOURCE-CODE-ORGANIZATION.TXT that was added recently. It explains how to build qemu-system-aarch64 and copy it to the appropriate location to start it through the 'emulator' launcher program.

That works with the system image from aosp_arm64-userdebug, the current version of prebuilts/qemu-kernel/arm64/kernel-qemu, and even the recent Linaro source build based on QEMU 2.2.

Hope this helps

--
You received this message because you are subscribed to the Google Groups "Android emulator development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-emulator...@googlegroups.com.
To post to this group, send email to android-em...@googlegroups.com.
Visit this group at http://groups.google.com/group/android-emulator-dev.
For more options, visit https://groups.google.com/d/optout.

[Rishi Ranjan]

Hi David,

              Thanks for pointing the doc to me. It helped me to proceed further. I have found following issues with the document. Capturing it in case someone else sees tries this. 

1. I was using android-5.0.2_r1 branch but external/qemu did not have latest files which supports --qemu-prebuilts-dir option. I pulled head of git clone https://android.googlesource.com/platform/external/qemu and then compiled the the files and it worked as documented in external/qemu/docs/SOURCE-CODE-ORGANIZATION.TXT

2. I switched to ranchu-opengles branch of https://qemu-android.googlesource.com/qemu-android/ to see if anything gets displayed in the window but looks like it's still work in progress as mentioned by David in this thread. I don't get anything displayed in the SDL window. But I am able to look at the logcat logs. 

Thanks,

Rishi

[David Turner]

On Wed, Dec 10, 2014 at 7:20 AM, Yu Ning <yu....@intel.com> wrote:

Hi Digit,

Thank you for all the writing you have done to explain "ranchu" to us :)

We at Intel have been playing with qemu-android and evaluating our options for enabling x86/64 emulation. Our intention is to use ACPI for the enumeration of goldfish devices, which is essentially similar to the device tree-driven approach for ARM64 and MIPS, but more consistent with x86/64 platforms.

We have built a prototype of the new x86/64 emulator. It is able to boot the x86_64 emulator kernel [1] to the shell with a functional goldfish battery device, enumerated to the kernel via ACPI. This works with both BIOS (SeaBIOS) and UEFI (OVMF [2]) as boot firmware.

Now we have the following questions for you:

1. We wonder if there are plans to use a more recent kernel for the Android emulator. There is a patch [3] for the kernel's ACPI driver that would simplify our kernel-side modifications. It has been merged into Linux kernel 3.16 and may not be easily back-ported to 3.10.

Historically, the kernel version used for emulation has always trailed a lot behind the one used by the Android internal tree. E.g. we used 2.6.29 up to Kit-Kat (!), which switched to 3.4.

I believe even the current Android system is only supporting 3.14 at this point, so using 3.16 or higher for the emulator seems very unlikely right now.

 

2. It seems qemu-android has not been updated lately, or probably there's a lot of work on your side which has not been made public. When do you think the new codebase will be ready for external contributions? Will the move to AOSP happen soon?

Most of the changes are done by Linaro at this point, though I've been working to refactor the classic code base to reuse the code with the new one in the future (in particular all of , android/skin/).

I'm currently testing Alex Bennée's latest version to see if we can upgrade to it. Note that for now I'm only testing the arm64 build with it, but the changes do not seem related to this CPU architecture.

I don't know yet when the move to AOSP will happen though.

Regards

- Digit

 

--

[Yu Ning]

Thanks. Please see below.

On Thursday, January 8, 2015 at 10:20:21 PM UTC+8, David Turner wrote:

On Wed, Dec 10, 2014 at 7:20 AM, Yu Ning <yu....@intel.com> wrote:

1. We wonder if there are plans to use a more recent kernel for the Android emulator. There is a patch [3] for the kernel's ACPI driver that would simplify our kernel-side modifications. It has been merged into Linux kernel 3.16 and may not be easily back-ported to 3.10.

Historically, the kernel version used for emulation has always trailed a lot behind the one used by the Android internal tree. E.g. we used 2.6.29 up to Kit-Kat (!), which switched to 3.4.

I believe even the current Android system is only supporting 3.14 at this point, so using 3.16 or higher for the emulator seems very unlikely right now.

 

I see.  We are fine with implementing our ACPI-based approach in 3.10, as we are doing now. It just incurs some extra code, which is actually self-contained and will just go away when we later switch to 3.16+.

2. It seems qemu-android has not been updated lately, or probably there's a lot of work on your side which has not been made public. When do you think the new codebase will be ready for external contributions? Will the move to AOSP happen soon?

Most of the changes are done by Linaro at this point, though I've been working to refactor the classic code base to reuse the code with the new one in the future (in particular all of , android/skin/).

I'm currently testing Alex Bennée's latest version to see if we can upgrade to it. Note that for now I'm only testing the arm64 build with it, but the changes do not seem related to this CPU architecture.

I don't know yet when the move to AOSP will happen though.

I appreciate this information. In fact, since my last post, we have also improved our prototype significantly by enabling three more goldfish devices (events, framebuffer and pipe). It is now able to boot the x86_64 Lollipop image to GUI. We'll be happy to upload our patches to AOSP (or the current qemu-android, if that works) for review when it's ready.

Regards,

Yu