Android-x86 6.0 (Marshmallow) on HVM, relatively usable for app debugging

[Alex]

Hello everybody,
I've been able to successfully install and run an Android-x86 iso and
would like to share with you the steps.

The main, huge issue is the barely usable mouse: I think that using a
VNC server the problem can be overcome. And since I use a VNC connection
to a virtual OSX machine for debug, it will not bother me to use another
VNC connection to the android VM. As mentioned in the Subject, the
target here is *a*debug*vm*.

Next steps for me will be having ADB work via the qubes internal
networking, minimizing the security implications, as if the android vm
was the usual emulator that comes with the Android Sdk.

That said, what follows is a *really* long e-mail message; most of us
are not bound to per-byte download costs anymore, but I'm sorry anyway
if receiving this message caused you trouble for its size. I will be
extremely verbose to avoid any misunderstanding because of implicit
assumptions. I hope my most advanced readers will excuse me for this.

The procedure is inspired by the article at qubes-os.org/doc/hvm, the
issue #2233 in qubes-issues repository, and the "Android-x86 on Qubes"
thread here on the ML. I downloaded the Android-x86_64 6.0 r3 iso from
the project's site. I renamed android-x86_64-6.0-r3.iso to a6.iso for
ease of typing in dom0.

First, start by creating a suitable HVM:
[dom0]$ qvm-create android-6 --hvm --label red

Now, make any change you'd like using vm manager or the command line
qubes tools (I set the RAM to 4096 MB - you could tweak the disk sizes,
I suggest keeping 2vcpus). Android will be stuck in a boot loop if it
does not have enough RAM; give it at least 1GB: the more you give it the
faster it will run.

Next step is starting the VM with the ISO as a cdrom drive:
[dom0]$ qvm-start android-6 --cdrom=personal:/home/user/Download/a6.iso

Now proceed to the installation:
* Select "Install to Hard Disk" from boot menu
* Once inside the installer choose "Create/Modify partition"
* Choose "sda" (first entry)
* Answer "NO" to the "use GPT" (no point in using GPT...)
* cfdisk will start.
* create a new primary partition that spans the whole drive
* select New
* accept all defaults
* make it bootable
* select Bootable
* save
* select Write
* type "yes"
* press Enter
* exit cfdisk
* select Quit
* back to partition selection, choose the new row "sda1 unknown"
* format as ext4
* (yes, confirm that you'll "LOSE" a bunch of zeros...)
* format will look like it hangs at 0% for a couple of minutes.
* don't panic, just wait, it didn't really hang
* the installer will then ask if you want GRUB; choose YES
* will come reaaaaally handy later on
* the installer will ask if you want the /system as read-write
* it depends on you. If unsure, select NO
* after data has been copied, installer will "sync to disk"
* also looks like it hangs. It doesn't, just be patient ~30 sec
* if you get "Android-x86 installed successfully" then you are ok!

You completed the first stage of three. On with the libvirt domain
configuration!

The fact that you started the VM with the cdrom is permanently saved in
the XML config file. We'll have to tweak it a little to be able to use
our VM.

Let's copy the config file to a new location (the original file will be
overwritten by Qubes before every VM boot!)
[dom0 /var/lib/qubes/appvms/android-6]$ cp android-6.conf cust-a-6.conf

Now edit the new copy with your favorite editor. The two main edits are
these:
* remove the <disk type='block' device='cdrom'> node for the setup ISO
* edit <input type='tablet' bus='usb' /> to be of type='mouse'

Start the VM like this:
[dom0]$ qvm-start android-6
--custom-config=/var/lib/qubes/appvms/android-6/cust-a-6.conf

At GRUB menu, *before*the*timeout*, press "e"
* now press "e" again to edit the kernel command line
* at the end append the vga mode you'd like
* you can append "vga=ask" to view a list of modes
* I suggest vga=0x343, meaning 800x600x32bit
* looks like Android will get stuck trying to determine which
modes are supported by Qubes
* press Enter to (temporarily) confirm changes and go back
* press "b" to boot
* if everything is ok, in ~5 seconds you'll see the boot animation loop
* first time can take 5-10 minutes; on my machine it took 2 minutes
* at the end you'll be at the setup wizard
* if stuck with one vcpu at 100% for more than 5 minutes:
* check VGA mode or
* increase RAM
* kill VM from VM Manager and restart with command line
* remember to perform again all steps (add vga= parameter)

If you are at the Android setup wizard then you reached the second
checkpoint!

Time to get the thing running. It looks to me that the setup wizard
tries to configure the wifi adapter, or something like that, because it
gets stuck at the "Just one sec..." screen. So that's what I did:

* Start the welcome procedure by clicking the yellow round button
* If you are stuck at the "Just one sec..." screen, try crashing it
* For example, by going back and forth a couple of times
* The mouse isn't very pleasing to work with.
* If it desynchronizes too much, try pushing it to a corner
* then re-enter the window from that corner and move slowly
* If the crashes freeze the UI just kill the VM and try again
* remember again to add the vga= parameter from GRUB!
* If you are at the "Select Wi-Fi network" screen, you are ready!
* Just continue the wizard to the end
* Once the wizard is over and desktop appears, wait ~1-2 minutes for
the icons

That's it! Asking for the VM to shutdown in the VM Manager will bring up
the classic Android shutdown menu; confirming the "Power down" option
will yield a clean shutdown.

You will probably want to:
- set up networking to another VM and a VNC connection to work around
mouse acceleration problems
- permanently set the vga= option in the GRUB config

I think the terminal emulator included in Android-x86 may reveal useful
for both of these tasks, but...
* I already spent too much time getting Android to work
* This e-mail is already way too long

Let everybody know if you try to go any further; I'll be trying to setup
networked ADB for debug from Android Studio.

Thank you all for your time and patience reading this,
--
Alex

[Alex]

Following up on my previous post on setting up Android 6.0 as a Qubes
HVM for app debugging purposes, I'm fixing some of the things that were
left unsolved, namely:
* screen goes to unrecoverable standby after little time
* fix vga settings in grub config
* enable adb via tcp
* allow VNC connections

First, we fix the screen timeout. That's relatively easy:
* Open Settings
* Go to "About Tablet"
* Tap 8 times over the build number (until you're a developer)
* Go back to settings
* Go to newly added "Developer Options"
* Enable "Stay awake"

Then we need to setup TCP connection to our work appvm (or the one in
which Android Studio is installed and will be used). These steps are
adapted from qubes-os.org/doc/firewall
* Gather IP addresses (qubes ones) for work and android-6
* Open a shell in sys-firewall
[sys-fw]$ sudo iptables -I FORWARD 2 -s <workIP> -d <andIP> -j ACCEPT
[sys-fw]$ sudo iptables -I FORWARD 2 -s <andIP> -d <workIP> -j ACCEPT
* Open a shell in work appVM
[work]$ sudo iptables _I INPUT -s <andIP> -j ACCEPT
* Firewall in Android-x86 already allows connections, so test:
[work]$ ping <andIP>
[android-terminal-emulator]$ ping <workIP>
* Should be able to ping. If this is the case, follow instructions in
qubes-os.org/doc/firewall to ensure persistence of these settings in
both work appVM and sys-firewall.

Now we can configure the ADB TCP port for remote debugging:
* inside Android open the terminal emulator again if closed
[android-terminal-emulator]$ su
[android-terminal-emulator]# setprop service.adb.tcp.port 5555
[android-terminal-emulator]# stop adbd
[android-terminal-emulator]# start adbd
* check from work appVM (Android studio and the tools should have
already been installed, so that you have "adb" available)
[work]$ adb connect <andIP>:5555
* Should say "connected to 10.137.xx.yy:5555"
If this is the case, you now have a friendlier connection for your system!

Now our Android HVM will have to be booted from dom0 command line, but
we can make it easier by persisting the vga= config in grub. To do that,
we must restart Android and:
* from the GRUB menu select the second line (Debug Mode)
* boot will stop at a temporary root MirBSD Korn shell
* give it a couple of Enter keys to clear logs out of the way
* now type:
[android]# mount -o rw,remount /mnt
[android]# cd /mnt/grub
[android]# vi menu.lst
* edit the file as to append the VGA parameter
* press "i", move the cursor, add text
* save and quit
* press Esc, then type ":wq"
* exit the shell TWICE and android will try to complete its boot
* but because we did not set the "vga" parameter in the debug line,
it will most likely fail
* kill the VM from VM Manager

Now we can start the android vm with a relatively easy dom0 command, it
will automatically and quickly boot to a relatively usable android
(sometimes some google background service will crash) with no problems
with screen timeouts. The only problem remaining is the mouse, that can
be bypassed with a VNC connection.

Problem is, our Android is x86 and most VNC servers out there are for
ARM devices. There is an x86 build of android-vnc-server at
http://xmodulo.com/how-to-run-vnc-server-on-android-x86.html but it has
not PIE enabled (some exploit protection) so Marshmallow will refuse to
load it.

Now you have TWO paths available:
one is
* download the android-vnc-server project
* update it and compile it for x86 with PIE
* load it into our HVM and use it to connect

the other is
* patch the /system/bin/linker executable in Marshmallow to avoid PIE check
* load the already compiled assembly found at previous link
* use it to connect

I went through the second path, since I'm more of a reverse engineer
than a library archaeologist, and really don't like the idea of trying
to compile things from the past with obscure (to me) options.

I found a nice explanation at
https://forum.xda-developers.com/google-nexus-5/development/fix-bypassing-pie-security-check-t2797731/page13#127
about a guy that did exactly this patch for Android 5.1. Following his
steps I disassembled /system/bin/linker from our Marshmallow HVM and
found that the 4 bytes to patch for the check are at 0xD25A (file
offset): they start with 0x0F 0x85. Patch the 4 bytes with four NOPs
(0x90), save, and replace the linker with your patched version.
* put the new linker in /storage/emulated/0/linker_new
* restart the Android HVM in Debug mode again
* once there the layout of the FS is completely different
* find linker_new inside /mnt/android-6.0-r1/data/media/0/
* copy it over to /android/system/bin
* chmod it to 755
* exit TWICE from the MirBSD shell and kill HVM
* start android normally (via the qvm-start custom command, as usual)
* push droidvnc_x86 you downloaded from the link above into the VM
[work]$ adb push droidvnc_x86 /storage/emulated/0/
[work]$ adb shell
[adb]$ su
[adb]# mv /storage/emulated/0/droidvnc_x86 /system/bin/
[adb]# chmod 755 /system/bin/droidvnc_x86
[adb]# droidvnc_x86
* if everything is ok, you will see the messages
Initializing VNC server:
width: 1024
height: 768
bpp: 32
port: 5901
Initializing server...
25/08/2017 11:35:27 Listening for VNC connections on TCP port 5901
25/08/2017 11:35:27 Listening for HTTP connections on TCP port 5801
25/08/2017 11:35:27 URL http://localhost:5801

Now you can connect via any VNC client (say, remmina?), tweak the
connection quality settings, and use your Android HVM to debug apps!

Some more issues you may find:
* Network is connected via cable, and Android knows it, not via WiFi
adapters. Apps may get this situation wrong.
* You cannot obviously multitouch via VNC. You could not either via
normal HVM screen.
* Application may sporadically crash, and this is an x86 machine: many
native apps are compiled for ARM devices only, with all due consequences.
* You will have to start the VM via the dom0 shell command, until I get
to understand how to avoid having the XML overwritten by VM Manager when
starting from there.

Hope somebody finds this useful
--
Alex

[7v5w7go9ub0o]

NICE!

Thank You!