Anbox?

[elsieb...@gmail.com]

This sounds interesting: https://www.slashgear.com/anbox-runs-android-apps-on-linux-just-like-native-apps-13481922/
Will this work on Qubes?

[Alex]

Since it looks like it's based on simple containerization instead of
virtualization it should work.

A little background: I usually do docker / rkt development on a "work"
appVM in qubes, and it works just out of the box. I've had a nasty but
simple quirk with flatpak, because of the already-mounted /proc/xen, but
unmounting that seems not to break anything and let some flatpak
archives work.

I really hope that this could be a viable solution for debugging
application: that would really speed up my qubes-based android
development tasks, instead of using the emulator with arm images which
is - to put it mildly - kinda slow.

--
Alex

[Reg Tiangha]

On 04/14/2017 12:45 AM, Alex wrote:
> Since it looks like it's based on simple containerization instead of
> virtualization it should work.
>
> A little background: I usually do docker / rkt development on a "work"
> appVM in qubes, and it works just out of the box. I've had a nasty but
> simple quirk with flatpak, because of the already-mounted /proc/xen, but
> unmounting that seems not to break anything and let some flatpak
> archives work.
>
> I really hope that this could be a viable solution for debugging
> application: that would really speed up my qubes-based android
> development tasks, instead of using the emulator with arm images which
> is - to put it mildly - kinda slow.
>
> --
> Alex

A potential issue is that it relies on OpenGL, but it might still work
in a Qubes VM. It's definitely on my list of things to try inside of a
Xenial template this long weekend, along with all the kernel stuff.
Unless someone does it first, in which case, please share the results
with the rest of the list!

[Alex]

On 04/14/2017 09:23 AM, Reg Tiangha wrote:
>
> A potential issue is that it relies on OpenGL, but it might still work
> in a Qubes VM. It's definitely on my list of things to try inside of a
> Xenial template this long weekend, along with all the kernel stuff.
> Unless someone does it first, in which case, please share the results
> with the rest of the list!
>

Fedora appvm, installed snapd (sudo dnf install snapd) and run the magic
command
$ snap install --classic anbox-installer && anbox-installer

requires root :D

Run with sudo, snap install does its thing (but prints a suspicious
"Waiting for restart" which I don't know how to interpret), but there's
no anbox-installer to be found by bash.

It is found by "snap run anbox-installer", but that command fails pretty
soon with "execv failed: no such file or directory".

That's all for now; as I said, I'm quite eager to have a fast-running
android environment on Qubes for app development and debug, and it seems
that Anbox (only having ADB as the app installation method as of now)
would be fit for the job.

I join Reg Tiangha's request for more information if anybody does any
further testing; i.e. wether it works on other templates, if snap needs
to be installed in the template to run, or anything else that may help.

--
Alex

[Reg Tiangha]

On 04/14/2017 02:02 AM, Alex wrote:
> Fedora appvm, installed snapd (sudo dnf install snapd) and run the magic
> command
> $ snap install --classic anbox-installer && anbox-installer
>
> requires root :D
>
> Run with sudo, snap install does its thing (but prints a suspicious
> "Waiting for restart" which I don't know how to interpret), but there's
> no anbox-installer to be found by bash.
>
> It is found by "snap run anbox-installer", but that command fails pretty
> soon with "execv failed: no such file or directory".
>
> That's all for now; as I said, I'm quite eager to have a fast-running
> android environment on Qubes for app development and debug, and it seems
> that Anbox (only having ADB as the app installation method as of now)
> would be fit for the job.
>
> I join Reg Tiangha's request for more information if anybody does any
> further testing; i.e. wether it works on other templates, if snap needs
> to be installed in the template to run, or anything else that may help.
>

Well, I think I read it was dependant on Mir or Unity or something
Ubuntu-centric like that, so for the moment, it'll only work on
Ubuntu-based machines. But I could have misread something somewhere and
I haven't had time yet to play with it myself.

[Vít Šesták]

Alex, have you tried to debug it with strace or something similar? Maybe there is some unlisted dependency…

OpenGL should work on Qubes through llvmpipe. Although its performance is not perfect, it should suffice for many Android apps. Except that I had problems with OpenGL (error message like “missing OpenGL”) when trying to run Android apps in Chromium, but I haven't seen such problem in any other case, including those that surely do use OpenGL.

Regards,
Vít Šesták 'v6ak'

[Dominique St-Pierre Boucher]

I built an ubuntu template to try to install anbox but unfortunately, it tries to create kernel module through DKMS and failed... I am not knowledgeable enough to try to fix this issue.

Dominique

[Reg Tiangha]

I got stuck too, but I might try again tomorrow. It complains about the
kernel not being patched with AppArmor 2.4 compatibility, which implies
having to use an Ubuntu kernel (I was trying this in a TemplateVM),
which then implies doing this on either an HVM or pvgrub2 VM. So if
going the pvgrub2 route, you'll need to compile qubes-kernel-vm-support
yourself (easy to do, especially if you built the template to begin
with; the deb files should still be there in the package directory). The
problem is that I can't figure out how to install grub on this thing. It
says it can find /dev/mapper/dmroot and I tried to grub-install it onto
all the /dev/xv** devices, but that didn't seem to work either. I'm
probably missing something stupidly simple, but I need to call it a day
for now. Hopefully someone else out there will have better luck.

[Vít Šesták]

IIUC, for pvgrub, you need to choose it as kernel for the particular AppVM. Have you done so?

[Reg Tiangha]

On 04/15/2017 12:12 AM, Vít Šesták wrote:
> IIUC, for pvgrub, you need to choose it as kernel for the particular AppVM. Have you done so?
>

I did; I boot coldkernel on Debian templates, so I know how to work with
pvgrub. The problem is that it just isn't generating a grub.cfg file.
This is the output of grub-mkconfig:

user@xenial-snaps:~$ sudo grub-mkconfig
Generating grub configuration file ...
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
set have_grubenv=true
load_env
fi
if [ "${next_entry}" ] ; then
set default="${next_entry}"
set next_entry=
save_env next_entry
set boot_once=true
else
set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
menuentry_id_option="--id"
else
menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
set saved_entry="${prev_saved_entry}"
save_env saved_entry
set prev_saved_entry=
save_env prev_saved_entry
set boot_once=true
fi

function savedefault {
if [ -z "${boot_once}" ]; then
saved_entry="${chosen}"
save_env saved_entry
fi
}
function recordfail {
set recordfail=1
grub-probe: error: cannot find a GRUB drive for /dev/mapper/dmroot.
Check your device.map.

and what gets stored in /boot/grub.cfg.new. It doesn't even make a
regular grub.cfg file.


HOWEVER, I think this whole native kernel thing was a red herring. The
problem with the Anbox installer was that apparmor wasn't running, even
though the dom0 qubes-vm-kernel I'm running supports it. Easily fixed with:

qvm-prefs -s XenialVM kernelopts "nopat apparmor=1 security=apparmor"

And right now the installation is going. We'll see what happens when it
finishes.

If I get this running, I might try again without installing
qubes-kernel-vm-support. That might be unnecessary after all and it
might have been apparmor not running that was the real problem.

[Reg Tiangha]

Edit: Err, maybe I spoke too soon. The Anbox installer is currently
trying to install a 4.4 kernel right now. No idea what'll happen when
it's done, but maybe pvgrub is actually needed in which case, I'll need
to figure out how to get grub working after all.

[Reg Tiangha]

I guess I was wrong. The Anbox installer tries to install a 4.4 kernel
and then fails compiling the Anbox dkms module due to the lack of kernel
sources (I'm running a Qubes dom0 vm kernel).

So I guess eventually, figuring out how to get a valid Grub
configuration working on this thing to boot a local Ubuntu kernel needs
to be done. It's weird that it doesn't work the same way as on my Debian
templates, though. My last resort is copying a valid grub.cfg from one
of my Debian templates over and manually editing it to point to use the
xenial kernel and initramfs, but I don't want to have to do that unless
absolutely necessary. But it's getting late in my part of the world, so
I'm calling it for now. If anyone else manages to figure this stuff out
in the meantime, please do share.

[Reg Tiangha]

On 04/15/2017 12:38 AM, Reg Tiangha wrote:
> I guess I was wrong. The Anbox installer tries to install a 4.4 kernel
> and then fails compiling the Anbox dkms module due to the lack of kernel
> sources (I'm running a Qubes dom0 vm kernel).
>
> So I guess eventually, figuring out how to get a valid Grub
> configuration working on this thing to boot a local Ubuntu kernel needs
> to be done. It's weird that it doesn't work the same way as on my Debian
> templates, though. My last resort is copying a valid grub.cfg from one
> of my Debian templates over and manually editing it to point to use the
> xenial kernel and initramfs, but I don't want to have to do that unless
> absolutely necessary. But it's getting late in my part of the world, so
> I'm calling it for now. If anyone else manages to figure this stuff out
> in the meantime, please do share.
>
>

Well, I couldn't sleep while I still had one more thing to try. So I
copied over a working grub.cfg from one of my Debian 8 coldkernel VMs
and edited it to use the Xeial kernel and initramfs (I installed the
linux-generic-hwe-16.04 version which is 4.8). The VM then booted.

I was following the instructions at the bottom here:

https://www.qubes-os.org/doc/managing-vm-kernel/

which is essentially what you need to do to boot a coldkernel in a
Debian template, but for some reason, it doesn't seem to work in Xenial
because running update-grub2 does not create a proper grub.cfg file.

So if someone else can figure out how to get grub on Xenial to produce a
proper grub.cfg file, then that'll make life much easier for everyone.
That should be Step 1 in figuring out how to get this stuff to work in a
Xenial Template VM.
Anyway, I had to regenerate all the dkms modules including Anbox's
(easiest way was to just force reinstall qubes-kernel-vm-support) and
then re-run the Anbox installer and then it installed successfully.

That said, I don't know exactly how to use this. The YouTube video shows
that you can access the manager by clicking on the icon in the Unity
dash, but obviously, we don't have access to that. I was able to install
an apk by starting the session-manager by typing in:

anbox session-manager

and then in a different window, installing an apk by running:

adb install <program.apk>

But I don't know how to access it. So I'm stuck again, but progress has
been made and the program does run in the VM.


What we need to figure out is:

- How to get pvgrub working properly without having to do my kludge of
copying over a grub.cfg file and manually editing it to work with a
Xenial kernel. That'll make Anbox installation much easier later on.

- How to actually use this program without being able to access the
Unity dashboard.


The other easy answer is to use a Xenial HVM, but I can't see why this
can't work in a normal TemplateVM.

Anyway, I think I've taken this as far as I can on my own. I now hand it
off to others to figure out. Hopefully someone can by the time I wake up
in the morning.

[Vít Šesták]

I've tried the Xenial HVM way. Maybe I should perform a clean installation of Xenial, because the VM is partially broken from previous experiments. (I have to boot to recovery mode, remount / as rw and continue booting.) Nevertheless, I got to a similar point: The Android is accessible from adb and anbox commands, but I cannot always access its GUI.

More importantly, I can give you a hint to try: AnBox generates normal *.destop files, just in slightly unusual location ~/snap/anbox/common/app-data/applications/anbox. These files seem to be generated on boot, you might need a while to see them. When you cat those files, you get a command to run. I have tried to do so, with varying results:

a. Timeout or similar error message.
b. Gallery app is runnung, but just a black window. (Nevertheless, it has shown timeout before showing the window.)
c. Segmentation fault (core dumped)

Maybe it is caused by my partially broken Xenial installation, maybe it is some Anbox bug (which cannot be surprising in such early stage) and maybe Anbox is not satisfied with llvmpipe (which would be unfortunate for Qubes users in general).

Regards,
Vít Šesták 'v6ak'

[Reg Tiangha]

Thanks for the hint! I can now launch things on the command line.

I haven't tried all of the programs, but I was able to launch the
program I installed (Obenkyo, a Japanese learning app that still works
with Gingerbread so I know it will usual work with most non-standard
Android environments too) and the Calendar app. I launched the Gallery3D
app and got a black screen too. I wonder if the '3D' part implies that
it needs OpenGL and thus, may have an issue under Qubes? Or maybe a
missing dependency will fix things. The Chromium webview app just
crashes. And I'm still not sure where to find the Application Manager
command that launches the program you see on their YouTube video. It's
too bad their website has no documentation on how to actually use this
program (at least, that I could find); I guess they assume that it
should be obvious if you can access the Unity dashboard (which it
probably is).

I don't know if this helps or if you tried it (I think you might have
though, which means this is for anyone else who wants to try this), but
I couldn't get adb or anything else to work until I ran 'anbox
session-manager' on the command line and had it running in the
background. If I didn't, adb (which I had to install separately; apt-get
install adb) would say it couldn't find a device to connect to.

...and the Anbox session-manager just segmentation faulted on me.

Well, at least there's some more progress. It sounds like that I'm
having a similar experience to you on my regular PV TemplateVM as you
are on your HVM, though. So at least the behaviour seems to be
consistent, and fixing things on one will probably fix things on the
other too.

[Vít Šesták]

Yes, other apps are working after reboot. Maybe I broke it with pm.

* `pm` command in Android seems to be broken – it segfaults and it seems that no app can be started then (though other will continue working)
* Calculator app works, Settings app works, E-Mail app seems to work, too. Maybe the Gallery 3D just has some 3D acceleration issues, but other apps are OK.
* adb lolcat and adb shell are your friends, it shows what is happening
* adb shell might be needed before adb lolcat
* don't worry about timeouts – app window can be shown even after timeout
* Somewhat laggy and CPU hog – maybe related to OpenGL.
* Not sure how to install apps. I've tried adb install FDroid.apk and it passed, but it does not seem to be installed.
* The desktop files sometimes disappear.

Maybe we should turn off GPU rendering in developer options. Unfortunately, I cannot enable developer mode in Settings app. It seems we sould have to disable ro.kernel.qemu.gles and qemu.gles in /system/build.prop, but I am not sure if this is easy. The build.prop might be a RO part of the snap. There is a similar file in /data, not sure what we can override there.

I am not sure if Anbox utilizes OpenGL. Maybe some lowlevel approach bypasses the mesa library, which can decide to use llvmpipe.

Regards,
Vít Šesták 'v6ak'

[Vít Šesták]

Theoretically, /data/local.prop allows to override system properties. But I've tried and after Ubuntu reboot, I see the same values (checked by getprop command). No time to investigate it further ATM.

Regards,
Vít Šesták 'v6ak'

[Reg Tiangha]

You're right; everything in the
snap/anbox/common/app-data/applications/anbox directory disappears for
some reason. No idea what to do about that. However, you can still
launch stuff from the command line if you remember the commands from the
.desktop files. When they reappeared again after a VM reboot, I copied
them all to another directory just so I could see the launch commands
when I needed them.

I've installed stuff with FDroid and adb and it does work; .desktop
files do get generated in that directory. But as you said, they
disappear sometimes.

I managed to enable Developer Mode, but clicking on it does nothing.
Clicking on it from the side menu just kicks you back to the main
Settings screen. The Anbox people say this is a slimmed down version of
Android 7.1.1; is it possible that Developer Mode wasn't compiled in?

I installed Build.prop editor from F-Droid and tried to set
ro.kernel.qemu.gles and qemu.gles to 0 using that. I'm not sure if it
did anything, but after a reboot, Gallery 3D doesn't even start. I don't
know how to revert what I changed though to see if it really had an
effect (can't find a way to delete those entries; long pressing does
nothing and I don't know where it saves its data to) but maybe that's a
route to go if it does work. Make sure to back up the VM first, though.

Does this Android image allow for root? If so, it might be possible to
install a text editor that works with root to try and see if editing
/system/build.prop persists across a reboot. But I too am now out of
time again at the moment.

One last idea: The Anbox docs say it's possible to use your own Android
image file. So maybe it's possible to build one that doesn't have the gl
stuff enabled. That could be another thing to try as a last resort,
especially if build.prop is read-only.

[Reg Tiangha]

On 04/16/2017 03:35 AM, Reg Tiangha wrote:
> On 04/16/2017 01:44 AM, Vít Šesták wrote:
>> Yes, other apps are working after reboot. Maybe I broke it with pm.
>>
>> * `pm` command in Android seems to be broken – it segfaults and it seems that no app can be started then (though other will continue working)

>> * Calculator app works, Settings app works, E-Mail app seems to work, too.. Maybe the Gallery 3D just has some 3D acceleration issues, but other apps are OK.

>> * adb lolcat and adb shell are your friends, it shows what is happening
>> * adb shell might be needed before adb lolcat
>> * don't worry about timeouts – app window can be shown even after timeout
>> * Somewhat laggy and CPU hog – maybe related to OpenGL.
>> * Not sure how to install apps. I've tried adb install FDroid.apk and it passed, but it does not seem to be installed.
>> * The desktop files sometimes disappear.
>>
>> Maybe we should turn off GPU rendering in developer options. Unfortunately, I cannot enable developer mode in Settings app. It seems we sould have to disable ro.kernel.qemu.gles and qemu.gles in /system/build.prop, but I am not sure if this is easy. The build.prop might be a RO part of the snap. There is a similar file in /data, not sure what we can override there.
>>
>> I am not sure if Anbox utilizes OpenGL. Maybe some lowlevel approach bypasses the mesa library, which can decide to use llvmpipe.
>>
>> Regards,
>> Vít Šesták 'v6ak'
>>
> You're right; everything in the
> snap/anbox/common/app-data/applications/anbox directory disappears for
> some reason. No idea what to do about that. However, you can still
> launch stuff from the command line if you remember the commands from the

> ..desktop files. When they reappeared again after a VM reboot, I copied

> them all to another directory just so I could see the launch commands
> when I needed them.
>
> I've installed stuff with FDroid and adb and it does work; .desktop
> files do get generated in that directory. But as you said, they
> disappear sometimes.
>
> I managed to enable Developer Mode, but clicking on it does nothing.
> Clicking on it from the side menu just kicks you back to the main
> Settings screen. The Anbox people say this is a slimmed down version of
> Android 7.1.1; is it possible that Developer Mode wasn't compiled in?
>
> I installed Build.prop editor from F-Droid and tried to set
> ro.kernel.qemu.gles and qemu.gles to 0 using that. I'm not sure if it
> did anything, but after a reboot, Gallery 3D doesn't even start. I don't
> know how to revert what I changed though to see if it really had an
> effect (can't find a way to delete those entries; long pressing does
> nothing and I don't know where it saves its data to) but maybe that's a
> route to go if it does work. Make sure to back up the VM first, though.
>
> Does this Android image allow for root? If so, it might be possible to
> install a text editor that works with root to try and see if editing
> /system/build.prop persists across a reboot. But I too am now out of
> time again at the moment.
>
> One last idea: The Anbox docs say it's possible to use your own Android
> image file. So maybe it's possible to build one that doesn't have the gl
> stuff enabled. That could be another thing to try as a last resort,
> especially if build.prop is read-only.
>

One last thing: I tried to install Busybox from F-Droid, and while it
said that /data had about 4.6GB free, /system has 0GB, which implies to
me that /system is either read-only or really is just an image file that
you can't modify.

System:
* device: Anbox
* android: 7.1.1
* architecture: x86_64

Free space:
* /data: 4.6G
* /system: 0

Latest BusyBox:
* version: v1.25.1-meefik
* applets: 334 items
* size: 1665860 bytes
* md5: 146921ec514d4828748a226dbed2fc25

Installed BusyBox:
* not installed

[Vít Šesták]

Yes, /system being read-only is a standard situation on Android. On some devices, you can perform mount -o remount,rw /system to get it RW, but there are some drawbacks on some devices:

* Some devices come with NAND lock. This was (is?) usually the case of HTC devices and it is likely also the case of Anbox – since snaps contain RO images, it seems to be the natural way to implement it.
* Some devices use dm-verity to reject booting of tampered system. This is the case of BlackBerry, but more vendors are likely to go this way, because this isn't specific for BlackBerry. But I doubt Anbox goes this way.
* Even if you manage to modify /system and then to boot, you are going to have troubles with updates. You can think of /data and /system in Android as rough equivalents of /rw and / in template-based VMs in Qubes. There are some differences (different update mechanisms and no CoW snapshot in Android), but the basic principles are the same. Moreover, in Android, you usually exchange one vendor-provided /system for another vendor-provided /system image (even if you use incremental update), so, unlike template-based VMs, you cannot easily customize it this way.

If you don't want to touch /system, you can go several ways:

* mount --bind (and manage its content to be up-to-date)
* mount -t tmpfs and copy old content (I probably have some script for that)
* modify / – this is a ramdisk you can write to after performing mount -o remount,rw / and there is even some directory on $PATH.

In all those cases, your modification gets lost after reboot. But you can write some script like adb wait-for-device && adb shell su -c /data/busybox/install. You will probably want to run this script as user in order not to have troubles with permissions when using adb later.

Specifically for busybox, its installation consists of just two steps:

1. Copy it to some directory on $PATH.
2. Install symlinks (IIRC by the following command: busybox --install /directory/to/install)

Regards,
Vít Šesták 'v6ak'

[Vít Šesták]

If you want to modify the android.img, look at /snap/anbox/current/android.img. This file is read-only. The dilesystem is squashfs, so it is not designed to be mutable. So, you'll probably have to unsquashfs it, modify it and mksquashfs it. Then, you'll have to pass the img to anbox somehow.

If you want to automatize it (in order to handle updates well), I suggest rebuilding when symlink /snap/anbox/current changes. This should imply update.

Or maybe we can try to ask the author to include some changes or options – maybe some modifications will be also appreciated by some other users.

Regards,
Vít Šesták 'v6ak'

[Reg Tiangha]

If that's the case, and assuming that someone knows what would need to
be changed/disabled in order to get Android to run properly without gl
support on Qubes, it would almost be easier to build a custom
Android-on-Qubes image file that everyone could download from a
repository and use with Anbox.

It would take 40GB of disk space, downloading 10GB+ of source code, and
a lot of time to build, though. But it would only have to be done
every-so-often. Bonus points if it could be somehow made persistent to
take updates, or at the very least, some persistence in order to keep
installed apps around in between sessions.

I did try to compile my own Android-x86 image a few weeks ago, and I
even managed to get it to install, but I encountered the exact same
mouse sync issues that everyone else did, and it no longer worked after
the first reboot. I don't know what makes Anbox's image different, but I
wonder if it would be possible to replace their .img file with one from
Android-x86 or even Remix OS.

[Vít Šesták]

Well, maybe we don't have to recompile it, just modifying build.prop might be enough.

Anbox has some potential advantages over VM with Android:

* Development: Not sure how you would connect adb to some Android VM. Maybe over network, but I am not sure.
* Integration: Anbox support multiple windows that can be resized etc. This fits better into Qubes environment. Also file transfers are easier (you probably could bind mount ~/QubesIncoming to Android's /sdcard) and clipboard integration might be better. (Not sure if clipboard integration is currently implemented in Anbox.) In short, with Anbox, you can utilize many Qubes tools without having to port them for Android.

Of course, Anbox is not mature yet.

On the issue with mouse: I have seen recently some solution. The solution lies in modification of some Xen conf file to use a different mouse emulation for the particular VM.

Regards,
Vít Šesták 'v6ak'