[Feature] Asking for support for termux on android(with termux-glibc)

[柳半仙（fish4terrisa-MSDSM）]

I don't have much knowledge of docker or some other container platforms.However, I do know much about proot, a rootless chroot-like program which is widely used to run linux in termux/android. However, proot is using ptrace and have many strange problems and large performance decrease.I want to use gvisor as a replacement for proot, and act as an rootless chroot.
Termux doesn't spawn a proot shell. Instead, termux run a normal android binary. A good way to understand this is that termux is a linux distro using bionic libc (glibc is also ported to termux) and use /data/data/com.termux/files as the rootfs.
It seems that gvisor re-implemented many linux syscalls and even the network,which can make linux running on termux faces less problems. And it's possible to run gvisor without root.

So is it possible to port gvisor to termux in order to make it possible for termux users to run linux in termux with less performance cost?

Now glibc is ported to termux, so we won't need to consider the difference between bionic libc and glibc.
In my expection, maybe we will have a simple tool based on gvisor(gvisor seems to be a backend of other container platforms), which can act like chroot but don't need root , /dev , /proc and even doesn't need some namespace support.
For example, if that program is called gvisor-chroot, i want to use it as this:

----------------

$ id
uid=10312(u0_a312) gid=10312(u0_a312) groups=10312(u0_a312),3003(inet),9997(everybody),20312(u0_a312_cache),50312(all_a312)
$ gvisor-chroot --rootless /data/data/com.termux/files/home/archlinux-rootfs --bind /sdcard:/media/sd --uid 0 --gid 0
[root@localhost]# id
uid=0(root) gid=0(root) group=0(root)
[root@localhost]# cat /etc/os-release
NAME="Arch Linux"                                                                                
PRETTY_NAME="Arch Linux"                                                                          
ID=arch                                                                                          
BUILD_ID=rolling                                                                                  
ANSI_COLOR="38;2;23;147;209"                                                                      
HOME_URL="https://archlinux.org/"                                                                
DOCUMENTATION_URL="https://wiki.archlinux.org/"  
SUPPORT_URL="https://bbs.archlinux.org/"                                                          BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues"                          
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

----------------

The reason why I don't use docker is that although docker itself is ported to termux, it will only work with root(even in its rootless mode, as some namespace feature is missing for rootless users in termux),and docker is something too heavy for a termux user who only wants to run linux distro in termux.
(This program may also be used as a test suite or a way for programmers to experience gvisor's features.)

(P.S. I have already made a issue in the gvisor github repo, and they told me that I can also make a feature request at this mailing list. Here is the issue:https://github.com/google/gvisor/issues/10640)

[Etienne Perot]

Hello again!

What you're describing seems feasible to me, except for one part I'll get to
later on (so please read to the end before going too deep into testing it).

The way Docker and other container tools like Podman work is through a lower-
level component called the "OCI runtime interface". When you start a container
using Docker or Podman or any other such tool, what is happening behind the
scenes is that the tool will generate an "OCI runtime configuration", which is
a JSON configuration describing the workload that needs to be executed
(command-line, environment, user, permissions, mount volumes, etc.), and which
also points to a rootfs directory where the root filesystem of the container is
present.

gVisor provides a binary called "runsc" which implements this OCI runtime
interface. It accepts this JSON config format and can execute the workload
inside a gVisor sandbox, using the provided rootfs directory as root
filesystem. So it seems like it would work for your use-case, so long as you
can create this OCI runtime config.

To get started on this, the best way is probably to first get an OCI runtime
configuration template that works on a Linux desktop or VM but that you can
later try inside termux on an Android device. Running this in regular non-
Android Linux successfully first will make it easy to determine whether any
problems that come up are termux/Android-specific, as opposed to problems with
the OCI configuration itself.

You can start by creating the following directory structure:

/some/path
/some/path/gvisor-testing
/some/path/gvisor-testing/state
/some/path/gvisor-testing/bundle
/some/path/gvisor-testing/bundle/rootfs

Put your rootfs files under "/some/path/gvisor-testing/bundle/rootfs" (for
example, you should have an empty directory at "/some/path/gvisor-testing/
bundle/rootfs/proc", and you should have a whole /usr-like tree under "/some/
path/gvisor-testing/bundle/usr".

Once you have this, run this:

$ runsc spec --bundle=/some/path/gvisor-testing/bundle -- /bin/echo 'Hello
world'

(Notice that this is pointing to the directory that contains 'rootfs', not to
'rootfs' itself).
This will create a file at /some/path/gvisor-testing/bundle/config.json which is
an OCI runtime config JSON. You can edit this as you wish. Notice that the
command (in this case `/bin/echo 'Hello world'`) is inside the `process.args`
part of the config, and the rootfs is under `root.path`. You can replace this
with an absolute path to the rootfs directory if you want. You could add
something like the `/sdcard` volume inside the `mounts` array, but I would
recommend not doing this before you can confirm that a simple "Hello world"
command works on Android.

Once you have this config set up, you should be able to run:

$ sudo runsc --root=/some/path/gvisor-testing/state run --bundle=/some/path/
gvisor-testing/bundle hello-world

And if all works well you should see "Hello world".
If it does not work, try adding debug logging to see what's wrong:

$ sudo runsc --root=/some/path/gvisor-testing/state --debug-log=/dev/stderr
run --bundle=/some/path/gvisor-testing/bundle hello-world

If running without `sudo`, you may need to add `--rootless=true`, and probably
`--network=none` or `--network=host`. Depending on your host configuration, you
may also need `--ignore-cgroups=true`. These flags need to go before the `run`
positional argument to `runsc`.

If any of the above fails (I typed the above commands by hand without testing
them so it's likely I missed some flag or something), try following the gVisor
OCI tutorial:
https://gvisor.dev/docs/user_guide/quick_start/oci/ which also walks through
creating a valid rootfs. And if that tutorial still doesn't work, please open
a bug about it.

Once you have a working OCI bundle (config.json + rootfs) on your Linux
desktop, then you can copy that directory structure over to your Android
device, and try to run it there.

What I expect you will run into at this point is that some Android kernels
lack essential namespace features that container runtimes require. This
discussion suggest that it may be a problem:
https://github.com/containers/podman/discussions/17717

These kernel features are not strictly essential for gVisor to function as a
userspace kernel, but they are used by gVisor as second-layer defenses in
order to strengthen its security. You can read more about this at
https://gvisor.dev/docs/
So while it is technically feasible to remove these extra security measures
from gVisor's code (so that it would run without needing namespaces support
from the Android kernel), doing so would weaken its security for gVisor's
intended primary use-case of being a _secure_ container runtime. Therefore,
this cannot be its main mode of operation, and there would need to be a good
reason for supporting such an alternative mode.

Anyway, if you do want to give it a shot, check if the above works or if you
run into the namespace support issue. If it works, then I think you now have
all the pieces necessary to build such a `gvisor-chroot` tool.

Cheers,
- E.

[柳半仙]

Can I disable this second-layer defenses manually?(by passing cli options?)

And, I've built runsc on android successfully, even with bionic libc, but it returns :

panic: couldn't open /proc/sys/vm/mmap_min_addr: open /proc/sys/vm/mmap_min_addr: permission denied                                                                                                

goroutine 1 [running]: gvisor.dev/gvisor/pkg/sentry/platform.init.0()

        /data/data/com.termux/files/home/go/pkg/mod/gvisor.dev/gvisor...@v0.0.0-20240710184332-33cc259d1ec7/pkg/sentry/platform/mmap_min_addr.go:52 +0x170

Is gvisor strictly depending on /proc/sys/vm/mmap_min_addr or any other files under /proc? If so, then which of them?

[Etienne Perot]

Thanks for trying it out.

It's odd that the Android kernel would have /proc/sys/vm/mmap_min_addr
as unreadable... On my Linux machine it's world-readable. Anyway, you
can probably try to stub it out by editing
pkg/sentry/platform/mmap_min_addr.go by setting `systemMMapMinAddr` in
the `init` function. The default value for this is 65536. So try
replacing the `init` function with the below, and then see if there's
another point where `runsc` gets stuck during initialization on Android.

func init() {
    systemMMapMinAddr = 65536
}

As for the namespaces question: That's the difficult part. By default,
it is not OK for gVisor to tolerate running without namespace support,
because gVisor's primary use-case is to provide an environment that is
as secure as possible. If it can't guarantee that, it's better for it to
complain loudly and force the user to acknowledge and explicitly
configure it to allow for lower security, rather than for gVisor to
silently degrade its security.

There is however no flag that currently allows runsc to run without
namespace support. That would have to be added, and non-default, hence
my earlier comment about there needing to be a good reason to support
such a mode. I don't think supporting Android users using termux would
by itself be sufficient (unless a lot of such users would ask). Until
then, it would probably instead need to be packaged as a patch on the
termux side. There seems to be a lot of precedents for this; for example
here's an existing patch for `runc` in termux:
https://github.com/termux/termux-packages/blob/master/root-packages/runc/cg-cpuset-noprefix-compat.patch

Anyway, I wouldn't give up just yet... Termux does seem to have packages
for software which uses Linux namespaces as well, such as Bubblewrap
(https://github.com/termux/termux-packages/tree/master/root-packages/bubblewrap),
runc
(https://github.com/termux/termux-packages/tree/master/root-packages/runc),
and Docker
(https://github.com/termux/termux-packages/tree/master/root-packages/docker).
If these programs work, then it means either namespaces are working on
the kernel, or they are ignored by the kernel, and either way that would
allow gVisor to run (though less securely). So you could give either of
those termux packages a try, and if they work, then chances are gVisor's
lack of "run without namespace support" mode is probably not going to be
a problem.

- Etienne

[柳半仙]

Thank you for the help on dealing with /proc/sys/vm/mmap_min_addr.That really helped. Now I have already generated the oci config. 

For the namespaces question, I agree with your option. And, in many situations the security rules made by the users may conflict with gvisor's, so it's worth to allow the users to choose.

For the docker, runc and many others, they are only for rooted devices, but many termux users' devices are unrooted. Android's security rules are strict and it didn't trust most binaries, including gvisor(Also, in many situations, the users may don't have rights to change the security rules), so namespace support is banned for ordinary users.

I'm currently dealing with the sandbox in ./runsc/container/container.go. Do you know how to force runsc to skip creating sandbox?

[Etienne Perot]

You are looking in the right place. You need to look at the places where
`runsc` creates processes as part of the sandbox startup sequence, which
are the `createGoferProcess` function in `container.go` and the
`createSandboxProcess` function in `sandbox.go`. Go through their code,
you will find that they all eventually create an `exec.Cmd` object (that
is the Go struct for managing subprocesses). What determines the
namespaces that a subprocess runs in is the `SysProcAttr.Cloneflags`
that is set within it. The other fields like `SysProcAttr.UidMappings`
and `SysProcAttr.GidMappings` are also meaningless when not creating a
new namespace, so they would need to be unset as well. There may be
other places in the code that create subprocesses but I am not sure
where they are, and they probably don't create namespaces.

Hope this helps,
- E

[柳半仙]

I have patched them and patched some other places. The repo is here:

https://github.com/fish4terrisa-MSDSM/gvisor-termux-for-test

(Very dirty hack, just to figure out whether gvisor can be ran in termux)

However, i'm faced with this problem:

running container: starting container: starting root container: starting sandbox: failed to setupFS: creating mount namespace: creating root file system: invalid argument

That's wield, as it seems to be related with mounting the rootfs inside gvisor. I enclosed the log. Can you find out what caused this?

My cmdline: ./bin/runsc -TESTONLY-unsafe-nonroot --rootless=true --network=host --ignore-cgroups=true --root=/data/data/com.termux/files/home/gvisor-testing/state --debug-log=/sdcard/runsc.log --debug run --bundle=/data/data/com.termux/files/home/gvisor-testing/bundle hello-world

The rootfs is at here: https://people.linaro.org/~loic.poulain/micro-rootfs/micro-rootfs-aarch64.tar.gz

I also enclosed my config.json(oci config) 

The runsc binary built for termux is also in the repo, maybe you can check with it?

(Also, is hard to make bazel working on termux due to java problems, so I used the go branch)

在 2024年7月12日星期五，Etienne Perot <eti...@perot.me> 写道 ：

[Etienne Perot]

Hello,

Thanks for uploading the code to a repo and providing logs + config.
Unfortunately I'm not very familiar with this part of the sandbox setup
code, but from the error message, it appears to be coming from the
initialization of a root tmpfs mount.
I suggest adding additional logging in
`pkg/sentry/fsimpl/tmpfs/tmpfs.go`'s `GetFilesystem` function to see
where exactly the error is coming from, and print more information than
just "invalid argument" (`EINVAL`).
Also, perhaps I followed the code incorrectly... Either way, add more
logging to try to find where the `EINVAL` is coming from and get more
information from it, as by itself this log is not enough information to
understand what's going on.

Alternatively, you may consider using EROFS as root filesystem. This was
recently added to gVisor, and it changes the way the rootfs is set up in
such a way that (perhaps, I haven't actually followed the code all the
way through) could avoid creating a new mount namespace or running a
Gofer process at all. There is unfortunately no good documentation on
how to use it, but there is a test in
`runsc/container/container_test.go` called `TestRootfsEROFS` that
exercises this functionality. It involves creating an EROFS root
filesystem image using `mkfs.erofs`, and you can see that it uses some
special annotations in the OCI runtime config to tell gVisor about the
EROFS image. You can see how it modifies the spec in the test code, such as:

```
spec.Annotations[boot.RootfsPrefix+"type"] = erofs.Name
spec.Annotations[boot.RootfsPrefix+"source"] = rootfsImage
spec.Annotations[boot.RootfsPrefix+"overlay"] = config.NoOverlay.String()
```

To figure out what that expands to, you can add some logging in the test
to make it log the `spec` after it is built (but before actually
creating the sandbox with it).
You can run the test with `make test
TARGETS=runsc/container:container_test` from the master branch (not Go
branch).
HTH,
- E

(P.S. I will be away for the next few weeks.)

[Ayush Ranjan]

The logs show `W0712 08:00:10.837172   10435 gofer.go:618] Mount RPC did not return host FD to mount point with directfs enabled`.

So the gofer process is not donating a file-descriptor to the rootfs directory on the host. You might want to see why this code in the gofer is not executing.

--
You received this message because you are subscribed to the Google Groups "gVisor Users [Public]" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gvisor-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gvisor-users/a498f656-85e8-4ec8-8127-3dfb14212332%40perot.me.

[柳半仙（fish4terrisa-MSDSM）]

It seems that the code mentioned in gofer is actually running, the 
unix.Dup(rootHostFD) runs with no error:

I added the line

log.Warningf("Debug: clientHostFD: %d , rootHostFD: %d", clientHostFD, rootHostFD)

after the line 

clientHostFD, err = unix.Dup(rootHostFD)

, and it prints 

Debug: clientHostFD: 17 , rootHostFD: 16

And I already tested that the err is equal to nil, so these code simply run well

Except the line cu.Release()

I haven't found a way to debug this line(cu.Release()) Maybe I'll try later.

[柳半仙（fish4terrisa-MSDSM）]

It seems that the c.SndRcvMessage in pkg/lisafs/client.go , at the line of 95, didn't return the mountHostFD[0] properly. That's strange, as the gofer got the rootHostFD successfully before...

[Etienne Perot]

If you can't get the Gofer to work, I'd recommend looking into what I
said earlier about providing the rootfs as an EROFS image. There is no
Gofer process running when using this mode.

[柳半仙]

I finally found the reason of gofer failing:

It's because that in the function `serve` in file runsc/cmd/gofer.go, it assumes that `fsgofer process is always chroot()ed`(but I forced it to not), so it always tries to mount the `/` for the guest. But Android users have no permissions for the `/`

After change the rootfs with a dirty hack, it can mount the rootfs , /sys , /proc and /dev successfully now.

However, now I'm faced with a new problem:

running container: starting container: starting root container: starting sandbox: creating process: failed to load /bin/echo: permission denied

Does anyone know how to fix it?

I have already uploaded all the codes in the same repo as above. The log is also contained in the attachment.

[柳半仙（fish4terrisa-MSDSM）]

Maybe I should note that the termux user have full right of the rootfs, so it seems not to be the permission problem of the binary on the host

[Etienne Perot]

You may have full read permission, but do you also have execute (+x)
permission on `/bin/echo`? If you copied the rootfs from somewhere
without copying permissions along with that (`cp -a`), it may end up as
a read/writable file without execute permissions.

You may also have to check if you have access to all shared libraries
used by `/bin/echo` (run `ldd /bin/echo` and see if the permissions are
OK too) or any interpreter it uses if it's a script.

If this isn't the problem, then add some logging inside `loadExecutable`
in `pkg/sentry/loader/loader.go` to get more details.

HTH,
- Etienne