[PATCH] Add support for rootless podman with userns keep-id

[Maximilian Schmidt]

Running `./kas-container` as rootless podman container fails with:

```
groupadd: Permission denied.
groupadd: cannot lock /etc/group; try again later.
useradd: Permission denied.
useradd: cannot lock /etc/passwd; try again later.
```

This patch add a check whether the correct USER_ID is already in use.

Co-authored-by: Nicolas Riebesel <nicolas....@gmx.com>
Signed-off-by: Maximilian Schmidt <maxim...@schmidt.so>
---
container-entrypoint | 2 ++
1 file changed, 2 insertions(+)

diff --git a/container-entrypoint b/container-entrypoint
index 8a41f02..a71c406 100755
--- a/container-entrypoint
+++ b/container-entrypoint
@@ -19,6 +19,8 @@ if [ $USER_ID == 0 ]; then
mkdir -p /builder

GOSU=""
+elif [ $USER_ID == $UID ]; then
+ GOSU=""
else
if ! grep -q "^builder:" /etc/group; then
groupadd -o --gid $GROUP_ID builder
--
2.30.0

[florian...@siemens.com]

Hi Maximilian,

On Wed, 2021-01-20 at 00:05 +0100, Maximilian Schmidt wrote:
> Running `./kas-container` as rootless podman container fails with:
>
> ```
> groupadd: Permission denied.
> groupadd: cannot lock /etc/group; try again later.
> useradd: Permission denied.
> useradd: cannot lock /etc/passwd; try again later.
> ```
>
> This patch add a check whether the correct USER_ID is already in use.
>
> Co-authored-by: Nicolas Riebesel <nicolas....@gmx.com>
> Signed-off-by: Maximilian Schmidt <maxim...@schmidt.so>
> ---
>  container-entrypoint | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/container-entrypoint b/container-entrypoint
> index 8a41f02..a71c406 100755
> --- a/container-entrypoint
> +++ b/container-entrypoint
> @@ -19,6 +19,8 @@ if [ $USER_ID == 0 ]; then
>   mkdir -p /builder
>    GOSU=""
> +elif [ $USER_ID == $UID ]; then
> + GOSU=""
>  else

IIRC that should not be necessary inside the container. If kas-
container detects podman as runtime engine we use something like
--userns=keep-id to keep the user-id mapping 1:1.

The container engine detection may fail if you have a "docker" command
in your PATH (that might be symlinked to podman). Is that the case on
your system? You would end up using podman with the docker cmdline
arguments.

[Jan Kiszka]

I also face problems with unprivileged invocation here, though not
around group/useradd:

Error: container_linux.go:349: starting container process caused
"process_linux.go:449: container init caused \"rootfs_linux.go:58:
mounting \\\"/some/work-dir\\\" to rootfs
\\\"/home/jan/.local/share/containers/storage/overlay/43f8467829623a5f90c39269265e8a6a4893ef735ff8a019d46127a4ced8b181/merged\\\"
at \\\"/work\\\" caused \\\"stat /some/work-dir: permission
denied\\\"\"": OCI runtime permission denied error

This goes away when dropping --userns=keep-id. The patch has no impact
on this.

I'm afraid the Yocto case has never been tested extensively.

Jan

--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

[florian...@siemens.com]

I did some tests when bringing in initial podman support. I went back
to kas 2.2 but as it turned out it does not work... Maybe podman
changed some defaults again...

Results of my investigations so far: The entry point expects to be run
as root, otherwise the user creation fails. I can reproduce that. That
is exactly what has been reported by Maximilian now. Removing 
--userns=keep-id fixes that, but the build may fail later due to
incorrect permissions somewhere in the output / build directories. That
actually depends on the filesystem / build state. Reason should be a
missing user-id mapping for the "builder" user.

The complete container-entrypoint looks like a trick. Creating a user
inside the container with the same UID and GID as the user that created
the container process on the host. I guess to get a 1:1 mapping
working. 

Maybe we should replace it with a user-id mapping cmdline that works
for docker and podman at the same time. No idea if something like that
exists ;-)

> Jan
>

[Jan Kiszka]

Yes, that is the purpose.

>
> Maybe we should replace it with a user-id mapping cmdline that works
> for docker and podman at the same time. No idea if something like that
> exists ;-)

Keep in mind that this trick has been developed "long" ago (4 years by
now) against Docker. Since then, it worked fine - against Docker. But
then unprivileged Podman came around, and that may mean we need to
address the topic of ID-mapping differently for it.

However, for the Isar case and as long as [1] isn't resolved, we need
root permissions in the entrypoint in order to set up binfmt-misc. That
makes things even more complicated, though some aspects can be addressed
by using a different entrypoint code in kas-isar (what we do already).

Jan

[1]
https://lore.kernel.org/lkml/8eb5498d-89f6-e39e...@vivier.eu/

[maxim...@schmidt.so]

We tested this setup with an up to date podman (version 2.2.1) on arch linux.
The podman detection of `kas-container` works correctly and `--userns=keep-id`
was successfully passed to the podman command.

When `keep-id` is passed to the podman command, the user inside the container
is the same as outside the container, i.e.

- outside $UID = inside $UID
- outside $GID = inside $GID
- also the outside user entry is present inside the container's /etc/passwd

Additionally, in combination with `--workdir=/work` the $HOME is also mapped correctly to /work.

The permissions of `/work` are set to have USER_ID and GROUP_ID as owner and group respectively.
Thus, the container seems to be setup correctly.

This means, we don't see the need to create an additional user via the entrypoint.
Furthermore, this creation does not work nevertheless, as the user inside the container is not root and thus has no permissions to add or modify users.
Note that this only applies to rootless podman usage as the keep-id flag is ignored when running podman as the root user.

--

[Jan Kiszka]

On 21.01.21 19:51, maxim...@schmidt.so wrote:
> We tested this setup with an up to date podman (version 2.2.1) on arch linux.
> The podman detection of `kas-container` works correctly and `--userns=keep-id`
> was successfully passed to the podman command.
>
> When `keep-id` is passed to the podman command, the user inside the container
> is the same as outside the container, i.e.
>
> - outside $UID = inside $UID
> - outside $GID = inside $GID
> - also the outside user entry is present inside the container's /etc/passwd
>
> Additionally, in combination with `--workdir=/work` the $HOME is also mapped correctly to /work.
>
> The permissions of `/work` are set to have USER_ID and GROUP_ID as owner and group respectively.
> Thus, the container seems to be setup correctly.
>
> This means, we don't see the need to create an additional user via the entrypoint.
> Furthermore, this creation does not work nevertheless, as the user inside the container is not root and thus has no permissions to add or modify users.
> Note that this only applies to rootless podman usage as the keep-id flag is ignored when running podman as the root user.
>

But there is still the mounting issue for some setup. I was testing on
OpenSUSE 15.2, podman 2.1.1, possibly distro-specific settings. But
Florian was seeing he same issue on Fedora. I wonder where this comes
from - it's not the entrypoint logic, it's earlier than that.

Jan

[Nicolas Riebesel]

We also needed a few hours to get rootless podman running and also
saw quite similar issues. Your issue looks pretty much like this error:

https://lists.podman.io/archives/list/pod...@lists.podman.io/thread/LA2J5LY6SZMNMPLDGE4DKIV2CFLGPOXC/

It seems to have something to do with the volume that is mounted
not being owned by the user that is running podman. Maybe the
whole subtree needs to be owned by that user. Rootless podman
seems to be a bit finicky. There is also a pretty comprehensive
troubleshooting guide for rootless podman:

https://github.com/containers/podman/blob/master/troubleshooting.md

Our podman problem turned out to be the $HOME directories group
not being the users group. You might also want to check that.

> >> https://lore.kernel.org/lkml/8eb5498d-89f6-e39e-d757-404cc3cfaa5c@viv

[Jan Kiszka]

In my case, it was indeed to issue that one of the mounted dirs had one
level without rx for other. But then I was able to use kas-container
without this patch, ie. with plain kas:2.3.3. Weird.

Anyway, I've applied the patch (with whitespace tweaking) so that we can
test :next if it causes issues for other users. It shouldn't as the
logic makes sense.

Jan