[PATCH] kas-container: add tmpfs option for podman

[Trevor Woerner]

Trying to build a simple nodistro qemux86 core-image-base image with
kas-container using podman on any branch after zeus fails.

build host: openSUSE 15.3
container: podman (specifically 2.1.1 on openSUSE 15.3)
branches: dunfell, gatesgarth, hardknott, honister, master
config.yml:
header:
version: 11
machine: qemux86
distro: nodistro
target:
- core-image-base
repos:
bitbake:
path: layers/bitbake
layers:
conf: disabled
openembedded-core:
path: layers/openembedded-core
layers:
meta:

Although I'm using openSUSE 15.3 and podman 2.1.1 specifically, others have
reported similar issues with other hosts and versions of podman. Every such
build fails at some point due to pseudo erroring out. Builds on warrior and
zeus with the exact same setup succeed. The underlying reasons why this patch
fixes pseudo issues post-zeus are not known.

This patch's trick of adding "--tmpfs /tmp" to podman's arguments was
suggested by Quentin Schulz.

Signed-off-by: Quentin Schulz <foss+...@0leil.net>
Signed-off-by: Trevor Woerner <twoe...@gmail.com>
---
kas-container | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kas-container b/kas-container
index c7b7b34..fe1bc36 100755
--- a/kas-container
+++ b/kas-container
@@ -125,7 +125,7 @@ docker)
;;
podman)
KAS_CONTAINER_COMMAND="podman"
- KAS_RUNTIME_ARGS="--userns=keep-id --security-opt label=disable"
+ KAS_RUNTIME_ARGS="--tmpfs /tmp --userns=keep-id --security-opt label=disable"
;;
*)
echo "$0: unknown container engine '${KAS_CONTAINER_ENGINE}'" >&2
--
2.34.1.75.gabe6bb3905

[Jan Kiszka]

Thanks for the patch. As mentioned in the other thread:

Rootless podman uses fuse-overlayfs for the rootfs, and that seems to be
incomplete or otherwise incompatible with pseudo. When starting in
--isar mode, thus privileged, a real overlayfs is used, and that works.
And the build folder is not affected by this as it is mounted from the host.

Would be good to correlate that in the commit log. Bonus for asking the
pseudo community if they are aware of such a clash and the underlying
reasons.

> This patch's trick of adding "--tmpfs /tmp" to podman's arguments was
> suggested by Quentin Schulz.
>
> Signed-off-by: Quentin Schulz <foss+...@0leil.net>
> Signed-off-by: Trevor Woerner <twoe...@gmail.com>
> ---
> kas-container | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/kas-container b/kas-container
> index c7b7b34..fe1bc36 100755
> --- a/kas-container
> +++ b/kas-container
> @@ -125,7 +125,7 @@ docker)
> ;;
> podman)
> KAS_CONTAINER_COMMAND="podman"
> - KAS_RUNTIME_ARGS="--userns=keep-id --security-opt label=disable"
> + KAS_RUNTIME_ARGS="--tmpfs /tmp --userns=keep-id --security-opt label=disable"

Please keep 80 chars as limit, thus wrap this around.

> ;;
> *)
> echo "$0: unknown container engine '${KAS_CONTAINER_ENGINE}'" >&2

Florian, as podman user, can you confirm this issue as well?

Thanks,
Jan

--
Siemens AG, Technology
Competence Center Embedded Linux

[Bezdeka, Florian]

On Mon, 2022-01-10 at 15:27 +0100, Jan Kiszka wrote:
> On 10.01.22 14:50, Trevor Woerner wrote:
> > Trying to build a simple nodistro qemux86 core-image-base image with
> > kas-container using podman on any branch after zeus fails.
> >
> > build host: openSUSE 15.3
> > container: podman (specifically 2.1.1 on openSUSE 15.3)
> > branches: dunfell, gatesgarth, hardknott, honister, master
> > config.yml:
> > header:
> > version: 11
> > machine: qemux86
> > distro: nodistro
> > target:
> > - core-image-base
> > repos:
> > bitbake:
> > path: layers/bitbake
> > layers:
> > conf: disabled
> > openembedded-core:
> > path: layers/openembedded-core
> > layers:
> > meta:

Hi!
Where do I have to place this file for being able to reproduce?

Hm. Any (default) limits involved?

- What is actually being written to /tmp? 
- What happens to the host if /tmp is heavily used? OOM?

>
> > ;;
> > *)
> > echo "$0: unknown container engine '${KAS_CONTAINER_ENGINE}'" >&2
>
> Florian, as podman user, can you confirm this issue as well?

Nope, not yet. Is there any build log available showing concrete error
messages?

We had some fun in the recent past when glibc (running inside the
container) started using syscalls not known to the syscall filtering
machinery on the host (libseccomp). Are there different glibc versions
involved?

>
> Thanks,
> Jan
>

[Jan Kiszka]

$ mkdir testing-kas
$ cd testing-kas
$ cat > kas.yml <<EOF

header:
version: 11

machine: qemux86
distro: nodistro

target: core-image-base

repos:
bitbake:
url: git://git.openembedded.org/bitbake.git
refspec: master
layers:
conf: disabled

openembedded-core:
url: git://git.openembedded.org/openembedded-core.git
refspec: master
layers:
meta:
EOF
$ kas-container build kas.yml

This is what I did to reproduce.

Undocumented preference of the guy merging things, yes.

>
> - What is actually being written to /tmp?

No idea what all, but everything some build process in the container
decides to use /tmp for, rather than some workdir.

> - What happens to the host if /tmp is heavily used? OOM?

OOD (out-of-disk), then also on the host. Containers tend to have default limits on their overlays. BUT: We already map the build folder in, and that is generally on the host filesystem as well and carries much more data.

>
>>
>>> ;;
>>> *)
>>> echo "$0: unknown container engine '${KAS_CONTAINER_ENGINE}'" >&2
>>
>> Florian, as podman user, can you confirm this issue as well?
>
> Nope, not yet. Is there any build log available showing concrete error
> messages?

Some example:

ERROR: Logfile of failure stored in: /build/tmp-glibc/work/core2-32-oe-linux/glibc-locale/2.34-r0/temp/log.do_package_write_ipk.445
ERROR: Task (/work/openembedded-core/meta/recipes-core/glibc/glibc-locale_2.34.bb:do_package_write_ipk) failed with exit code '1'
Pseudo log:
path mismatch [1 link]: ino 33756398 db '/tmp/sh-thd.OrwpmG' req '/tmp/sh-thd.gJsVnF'.
path mismatch [1 link]: ino 33756397 db '/tmp/sh-thd.Y8ogzA' req '/tmp/sh-thd.fVzjsz'.
path mismatch [1 link]: ino 33756396 db '/tmp/sh-thd.X5N4vv' req '/tmp/sh-thd.UWbAAD'.
path mismatch [1 link]: ino 33756395 db '/tmp/sh-thd.i1F8Tv' req '/tmp/sh-thd.4oyP3z'.
path mismatch [1 link]: ino 33756394 db '/tmp/sh-thd.A2s3Sw' req '/tmp/sh-thd.v5F79v'.
path mismatch [1 link]: ino 33756393 db '/tmp/sh-thd.t3YSau' req '/tmp/sh-thd.LcgLDw'.
path mismatch [1 link]: ino 33756392 db '/tmp/sh-thd.k8agFq' req '/tmp/sh-thd.5MiKZu'.
path mismatch [1 link]: ino 33756389 db '/tmp/sh-thd.r0o79q' req '/tmp/sh-thd.cOslPy'.
path mismatch [1 link]: ino 33756388 db '/tmp/sh-thd.q5twCp' req '/tmp/sh-thd.78ocxs'.
Setup complete, sending SIGUSR1 to pid 449.

>
> We had some fun in the recent past when glibc (running inside the
> container) started using syscalls not known to the syscall filtering
> machinery on the host (libseccomp). Are there different glibc versions
> involved?
>

No idea, but adding --isar to kas-container resolves it, and that speaks
a bit against this explanation.

[Bezdeka, Florian]

Confirmed. Happens here as well but:

That seems to be introduced with
https://git.openembedded.org/openembedded-core/commit/?id=2db491d97da08d44ebd257f98489550a82a7935c

and there is a Wiki page at
https://wiki.yoctoproject.org/wiki/Pseudo_Abort

So there must be something special within recent OE layers. The wiki
page is talking about "clean tmp directories". Maybe there is a better
way of getting a clean tmp dir instead of mounting a tmpfs into the
container?

[Jan Kiszka]

That commit talks about checking inode numbers - I rather suspect that
fuse-overlayfs is not presenting consistent inode emulation, and that
makes the check fail.

[Trevor Woerner]

Richard Purdie had noticed that, on occasion, there would be discrepancies
in the filename-to-inode mappings in the psuedo database and that, in the
cases he was investigating, these would arise from processes outside the build
system touching/modifying the files under pseudo's control. He therefore added
this check and error path to be notified of and catch these issues.

It appears the same thing is happening here, though not because something
outside the build is touching these files but simply as a consequence of how
podman (or fuse-overlayfs?) handles inode numbers/mappings in unprivileged
containers.

I'm still digging into podman/fuse/overlayfs to try to find exact proof of a
discrepancy between how inodes are handled in privileged vs unprivileged
containers. Once I find that, then I can write a better commit message for
this patch.

[Jan Kiszka]

Any news on this?

[Trevor Woerner]

I'm still working on this (albeit slowly).

I was able to have a chat with a podman developer who confirmed that this
behaviour was a result of using an unprivileged container, which causes
fuse-overlayfs to be used (which is what everyone suspected) and
fuse-overlayfs is where the issue is happening.

The fuse(8) man page has the following interesting tidbit:

noforget

remember=T
Normally, libfuse assigns inodes to paths only for as long as the kernel is aware of
them. With this option inodes are instead assigned for at least T seconds (or, in the
case of noforget, the life-time of the filesystem). This will require more memory,
but may be necessary when using applications that make use of inode numbers.

It looks like fuse-overlayfs's has a FORGET command that is part of its
user←→kernel protocol that, by default, will drop inodes after a period of
(what it thinks is) inactivity. If my understanding is correct, then this is
very likely what is upsetting pseudo.

The podman developer was hoping that I could set the "noforget" option in a
mountopt setting in a storage.conf file; I tried a couple ways of doing this,
but none had any noticeable effect. However, I'm on openSUSE 15.3, and
although this is openSUSE's latest release, it does bundle a rather old
version of podman (2.1.1) so it's possible that with a new version of podman
I might be able to actually fix this. Upgrading podman is a bit… tricky it
seems.

The podman developer also indicated that privileged vs non-privileged
containers are handled differently (i.e. overlay-via-the-kernel vs
fuse-overlayfs) but that with newer kernels they won't be. I.e. with a new
kernel all containers will be handled with the overlay-via-the-kernel method,
avoiding fuse-overlayfs, and, thus, avoiding this issue. So hopefully this
issue will go away in time.

My --tmpfs /tmp solution is a complete band-aid. Non-privileged containers use
fuse-overlayfs for filesystems, which, by default, will FORGET inodes after a
period of time. Adding --tmpfs /tmp to the podman call simply sidesteps the
use of fuse-overlayfs for the /tmp drive. By coincidence, /tmp is where files
are being placed and if those inodes change pseudo notices and throws a fault.
It's a "fix" in the loose sense of the word :-) Better solutions would be to
wait for the host's kernel to get updated, or to add the "noforget" option,
somehow, to fuse-overlayfs (via libfuse).

I'm just a bit stuck proving either of these solutions will work.

Thoughts?

[Jan Kiszka]

Is there a way to detect the case when we need you workaround? Then I
would have no problem to apply it conditionally. And, as Florian
confirmed, it's not just an OpenSUSE issue at the moment due to old
kernels and old podman. So we will likely have to deal with it for a while.

[Trevor Woerner]

Yes. I could add a check for non-root user

id -ur

and a check for kernel version

uname -r

and add the "--tmpfs /tmp" option if user is non-zero and kernel < 5.13. Does
that sound good?

https://www.redhat.com/sysadmin/podman-rootless-overlay

[Jan Kiszka]

I suppose we would also have to take the podman version into the
equation: I am on 5.16, and I was still able to trigger this with
OpenSUSE's podman version.

[Florian Bezdeka]

I'm on 5.15.17 (which I would consider "new") and I confirmed this issue
some time ago. Any (exact) information which kernel would make a
difference here?

>>>>
>>>> My --tmpfs /tmp solution is a complete band-aid. Non-privileged
>>>> containers use
>>>> fuse-overlayfs for filesystems, which, by default, will FORGET
>>>> inodes after a
>>>> period of time. Adding --tmpfs /tmp to the podman call simply
>>>> sidesteps the
>>>> use of fuse-overlayfs for the /tmp drive. By coincidence, /tmp is
>>>> where files
>>>> are being placed and if those inodes change pseudo notices and
>>>> throws a fault.
>>>> It's a "fix" in the loose sense of the word :-) Better solutions
>>>> would be to
>>>> wait for the host's kernel to get updated, or to add the "noforget"
>>>> option,
>>>> somehow, to fuse-overlayfs (via libfuse).
>>>>
>>>> I'm just a bit stuck proving either of these solutions will work.
>>>>
>>>> Thoughts?
>>>>
>>>
>>> Is there a way to detect the case when we need you workaround? Then I
>>> would
>>> have no problem to apply it conditionally. And, as Florian confirmed,
>>> it's
>>> not just an OpenSUSE issue at the moment due to old kernels and old
>>> podman.
>>> So we will likely have to deal with it for a while.

I would be fine with that. I can try to test on my side again within
next days. From kas-container perspective the following drawbacks come
into mind:

- One more difference between docker and podman
- We might take that for a really long time,
unclear when we are save to remove this difference again

>>
>> Yes. I could add a check for non-root user
>>
>>     id -ur
>>
>> and a check for kernel version
>>
>>     uname -r
>>
>> and add the "--tmpfs /tmp" option if user is non-zero and kernel <
>> 5.13. Does
>> that sound good?
>>
>> https://www.redhat.com/sysadmin/podman-rootless-overlay
>>
>
> I suppose we would also have to take the podman version into the
> equation: I am on 5.16, and I was still able to trigger this with
> OpenSUSE's podman version.

Did we identify a version (or combination) that does not show this
behavior? I'm on 5.15.x (kernel) and podman 3.4.4 and confirmed this
issue too.

>
> Jan
>

[Trevor Woerner]

According to the following article:

https://www.redhat.com/sysadmin/podman-rootless-overlay

Podman can use native overlay file system with the Linux kernel versions 5.13.
Up until now, we have been using fuse-overlayfs. The kernel gained rootless
support in the 5.11 kernel, but a bug prevented SELinux use with the file
system; this bug was fixed in 5.13.

but a tweak is required to take advantage of it:

Sadly, you will only be able to use the native overlay with fresh storage,
meaning you will need to destroy all of your container's existing
storage. It is necessary to do a podman system reset if you already have
images/containers.

The reason for this is when a mount program is used, we store a flag file in
the storage directory: $STORAGE/overlay/.has-mount-program. If the file is
present, then c/storage ignores native overlay support. The reason for such
a check is that there are differences in how fuse-overlayfs stored metadata,
including whiteout files on older kernels that didn't allow creating the
special whiteout device for unprivileged users, and that wouldn't work if
native overlay is enabled. This means just removing the files will cause
issues with your existing containers.

The podman system reset command deletes the flag file as well. Afterward,
native overlay will be used if supported by the underlying kernel.

As far as other distributions are concerned, this support will show up when
kernel 5.13 is released. For the RHEL/CentOS Stream, we plan on backporting
the feature for the RHEL8.5 release in the fall.

The article doesn't mention anything about podman versions. I'll check with
the podman developers and see if I can get that question answered. If the
version of podman doesn't matter then all we'll need is a check on the kernel
version and a check for pre-existing storage.

[Jan Kiszka]

Given all these complications, I would suggest the following:

- add the workaround as default
- add a check inside the container (entrypoint) that we are not running
against fuse-overlay (we have one already for broken aufs)
- maybe add a version check or a switch to kas-container for disabling
the workaround later - it will take some time to propagate the
solution and make user reset their podman setups in any case

[Claudius Heine]

Hi,

I investigated this issue some time ago. What I found out was the the
cause is actually fuse-overlayfs. I think the related issue is [1]. If
you use a up-to-date version of podman and kernel, then rootless
overlayfs is supported [2].

With this, building in non-privileged containers is possible now.

regards,
Claudius

[1] https://github.com/containers/fuse-overlayfs/issues/298
[2] https://www.redhat.com/sysadmin/podman-rootless-overlay

[Trevor Woerner]

Thankfully I think we all agree on this.

>I think the related issue is [1].

The issue to which you point talks about caching and changes in lower fs
layers not bubbling up to higher layers. I don't think that's what we're
seeing here.

As I mentioned in a previous reply I believe the issue is related to fuse(8)'s
"noforget" and "remember=T" options. By default fuse will only retain so many
inodes, when it goes over its limit, it simply throws them away. If the file
in question is accessed again, it simply gets assigned a new inode.

For most usages this would be completely acceptable and the user would never
be the wiser. However, in our specific case, pseudo is purposefully creating a
database of every file←→inode mapping to make sure they never change. When
fuse rolls over and starts assigning new inodes to files that were opened
earlier, pseudo notices and throws an error assuming something nefarious is
going on.

> If you use a
> up-to-date version of podman and kernel, then rootless overlayfs is
> supported [2].

Yes, but with caveats. If the user has already created their filesystems then
they won't use the new functionality until they throw them away and
re-generate them. Therefore, unfortunately enabling or not enabling the
work-around isn't as simple as checking the versions of the kernel and podman.

[Jan Kiszka]

Is anything going to happen along what I sketched? I'm now really
considering to close for 3.0, but this issue would remain unsolved then.

[Trevor Woerner]

> > Given all these complications, I would suggest the following:
> >
> > - add the workaround as default

Okay I can do that, or the patch could just simply be applied (unless you'd
prefer a better commit message now that we understand the problem better?)

> > - add a check inside the container (entrypoint) that we are not running
> > against fuse-overlay (we have one already for broken aufs)

I have no idea how to do this.

> > - maybe add a version check or a switch to kas-container for disabling
> > the workaround later - it will take some time to propagate the
> > solution and make user reset their podman setups in any case

I'm not sure what the magic combination is that works and which ones don't, so
I'm not sure which items' versions would need checking: libfuse? Linux kernel?
podman? All of the above? Not to mention that the RedHat people back-ported
the required fix to older kernels so a simple version check of the kernel
itself would not suffice, the script would also have to add a check for the
distribution too.

Plus, a version check by itself is not enough. One could be running the latest
everything but have an old filesystem setup; in which case they would still
need this option. I'd have to investigate how to detect whether the virtual
filesystem had been updated or not.

A switch could work. Would you want it default-on or default-off? Enabling the
flag default-on for now would lead to the fewest number of users having
problems; albeit with slower builds. Maybe in 3 years we simply remove the
flag and see what happens?

[Jan Kiszka]

On 25.02.22 04:45, Trevor Woerner wrote:
> On Thu 2022-02-24 @ 06:29:59 PM, Jan Kiszka wrote:
>> On 03.02.22 07:04, Jan Kiszka wrote:
>> Is anything going to happen along what I sketched? I'm now really
>> considering to close for 3.0, but this issue would remain unsolved then.
>>> Given all these complications, I would suggest the following:
>>>
>>> - add the workaround as default
>
> Okay I can do that, or the patch could just simply be applied (unless you'd
> prefer a better commit message now that we understand the problem better?)
>

Fresh commit messages are already welcome when more information is
available.

>>> - add a check inside the container (entrypoint) that we are not running
>>> against fuse-overlay (we have one already for broken aufs)
>
> I have no idea how to do this.
>

Look at https://github.com/siemens/kas/blob/master/container-entrypoint#L3

>>> - maybe add a version check or a switch to kas-container for disabling
>>> the workaround later - it will take some time to propagate the
>>> solution and make user reset their podman setups in any case
>
> I'm not sure what the magic combination is that works and which ones don't, so
> I'm not sure which items' versions would need checking: libfuse? Linux kernel?
> podman? All of the above? Not to mention that the RedHat people back-ported
> the required fix to older kernels so a simple version check of the kernel
> itself would not suffice, the script would also have to add a check for the
> distribution too.
>
> Plus, a version check by itself is not enough. One could be running the latest
> everything but have an old filesystem setup; in which case they would still
> need this option. I'd have to investigate how to detect whether the virtual
> filesystem had been updated or not.

Yeah, we can safely postpone this magic to later.

>
> A switch could work. Would you want it default-on or default-off? Enabling the
> flag default-on for now would lead to the fewest number of users having
> problems; albeit with slower builds. Maybe in 3 years we simply remove the
> flag and see what happens?

For now: default on. Can we changed once major distros fulfill the
preconditions and only legacy setups may prevent benefiting from that.
Then the entrypoint check will at least help catching those cases.

Thanks,

[Jan Kiszka]

Just saw the open original PR on github (closed it now) and recalled
this thread. What happened to the issue?

[Bezdeka, Florian]

I re-tested that today with the following combinations at hand:

Kernel 5.18.5 / podman 4.1.1: Works (no fix needed)
Kernel 5.18.5 / podman 3.4.7: Works (no fix needed)

My Debian 11 system with
Kernel 5.10.120-1 / podman 3.0.1: Fails 
(--userns=keep-id triggers a permission denied)
This worked a few weeks ago, it seems I'm hitting a stable kernel issue
here (related to cgroupsv2).



> Jan
>

[Bezdeka, Florian]

I have to correct myself. That has been a local issue which has been
resolved now. So we additionally have

Kernel 5.10.120-1 / podman 3.0.1: Works (no fix needed)

As the issue now disappeared on my end I assume that OE has fixed /
removed the duplicate inode detection / checks. No need for fixing kas.

>
> > Jan
> >
>