ISAR schroot mountpoints when running in container

[Moessbauer, Felix]

Hi,

 

as we now have sbuild in ISAR next, first users stumble upon issues when running the kas docker image in the gitlab-ci.

This requires additional mountpoints, as the schroot itself uses overlayfs and stacking two overlayfs is not possible.

By that, you have to mount the overlay from the host.

 

The kas-container script already has support for that, but it lacks documentation on how to configure this manually.
In short: The following mountpoint has to be added (as RW): /var/lib/schroot/union/overlay

 

As this is neither an ISAR issue, nor a KAS issue per-se, I send this to both lists.

 

Felix

--

Siemens AG, Linux Expert Center
Otto-Hahn-Ring 6, 81739 München, Germany

 

[Bezdeka, Florian]

On Fri, 2022-07-01 at 09:11 +0000, Moessbauer, Felix wrote:
> Hi,
>  
> as we now have sbuild in ISAR next, first users stumble upon issues
> when running the kas docker image in the gitlab-ci.
> This requires additional mountpoints, as the schroot itself uses
> overlayfs and stacking two overlayfs is not possible.
> By that, you have to mount the overlay from the host.
>  
> The kas-container script already has support for that, but it lacks
> documentation on how to configure this manually.
> In short: The following mountpoint has to be added (as RW):
> /var/lib/schroot/union/overlay

So you added this mount point to your gitlab runner configuration? If
so, can this mount (safely) be shared between multiple gitlab build
jobs (running at the same time)?

[Moessbauer, Felix]

> -----Original Message-----
> From: Bezdeka, Florian (T CED SES-DE) <florian...@siemens.com>
> Sent: Friday, July 1, 2022 11:28 AM
> To: isar-...@googlegroups.com; kas-...@googlegroups.com; Moessbauer,
> Felix (T CED SES-DE) <felix.mo...@siemens.com>
> Cc: Bossert, Andre (DI FA CTR SVC PRC2) <andre....@siemens.com>; Kiszka,
> Jan (T CED) <jan.k...@siemens.com>; Schild, Henning (T CED SES-DE)
> <henning...@siemens.com>
> Subject: Re: ISAR schroot mountpoints when running in container
>
> On Fri, 2022-07-01 at 09:11 +0000, Moessbauer, Felix wrote:
> > Hi,
> >
> > as we now have sbuild in ISAR next, first users stumble upon issues
> > when running the kas docker image in the gitlab-ci.
> > This requires additional mountpoints, as the schroot itself uses
> > overlayfs and stacking two overlayfs is not possible.
> > By that, you have to mount the overlay from the host.
> >
> > The kas-container script already has support for that, but it lacks
> > documentation on how to configure this manually.
> > In short: The following mountpoint has to be added (as RW):
> > /var/lib/schroot/union/overlay
>
> So you added this mount point to your gitlab runner configuration? If so, can this
> mount (safely) be shared between multiple gitlab build jobs (running at the same
> time)?

The question boils down to "can this mountpoint safely be shared across ISAR runs", and this should be possible.
We run this code for a while and never experienced any sharing issues.

The schroots also have a PID component in their name. [1]
But I'm not 100% sure that the PIDs in the privileged container are unique across all containers.
At least for non-privileged containers you have local-only PIDs (or better, a local PID namespace).

Felix

[1] https://github.com/ilbers/isar/blob/next/meta/classes/sbuild.bbclass#L26

[Bezdeka, Florian]

On Fri, 2022-07-01 at 09:36 +0000, Moessbauer, Felix (T CED SES-DE)
wrote:

Privileged containers have their own PID namespace as well - as long as
you do not tell the container runtime to share it - so we might end up
with bitbake processes having the same PID for different builds.

[Uladzimir Bely]

In the email from Friday, 1 July 2022 12:11:42 +03 user Moessbauer, Felix
wrote:

> Hi,
>
> as we now have sbuild in ISAR next, first users stumble upon issues when
> running the kas docker image in the gitlab-ci. This requires additional
> mountpoints, as the schroot itself uses overlayfs and stacking two
> overlayfs is not possible. By that, you have to mount the overlay from the
> host.
>
> The kas-container script already has support for that, but it lacks
> documentation on how to configure this manually. In short: The following
> mountpoint has to be added (as RW): /var/lib/schroot/union/overlay
>
> As this is neither an ISAR issue, nor a KAS issue per-se, I send this to
> both lists.
>

Hello.

Yes, this overlayfs-over-overlayfs issue can be solved by something like

`volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]`

placed to `/etc/gitlab-runner/config.toml`, as mentioned in 'sbuild' series
cover-letter

> Felix
> --
> Siemens AG, Linux Expert Center
> Otto-Hahn-Ring 6, 81739 München, Germany

--
Uladzimir Bely

[Bezdeka, Florian]

On Fri, 2022-07-01 at 13:23 +0300, Uladzimir Bely wrote:
> In the email from Friday, 1 July 2022 12:11:42 +03 user Moessbauer, Felix
> wrote:
> > Hi,
> >
> > as we now have sbuild in ISAR next, first users stumble upon issues when
> > running the kas docker image in the gitlab-ci. This requires additional
> > mountpoints, as the schroot itself uses overlayfs and stacking two
> > overlayfs is not possible. By that, you have to mount the overlay from the
> > host.
> >
> > The kas-container script already has support for that, but it lacks
> > documentation on how to configure this manually. In short: The following
> > mountpoint has to be added (as RW): /var/lib/schroot/union/overlay
> >
> > As this is neither an ISAR issue, nor a KAS issue per-se, I send this to
> > both lists.
> >
>
> Hello.
>
> Yes, this overlayfs-over-overlayfs issue can be solved by something like
>
> `volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]`
>
> placed to `/etc/gitlab-runner/config.toml`, as mentioned in 'sbuild' series
> cover-letter

Oh no. That's going to kill a lot of gitlab-ci setups. Even kas-
container based environments (often used as local development
environment) will need adjustments. Uncool.

Anyway, I will try to look into the kas-container script. Let's hope
there is a simple solution for adding one more mount.

[Jan Kiszka]

When designing that, we should also have an eye on the optimization of
mapping build/tmp/ onto a tmpfs mount inside the container.

Jan

--
Siemens AG, Technology
Competence Center Embedded Linux

[Moessbauer, Felix]

> -----Original Message-----
> From: Kiszka, Jan (T CED) <jan.k...@siemens.com>
> Sent: Friday, July 1, 2022 12:44 PM
> To: Bezdeka, Florian (T CED SES-DE) <florian...@siemens.com>;
> ub...@ilbers.de; isar-...@googlegroups.com; Moessbauer, Felix (T CED SES-
> DE) <felix.mo...@siemens.com>
> Subject: Re: ISAR schroot mountpoints when running in container
>

Wait a minute: We are mixing things up here.

1. kas-container (script) already provides this mountpoint when enabling isar mode. By that, local deployments are not affected
2. When running the kas container image directly (via docker, podman or gitlab-ci), the mount point is missing. There is not much we can do about, except to document it for the user.
3. Schroot ID collisions: When running two kas containers in parallel, that also use the same mountpoints on the host, we likely get name collisions (as PIDs are no longer unique). This scenario is common for gitlab-ci.
4. Using tmpfs: With Schroot, we could easily put just the schroot onto a tmpfs. This has the further advantage, that the upper-dir of the overlayfs can be a tmpfs which itself can be mounted inside the container. By that, no bind-mount from the host is required. But this might only work for machines with a lot of memory.

Required changes:

In ISAR, we have to make the name of the Schroot folder more unique. But as BB requires recipes to be deterministic (per-build), we have to inject the ID from the outside. This could happen either via local.conf or via an env-var. This env-var has to be provided by KAS, with an fallback in ISAR to use the PID of the bitbake process if not provided.

A probably better strategy would be to get a per-bitbake invocation constant UUID directly from Bitbake. Don't know if that already exists in BB.
Putting Adriaan in CC.

Felix

[Henning Schild]

Am Fri, 1 Jul 2022 11:30:39 +0000
schrieb "Moessbauer, Felix" <felix.mo...@siemens.com>:

What if the runner had btrfs? I guess that would be a good
suggestion which would lead to no overlay at the base, not needing a
volume mount in the config.

Henning

[Bezdeka, Florian]

My fault. Looked at the wrong sources. Fine, nothing to do here.

> > 2.
> > When running the kas container image directly (via docker, podman or
> > gitlab-ci), the mount point is missing. There is not much we can do
> > about, except to document it for the user. 
> >

True but still a new breaking change.

> > 3. Schroot ID collisions:
> > When running two kas containers in parallel, that also use the same
> > mountpoints on the host, we likely get name collisions (as PIDs are
> > no longer unique). This scenario is common for gitlab-ci. 
> >

That was my main concern. Even when the mount is properly propagated
into the container we might see strange build behavior.

> > 4. Using
> > tmpfs: With Schroot, we could easily put just the schroot onto a
> > tmpfs. This has the further advantage, that the upper-dir of the
> > overlayfs can be a tmpfs which itself can be mounted inside the
> > container. By that, no bind-mount from the host is required. But this
> > might only work for machines with a lot of memory.

Interesting idea, but yes I would expect OOM everywhere. Do we have
some numbers from our internal layers how much space would be required?

>
> What if the runner had btrfs? I guess that would be a good
> suggestion which would lead to no overlay at the base, not needing a
> volume mount in the config.

We might have both. There is no guarantee that all runners have btrfs.
We need something that works FS independent.

[Anton Mikanovich]

01.07.2022 14:30, Moessbauer, Felix wrote:
> Required changes:
>
> In ISAR, we have to make the name of the Schroot folder more unique. But as BB requires recipes to be deterministic (per-build), we have to inject the ID from the outside. This could happen either via local.conf or via an env-var. This env-var has to be provided by KAS, with an fallback in ISAR to use the PID of the bitbake process if not provided.
>
> A probably better strategy would be to get a per-bitbake invocation constant UUID directly from Bitbake. Don't know if that already exists in BB.
> Putting Adriaan in CC.
>
> Felix

Hello, I've already proposed unique per-build ID generation in '[PATCH
2/6] base: Implement bitbake build ID'.
Not sure it suits mentioned requirements, but can be good starting point.

[Moessbauer, Felix]

> -----Original Message-----
> From: Anton Mikanovich <ami...@ilbers.de>
> Sent: Friday, July 1, 2022 2:09 PM
> To: Moessbauer, Felix (T CED SES-DE) <felix.mo...@siemens.com>; isar-
> us...@googlegroups.com
> Cc: Kiszka, Jan (T CED) <jan.k...@siemens.com>; Bezdeka, Florian (T CED SES-
> DE) <florian...@siemens.com>; ub...@ilbers.de; Schmidt, Adriaan (T CED
> SES-DE) <adriaan...@siemens.com>
> Subject: Re: ISAR schroot mountpoints when running in container
>

Just had a look at the patch. That should also work, but only if the date / time information is valid.
In some environments which are used to test reproducible builds, date / time might be fixed or redacted.

I just sent out another approach that relies on an externally provided UUID.
Don't know which one is better.

Felix

[Gylstorff Quirin]

I did not find anything about it on the mailing list for sbuild but
there are alternative to schroot available with the sbuild option
--chroot-mode(schroot|sudo|autopkgtest|unshare)[1]. Did we test or
discuss any of these modes?

Also as we generate the schroot configuration can we disable the overlay
usage[2]?


[1]: https://manpages.debian.org/bullseye/sbuild/sbuild.1.en.html
[2]:
https://manpages.debian.org/bullseye/schroot/schroot.conf.5.en.html#Filesystem_Union_chroot_options

Quirin

[Moessbauer, Felix]

I don't know if this has been discussed, but personally I tried multiple:

- schroot: That's what we currently use (base fs layer + overlay per sbuild invocation)
- sudo: similar to "sudo chroot ...". This suffers from the same problems as the previous ISAR implementation of the global buildchroot
- unshare: That's the best (IMO), but the feature-support heavily depends on the host-system. Issues are around missing /dev/pts, binfmt, broken pkg-autotest

Apart from that, two additional things have to be considered as well:
Mem usage: I personally run sbuild with unshare backend on a tmpfs, but depending on the package this requires gigabytes of RAM. In ISAR, the builds run in parallel, hence it does not really scale.
Disk usage: Having multiple full-blown chroots requires a lot of disk space. That's why the basic build infrastructure is put into the lower-dir of the overlayfs, while only the per-package build-dependencies are installed into the upper.

I hope this clarifies some of the design decisions, although they have not been made by me 😉

Felix

>
> Also as we generate the schroot configuration can we disable the overlay
> usage[2]?

No, this is not possible (at least it is not implemented).

Felix

>
>
> [1]:
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmanpage
> s.debian.org%2Fbullseye%2Fsbuild%2Fsbuild.1.en.html&amp;data=05%7C01%7
> Cfelix.moessbauer%40siemens.com%7Cd2f153a84cc84974391408da5b73840c
> %7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C63792284893296229
> 8%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ
> BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=QoQWuM8
> dI2drew1QAKnLXSIFWmKA5baR1PuiZ%2FF73MQ%3D&amp;reserved=0
> [2]:
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmanpage
> s.debian.org%2Fbullseye%2Fschroot%2Fschroot.conf.5.en.html%23Filesystem_
> Union_chroot_options&amp;data=05%7C01%7Cfelix.moessbauer%40siemens.c
> om%7Cd2f153a84cc84974391408da5b73840c%7C38ae3bcd95794fd4addab42e
> 1495d55a%7C1%7C0%7C637922848932962298%7CUnknown%7CTWFpbGZsb3d
> 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> %7C3000%7C%7C%7C&amp;sdata=1%2BcWUkyDvXEn20dGcIE5%2F0qn2AK4RA
> Dx%2F03QUUjh%2Fiw%3D&amp;reserved=0
>
> Quirin