[PATCH] meta: Drop lazy and recursive unmounts

[Anton Mikanovich]

From: Ilia Skochilov <iskoc...@ilbers.de>

Cleanup lazy and recursive unmounting because they just mask other
issues caused by wrong mounting.
Also remove umount || true usages for the same reason.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
Signed-off-by: Ilia Skochilov <iskoc...@ilbers.de>
---
meta/classes/deb-dl-dir.bbclass | 4 ++--
meta/classes/image.bbclass | 14 +++++++++-----
meta/classes/isar-events.bbclass | 2 +-
meta/classes/rootfs.bbclass | 10 +++++-----
meta/classes/sbuild.bbclass | 12 ++++++------
meta/classes/sdk.bbclass | 13 ++++++++++---
.../isar-bootstrap/isar-bootstrap.inc | 19 ++++++++++---------
.../sdk-files/files/README.sdk | 2 +-
scripts/mount_chroot.sh | 2 +-
9 files changed, 45 insertions(+), 33 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index d36b7190..8e0243fe 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -27,7 +27,7 @@ debsrc_do_mounts() {
set -e
mkdir -p "${1}/deb-src"
mountpoint -q "${1}/deb-src" || \
- mount --bind "${DEBSRCDIR}" "${1}/deb-src"
+ mount -o bind,private "${DEBSRCDIR}" "${1}/deb-src"
EOSUDO
}

@@ -36,7 +36,7 @@ debsrc_undo_mounts() {
set -e
mkdir -p "${1}/deb-src"
mountpoint -q "${1}/deb-src" && \
- umount -l "${1}/deb-src"
+ umount "${1}/deb-src"
rm -rf "${1}/deb-src"
EOSUDO
}
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 4f774bbc..0a80273f 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -407,19 +407,23 @@ do_rootfs_finalize() {
fi

mountpoint -q '${ROOTFSDIR}/isar-apt' && \
- umount -l ${ROOTFSDIR}/isar-apt && \
+ umount '${ROOTFSDIR}/isar-apt' && \
rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt

mountpoint -q '${ROOTFSDIR}/base-apt' && \
- umount -l ${ROOTFSDIR}/base-apt && \
+ umount '${ROOTFSDIR}/base-apt' && \
rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt

+ mountpoint -q '${ROOTFSDIR}/dev/pts' && \
+ umount '${ROOTFSDIR}/dev/pts'
+ mountpoint -q '${ROOTFSDIR}/dev/shm' && \
+ umount '${ROOTFSDIR}/dev/shm'
mountpoint -q '${ROOTFSDIR}/dev' && \
- umount -l ${ROOTFSDIR}/dev
+ umount '${ROOTFSDIR}/dev'
mountpoint -q '${ROOTFSDIR}/proc' && \
- umount -l ${ROOTFSDIR}/proc
+ umount '${ROOTFSDIR}/proc'
mountpoint -q '${ROOTFSDIR}/sys' && \
- umount -l ${ROOTFSDIR}/sys
+ umount '${ROOTFSDIR}/sys'

if [ -e "${ROOTFSDIR}/etc/apt/sources-list" ]; then
mv "${ROOTFSDIR}/etc/apt/sources-list" \
diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
index a6ba0a9e..f5061a8b 100644
--- a/meta/classes/isar-events.bbclass
+++ b/meta/classes/isar-events.bbclass
@@ -55,7 +55,7 @@ python build_completed() {
if basepath in line:
bb.debug(1, '%s left mounted, unmounting...' % line.split()[1])
subprocess.call(
- ["sudo", "umount", "-l", line.split()[1]],
+ ["sudo", "umount", line.split()[1]],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 498fbfd6..2e091e0c 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -37,11 +37,11 @@ rootfs_do_mounts() {
mountpoint -q '${ROOTFSDIR}/dev' || \
( mount -o bind,private /dev '${ROOTFSDIR}/dev' &&
mount -t tmpfs none '${ROOTFSDIR}/dev/shm' &&
- mount --bind /dev/pts '${ROOTFSDIR}/dev/pts' )
+ mount -o bind,private /dev/pts '${ROOTFSDIR}/dev/pts' )
mountpoint -q '${ROOTFSDIR}/proc' || \
mount -t proc none '${ROOTFSDIR}/proc'
mountpoint -q '${ROOTFSDIR}/sys' || \
- mount --rbind /sys '${ROOTFSDIR}/sys'
+ mount -o bind,private /sys '${ROOTFSDIR}/sys'
mount --make-rslave '${ROOTFSDIR}/sys'

# Mount isar-apt if the directory does not exist or if it is empty
@@ -51,7 +51,7 @@ rootfs_do_mounts() {
then
mkdir -p '${ROOTFSDIR}/isar-apt'
mountpoint -q '${ROOTFSDIR}/isar-apt' || \
- mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${ROOTFSDIR}/isar-apt'
+ mount -o bind,private '${REPO_ISAR_DIR}/${DISTRO}' '${ROOTFSDIR}/isar-apt'
fi

# Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set
@@ -59,7 +59,7 @@ rootfs_do_mounts() {
then
mkdir -p '${ROOTFSDIR}/base-apt'
mountpoint -q '${ROOTFSDIR}/base-apt' || \
- mount --bind '${REPO_BASE_DIR}' '${ROOTFSDIR}/base-apt'
+ mount -o bind,private '${REPO_BASE_DIR}' '${ROOTFSDIR}/base-apt'
fi

EOSUDO
@@ -360,7 +360,7 @@ rootfs_install_sstate_prepare() {
# tar --one-file-system will cross bind-mounts to the same filesystem,
# so we use some mount magic to prevent that
mkdir -p ${WORKDIR}/mnt/rootfs
- sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
+ sudo mount -o bind,private '${WORKDIR}/rootfs' '${WORKDIR}/mnt/rootfs' -o ro
lopts="--one-file-system --exclude=var/cache/apt/archives"
sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts ${SSTATE_TAR_ATTR_FLAGS} rootfs
sudo umount ${WORKDIR}/mnt/rootfs
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index f1193c20..9c268281 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -40,14 +40,14 @@ EOF
cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
sbuild_fstab="${SBUILD_CONF_DIR}/fstab"

- fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+ fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind,private 0 0"
grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}

- fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+ fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind,private 0 0"
grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}

if [ -d ${DL_DIR} ]; then
- fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+ fstab_downloads="${DL_DIR} /downloads none rw,bind,private 0 0"
grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
fi
EOSUDO
@@ -98,7 +98,7 @@ insert_mounts() {
sudo -s <<'EOSUDO'
set -e
for mp in ${SCHROOT_MOUNTS}; do
- FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+ FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind,private 0 0"
grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
done
@@ -109,7 +109,7 @@ remove_mounts() {
sudo -s <<'EOSUDO'
set -e
for mp in ${SCHROOT_MOUNTS}; do
- FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+ FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind,private 0 0"
sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
done
EOSUDO
@@ -122,7 +122,7 @@ schroot_configure_ccache() {

sbuild_fstab="${SBUILD_CONF_DIR}/fstab"

- fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind 0 0"
+ fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind,private 0 0"
grep -qxF "${fstab_ccachedir}" ${sbuild_fstab} || echo "${fstab_ccachedir}" >> ${sbuild_fstab}

(flock 9
diff --git a/meta/classes/sdk.bbclass b/meta/classes/sdk.bbclass
index 71db6f3a..754fd4cd 100644
--- a/meta/classes/sdk.bbclass
+++ b/meta/classes/sdk.bbclass
@@ -92,9 +92,16 @@ sdkchroot_configscript () {

ROOTFS_POSTPROCESS_COMMAND:append:class-sdk = " sdkchroot_finalize"
sdkchroot_finalize() {
- sudo umount -R ${ROOTFSDIR}/dev || true
- sudo umount ${ROOTFSDIR}/proc || true
- sudo umount -R ${ROOTFSDIR}/sys || true
+ mountpoint -q "${ROOTFSDIR}/dev/pts" && \
+ sudo umount "${ROOTFSDIR}/dev/pts"
+ mountpoint -q "${ROOTFSDIR}/dev/shm" && \
+ sudo umount "${ROOTFSDIR}/dev/shm"
+ mountpoint -q "${ROOTFSDIR}/dev" && \
+ sudo umount "${ROOTFSDIR}/dev"
+ mountpoint -q "${ROOTFSDIR}/proc" && \
+ sudo umount "${ROOTFSDIR}/proc"
+ mountpoint -q "${ROOTFSDIR}/sys" && \
+ sudo umount "${ROOTFSDIR}/sys"

# Remove setup scripts
sudo rm -f ${ROOTFSDIR}/chroot-setup.sh ${ROOTFSDIR}/configscript.sh
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index faf22a50..6bc667e7 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -341,7 +341,7 @@ do_bootstrap() {
echo "deb-src ${line}" >> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"

mkdir -p ${ROOTFSDIR}/base-apt
- mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt
+ mount -o bind,private "${REPO_BASE_DIR}" "${ROOTFSDIR}/base-apt"
else
install -v -m644 "${APTSRCS}" \
"${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
@@ -378,10 +378,10 @@ do_bootstrap() {

# update APT
mount -o bind,private /dev ${ROOTFSDIR}/dev
- mount --bind /dev/pts ${ROOTFSDIR}/dev/pts
+ mount -o bind,private /dev/pts "${ROOTFSDIR}/dev/pts"
mount -t tmpfs none "${ROOTFSDIR}/dev/shm"
mount -t proc none ${ROOTFSDIR}/proc
- mount --rbind /sys ${ROOTFSDIR}/sys
+ mount -o bind,private /sys "${ROOTFSDIR}/sys"
mount --make-rslave ${ROOTFSDIR}/sys

export DEBIAN_FRONTEND=noninteractive
@@ -400,12 +400,13 @@ do_bootstrap() {
chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
-o Debug::pkgProblemResolver=yes

- umount -l "${ROOTFSDIR}/dev/shm"
- umount -l "${ROOTFSDIR}/dev/pts"
- umount -l "${ROOTFSDIR}/dev"
- umount -l "${ROOTFSDIR}/proc"
- umount -l "${ROOTFSDIR}/sys"
- umount -l "${ROOTFSDIR}/base-apt" || true
+ umount "${ROOTFSDIR}/dev/shm"
+ umount "${ROOTFSDIR}/dev/pts"
+ umount "${ROOTFSDIR}/dev"
+ umount "${ROOTFSDIR}/proc"
+ umount "${ROOTFSDIR}/sys"
+ mountpoint -q "${ROOTFSDIR}/base-apt" && \
+ umount "${ROOTFSDIR}/base-apt"

# Finalize debootstrap by setting the link in deploy
ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
diff --git a/meta/recipes-devtools/sdk-files/files/README.sdk b/meta/recipes-devtools/sdk-files/files/README.sdk
index 3e06d8c5..29c09950 100644
--- a/meta/recipes-devtools/sdk-files/files/README.sdk
+++ b/meta/recipes-devtools/sdk-files/files/README.sdk
@@ -29,7 +29,7 @@ $ sudo <sdk_rootfs>/mount_chroot.sh <sdk_rootfs>

Bind-mount the project into the rootfs:

-$ sudo mount -o bind /path/to/project <sdk_rootfs>/mnt
+$ sudo mount -o bind,private /path/to/project <sdk_rootfs>/mnt

If you have relocated the SDK previously for using option 1, you need to call
this next:
diff --git a/scripts/mount_chroot.sh b/scripts/mount_chroot.sh
index e238f1cc..122bb33b 100755
--- a/scripts/mount_chroot.sh
+++ b/scripts/mount_chroot.sh
@@ -2,7 +2,7 @@

set -e

-mount /tmp "$1/tmp" -o bind
+mount /tmp "$1/tmp" -o bind,private
mount proc "$1/proc" -t proc -o nosuid,noexec,nodev
mount sysfs "$1/sys" -t sysfs -o nosuid,noexec,nodev
mount devtmpfs "$1/dev" -t devtmpfs -o mode=0755,nosuid
--
2.34.1

[Uladzimir Bely]

On Wed, 2024-06-19 at 13:41 +0300, Anton Mikanovich wrote:
> From: Ilia Skochilov <iskoc...@ilbers.de>
>
> Cleanup lazy and recursive unmounting because they just mask other
> issues caused by wrong mounting.
> Also remove umount || true usages for the same reason.
>
> Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
> Signed-off-by: Ilia Skochilov <iskoc...@ilbers.de>
> ---
>  meta/classes/deb-dl-dir.bbclass               |  4 ++--
>  meta/classes/image.bbclass                    | 14 +++++++++-----
>  meta/classes/isar-events.bbclass              |  2 +-
>  meta/classes/rootfs.bbclass                   | 10 +++++-----
>  meta/classes/sbuild.bbclass                   | 12 ++++++------
>  meta/classes/sdk.bbclass                      | 13 ++++++++++---
>  .../isar-bootstrap/isar-bootstrap.inc         | 19 ++++++++++-------
> --
>  .../sdk-files/files/README.sdk                |  2 +-
>  scripts/mount_chroot.sh                       |  2 +-
>  9 files changed, 45 insertions(+), 33 deletions(-)

Applied to next.

--
Best regards,
Uladzimir.

[Florian Bezdeka]

Hi all,

I'm not 100% sure, but it seems I'm facing some problems with this
patch. I know about the situation that we sporadically saw some
mounting issues and that patch tries to expose such issues.

At least my issue seems to reproduce to 100%...

See below...

On Wed, 2024-06-19 at 13:41 +0300, Anton Mikanovich wrote:

That means that we mount /dev from "the host" into the rootfs/chroot
environment, right? "Submounts included", no?

I'm facing the following error with that. The isar build is executed
inside a container / pod running on a k8s cluster.

Inside the container the following steps are executed:

build-bookworm:
script:
- kas build <path-to-kas-file>
- sudo rm -rf build/tmp/work/debian-bookworm-amd64/buildchroot-target

The rm command triggers the following errors:

rm: cannot remove 'build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev/shm': Device or resource busy
rm: cannot remove 'build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev/termination-log': Device or resource busy
rm: cannot remove 'build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev/mqueue': Device or resource busy
rm: cannot remove 'build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev/pts': Device or resource busy

My assumption is that isar is not cleaning up all mounts. I think the
interesting part is about /dev/termination-log.
That device is mounted into the pod by the k8s infrastructure.

I think we should never mount that into the chroot, or make sure it is
properly unmounted again...

Any thoughts on that? Any additional information needed?

Florian

> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240619104126.105252-1-amikan%40ilbers.de.

[Anton Mikanovich]

Hello Florian,

The line you are pointing to is inside isar-bootstrap preparations,
while your
error is pointing to busy mountpoints in sbuild-chroot, managed by
rootfs.bbclass.
Anyway private mount private mount does not forward or receive
propagations. It means no external submounts will be mounted inside
rootfs and
no Isar mounts will be mounted to external /dev.
Can you check which mountpoints are left after the build (before rm)?

[Florian Bezdeka]

> > - sudo rm -rf build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target

> >
> > The rm command triggers the following errors:
> >
> > rm: cannot remove 'build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev/shm': Device or resource busy
> > rm: cannot remove 'build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev/termination-log': Device or resource busy
> > rm: cannot remove 'build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev/mqueue': Device or resource busy
> > rm: cannot remove 'build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev/pts': Device or resource busy
> >
> > My assumption is that isar is not cleaning up all mounts. I think the
> > interesting part is about /dev/termination-log.
> > That device is mounted into the pod by the k8s infrastructure.
> >
> > I think we should never mount that into the chroot, or make sure it is
> > properly unmounted again...
> >
> > Any thoughts on that? Any additional information needed?
> >
> > Florian
>
> Hello Florian,
>
> The line you are pointing to is inside isar-bootstrap preparations,
> while your
> error is pointing to busy mountpoints in sbuild-chroot, managed by
> rootfs.bbclass.
> Anyway private mount private mount does not forward or receive
> propagations. It means no external submounts will be mounted inside
> rootfs and
> no Isar mounts will be mounted to external /dev.
> Can you check which mountpoints are left after the build (before rm)?

I think this is the relevant line from "mount":

tmpfs on /builds/<project>/build/tmp/work/debian-bookworm-amd64/sbuild-
chroot-target/1.0-r0/rootfs/dev type tmpfs
(rw,nosuid,size=65536k,mode=755,inode64)

Florian

[Anton Mikanovich]

29/08/2024 15:26, Florian Bezdeka wrote:
> I think this is the relevant line from "mount":
>
> tmpfs on /builds/<project>/build/tmp/work/debian-bookworm-amd64/sbuild-
> chroot-target/1.0-r0/rootfs/dev type tmpfs
> (rw,nosuid,size=65536k,mode=755,inode64)
>
> Florian

So it is sbuild who should manage mount in this rootfs. Not sure how
this patch
influence on internal sbuild mount/unmount logic. But sbuild shouldn't
left any
mounts without any errors. Do you have any packages failed to be built?

Are there any lost schroot sessions running before external rm you have
issue
with? This can be checked with 'schroot -la' command.

You can also try to add some verbose to sbuild with following change:

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index bcc3f828..b54e3ab9 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -113,6 +113,8 @@ dpkg_runbuild() {
     DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name
"${DEBIAN_SOURCE}_*.dsc" -print)

     sbuild -A -n -c ${SBUILD_CHROOT} \
+        -D \
+        --chroot-setup-commands="mount" \
         --host=${PACKAGE_ARCH} --build=${BUILD_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest
--resolve-alternatives \
         --bd-uninstallable-explainer=apt \

This will be very helpfull to get some build logs from your setup.

[Florian Bezdeka]

On Fri, 2024-09-06 at 17:34 +0300, Anton Mikanovich wrote:
> 29/08/2024 15:26, Florian Bezdeka wrote:
> > I think this is the relevant line from "mount":
> >
> > tmpfs on /builds/<project>/build/tmp/work/debian-bookworm-amd64/sbuild-
> > chroot-target/1.0-r0/rootfs/dev type tmpfs
> > (rw,nosuid,size=65536k,mode=755,inode64)
> >
> > Florian
>
> So it is sbuild who should manage mount in this rootfs. Not sure how
> this patch
> influence on internal sbuild mount/unmount logic. But sbuild shouldn't
> left any
> mounts without any errors. Do you have any packages failed to be built?
>
> Are there any lost schroot sessions running before external rm you have
> issue
> with? This can be checked with 'schroot -la' command.

This one, right after the build completed, delivers:

W: No chroots are defined in ‘/etc/schroot/schroot.conf’ or
‘/etc/schroot/chroot.d’

>
> You can also try to add some verbose to sbuild with following change:
>
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index bcc3f828..b54e3ab9 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -113,6 +113,8 @@ dpkg_runbuild() {
>      DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name
> "${DEBIAN_SOURCE}_*.dsc" -print)
>
>      sbuild -A -n -c ${SBUILD_CHROOT} \
> +        -D \
> +        --chroot-setup-commands="mount" \
>          --host=${PACKAGE_ARCH} --build=${BUILD_ARCH} ${profiles} \
>          --no-run-lintian --no-run-piuparts --no-run-autopkgtest
> --resolve-alternatives \
>          --bd-uninstallable-explainer=apt \
>
> This will be very helpfull to get some build logs from your setup.

I scanned the logs and searched for /dev mounts. Typical example:

tmp/work/debian-bookworm-amd64/isar-exclude-docs/0.2.2-r0/temp/log.do_dpkg_build.18089
521:devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
522:tmpfs on /dev/shm type tmpfs (rw,relatime,inode64)

At the end I still have one active /dev mount:
tmpfs on /builds/myproject/build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)

Trying to delete the sbuild-chroot-target delivers:

rm: cannot remove 'build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev/shm': Device or resource busy
rm: cannot remove 'build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev/termination-log': Device or resource busy
rm: cannot remove 'build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev/mqueue': Device or resource busy
rm: cannot remove 'build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev/pts': Device or resource busy

mqueue and termination-log are host specific. That might be the reason
why I can only reproduce in CI setups. k8s has some "specials" like
termination-log here.

It seems that the mount comes from the rootfs_postprocess task of
sbuild-chroot-target. Which task should undo/clean them up?

Florian

[Jan Kiszka]

If you are building an image (image.bbclass), do_rootfs_finalize will
clean up. If no (see the initramfs issue), those mounts will stick.

Jan

--
Siemens AG, Technology
Linux Expert Center

[Florian Bezdeka]

There is no do_rootfs_finalize task in the list of tasks:

20241001-074332.296794 do_fetch (805): log.do_fetch.805
20241001-074332.387021 do_unpack (969): log.do_unpack.969
20241001-074503.068122 do_rootfs_install (11792): log.do_rootfs_install.11792
20241001-074728.511151 do_rootfs_postprocess (18028): log.do_rootfs_postprocess.18028
20241001-074728.628202 do_rootfs (18057): log.do_rootfs.18057
20241001-074728.675634 do_sbuildchroot_deploy (18068): log.do_sbuildchroot_deploy.18068

Missing task dependency somewhere?

[Jan Kiszka]

Missing cleanup then for sbuild-chroot-*, analogously to initramfs_rootfs.

[Baurzhan Ismagulov]

Hello Florian,

On 2024-10-01 13:59, Florian Bezdeka wrote:
> I scanned the logs and searched for /dev mounts. Typical example:
>
> tmp/work/debian-bookworm-amd64/isar-exclude-docs/0.2.2-r0/temp/log.do_dpkg_build.18089
> 521:devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
> 522:tmpfs on /dev/shm type tmpfs (rw,relatime,inode64)

We'll address the patches with priority. Could you please provide 1. the output
of bitbake ... 2>&1 |tee bitbake.txt and 2. the related failed task
log.do_build? Off-list if sensitive.

> At the end I still have one active /dev mount:
> tmpfs on /builds/myproject/build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)

Thanks for the info, this looks like an sbuild migration bug. Checking.


With kind regards,
Baurzhan

[Florian Bezdeka]

On Thu, 2024-10-03 at 16:37 +0200, Baurzhan Ismagulov wrote:
> Hello Florian,
>
> On 2024-10-01 13:59, Florian Bezdeka wrote:
> > I scanned the logs and searched for /dev mounts. Typical example:
> >
> > tmp/work/debian-bookworm-amd64/isar-exclude-docs/0.2.2-r0/temp/log.do_dpkg_build.18089
> > 521:devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
> > 522:tmpfs on /dev/shm type tmpfs (rw,relatime,inode64)
>
> We'll address the patches with priority. Could you please provide 1. the output
> of bitbake ... 2>&1 |tee bitbake.txt and 2. the related failed task
> log.do_build? Off-list if sensitive.

I already sent patches for fixing this issue. There is no failed task.
rootfs::do_rootfs_postprocess and rootfs::do_rootfs_install both call
rootfs::rootfs_do_mounts but never clean up.

In case the mounts are left behind I will run into "can not umount,
device still active" later.

>
>
> > At the end I still have one active /dev mount:
> > tmpfs on /builds/myproject/build/tmp/work/debian-bookworm-amd64/sbuild-chroot-target/1.0-r0/rootfs/dev type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
>
> Thanks for the info, this looks like an sbuild migration bug. Checking.

Reviewing patches should do it...

>
>
> With kind regards,
> Baurzhan

[Uladzimir Bely]

On Fri, 2024-10-04 at 09:28 +0200, 'Florian Bezdeka' via isar-users
wrote:

Hello all.

I've checked this case and we are really have this mount left. It's not
related to sbuild (e.g., building packages) but to producing sbuild-
chroot itself.

E.g., entering kas shell and running "bitbake sbuild-chroot-target"
always results in "/dev" mount left in sbuild-chroot workdir rootfs.

For "image" targets we have umount routiness in image.bbclass. For
initramfs, we now have Jan's patches.

But I guess that "[PATCH 0/2] Fix leftover mounts in rootfs.bbclass"
patchset should itself cover all cases. And we might not even need
special handling in image.bbclass and initramfs.bbclass.

Probably, only the case when an error happens before rootf_postprocess
(e.g., in image_install) may lead to some mounts left.

> >
> >
> > With kind regards,
> > Baurzhan
>

--
Best regards,
Uladzimir.

[Jan Kiszka]

We can and possibly should make this work - but not in the way those
patches currently interact. Please check once again.

Jan

>
> Probably, only the case when an error happens before rootf_postprocess
> (e.g., in image_install) may lead to some mounts left.
>
>>>
>>>
>>> With kind regards,
>>> Baurzhan
>>
>

--

[Jan Kiszka]

No, we should not start to consolidate the umounts again. If we do that,
we could also drop the warning from the catch-all handler
build_completed and let it umount everything, as we've done in the past.
If we don't want this anymore, every individual task that does mounting
must also do the umounting at its end. And that means even the umount in
do_rootfs_finalize would become obsolete.

Jan

[Baurzhan Ismagulov]

On 2024-10-10 06:33, 'Jan Kiszka' via isar-users wrote:
> No, we should not start to consolidate the umounts again. If we do that,
> we could also drop the warning from the catch-all handler
> build_completed and let it umount everything, as we've done in the past.
> If we don't want this anymore, every individual task that does mounting
> must also do the umounting at its end. And that means even the umount in
> do_rootfs_finalize would become obsolete.

I agree, the vision has always been to mount-umount as locally as possible. To
avoid misunderstanding, it isn't about "consolidate the umounts again" -- the
single goal of the cleanup handler is to warn about the problem; unmounting was
done as a by-product only to return a saner state to the user, as executing
sudo rm -rf build afterwards might delete the host /dev, making the system
unusable.

On the whole, the approach in the patches looks good to us; we'll provide a
consolidated patchset today / tomorrow as long as we don't encounter any
surprises underway.

With kind regards,
Baurzhan