schroot umount target is busy

[Schultschik, Sven]

Hi all,

Our ISAR setup is running smooth on many machines different sizes. But
on our CI/CD pipeline it throws from time to time an error. Starting it
over again it works.

Does someone knows this kind of error and where I have to look or how
to fix?

2024-06-05 04:00:41 - INFO - | Processing triggers for man-db
(2.11.2-2) ...
2024-06-05 04:00:41 - INFO - | e2fsck 1.47.0 (5-Feb-2023)
2024-06-05 04:00:41 - INFO - | Pass 1: Checking inodes, blocks, and
sizes
2024-06-05 04:00:41 - INFO - | Pass 2: Checking directory structure
2024-06-05 04:00:41 - INFO - | Pass 3: Checking directory
connectivity
2024-06-05 04:00:41 - INFO - | Pass 3A: Optimizing directories
2024-06-05 04:00:41 - INFO - | Pass 4: Checking reference counts
2024-06-05 04:00:41 - INFO - | Pass 5: Checking group summary
information
2024-06-05 04:00:41 - INFO - |
2024-06-05 04:00:41 - INFO - | /home/builder/edge-debug-machine-
std/deploy/debug-machine-std.ext4: ***** FILE SYSTEM WAS MODIFIED *****
2024-06-05 04:00:41 - INFO - | /home/builder/debug-machine-
std/deploy/debug-machine-std.ext4: 25039/77600 files (0.1% non-
contiguous), 292861/310076 blocks
2024-06-05 04:00:41 - INFO - | E: 10mount: umount:
/run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd-11546112ec7c-
982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: target is busy.
2024-06-05 04:00:41 - INFO - | E: 10mount: rmdir: failed to remove
'/var/run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd-
11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3': Device or
resource busy
2024-06-05 04:00:41 - INFO - | E: isar-builder-e7f8d532-aa0e-41cf-
85dd-11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: Chroot
setup failed: stage=setup-stop
2024-06-05 04:00:41 - INFO - NOTE: recipe debug-1.0-r0: task
do_image_ext4: Failed
2024-06-05 04:00:41 - ERROR - ERROR: Task (/build/../repo/recipes-
core/images/debug.bb:do_image_ext4) failed with exit code '1'
2024-06-05 04:00:42 - INFO - NOTE: Tasks Summary: Attempted 896
tasks of which 0 didn't need to be rerun and 1 failed.
2024-06-05 04:00:43 - INFO -
2024-06-05 04:00:43 - INFO - Summary: 1 task failed:
2024-06-05 04:00:43 - INFO - /build/../repo/recipes-
core/images/debug.bb:do_image_ext4
2024-06-05 04:00:43 - INFO - Summary: There was 1 WARNING message.
2024-06-05 04:00:43 - INFO - Summary: There was 1 ERROR message,
returning a non-zero exit code.
2024-06-05 04:00:43 - ERROR - Command "/build$
/work/isar/bitbake/bin/bitbake -c build debug" failed
--- Error summary ---
ERROR: debug-1.0-r0 do_image_ext4:
ExecutionError('/build/tmp/work/arm64/debug-machine-std/1.0-
r0/temp/run.do_image_ext4.982511', 1, None, None)
ERROR: Logfile of failure stored in: /build/tmp/work/arm64/debug-
machine-std/1.0-r0/temp/log.do_image_ext4.982511
ERROR: Task (/build/../repo/recipes-core/images/debug.bb:do_image_ext4)
failed with exit code '1'

Thank you

br Sven

[Jan Kiszka]

Cedric, weren't you seeing something very similar?

Jan

--
Siemens AG, Technology
Linux Expert Center

[Gökhan Çetin]

Hi everyone,

Similarly, we have been observing do_image tasks fail sporadically in CI/CD pipelines for a while.

You can see a few examples below:

INFO - | E: 10mount: umount: /run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: target is busy.
INFO - | E: 10mount: rmdir: failed to remove '/var/run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b': Device or resource busy
INFO - | E: isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: Chroot setup failed: stage=setup-stop
INFO - NOTE: recipe .......: task do_image_swu: Failed
ERROR - ERROR: Task (/builds/.......bb:do_image_swu) failed with exit code '1'

INFO - | E: 10mount: umount: /run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: target is busy.
INFO - | E: 10mount: rmdir: failed to remove '/var/run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82': Device or resource busy
INFO - | E: isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: Chroot setup failed: stage=setup-stop
INFO - NOTE: recipe ......: task do_image_ova: Failed
ERROR - ERROR: Task (/builds/.....bb:do_image_ova) failed with exit code '1'

Do you have any information regarding the root cause?

Thanks,

Gokhan

[Jan Kiszka]

On 08.07.24 16:17, Gökhan Çetin wrote:
> Hi everyone,
>
> Similarly, we have been observing do_image tasks fail sporadically in
> CI/CD pipelines for a while.
> You can see a few examples below:
>
> INFO - | E: 10mount: umount:
> /run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: target is busy.
> INFO - | E: 10mount: rmdir: failed to remove
> '/var/run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b': Device or resource busy
> INFO - | E:
> isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: Chroot setup failed: stage=setup-stop
> INFO - NOTE: recipe .......: task do_image_swu: Failed
> ERROR - ERROR: Task (/builds/.......bb:do_image_swu) failed with exit
> code '1'
>
> INFO - | E: 10mount: umount:
> /run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: target is busy.
> INFO - | E: 10mount: rmdir: failed to remove
> '/var/run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82': Device or resource busy
> INFO - | E:
> isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: Chroot setup failed: stage=setup-stop
> INFO - NOTE: recipe ......: task do_image_ova: Failed
> ERROR - ERROR: Task (/builds/.....bb:do_image_ova) failed with exit code '1'
>
> Do you have any information regarding the root cause?

Trying once more to pull Cedric in (I think I used the wrong address
last time): Weren't you trying to resolve that with a lazy umount? Or
was it for a different corner of isar?

But even if we lazily umount, this won't help when the next command is
trying to rmdir the mount point...

Does anyone have a feeling when this started? Does it correlate with any
major changes in isar?

Jan

> --
> You received this message because you are subscribed to the Google
> Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to isar-users+...@googlegroups.com
> <mailto:isar-users+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/d2508b62-d2fb-44a7-936f-aa2367347662n%40googlegroups.com <https://groups.google.com/d/msgid/isar-users/d2508b62-d2fb-44a7-936f-aa2367347662n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Gökhan Çetin]

Hi Jan,

Thanks for pointing that out, I was not aware the fix on lazy umounts commited 2 weeks ago: https://github.com/ilbers/isar/commit/ed2f8fda7cc85309c4f9a1f49a3f4b96df9886bb

Don't know if the issue on Sven's side has been resolved by this. Just trying out, will let you know if it will help us.

Best,

Gokhan

[Anton Mikanovich]

09/07/2024 09:28, Gökhan Çetin wrote:
> Hi Jan,
>
> Thanks for pointing that out, I was not aware the fix on lazy umounts
> commited 2 weeks
> ago: https://github.com/ilbers/isar/commit/ed2f8fda7cc85309c4f9a1f49a3f4b96df9886bb
>
> Don't know if the issue on Sven's side has been resolved by this. Just
> trying out, will let you know if it will help us.
>
> Best,
> Gokhan

Hello Gökhan,

The issue you've observed is probably caused by some mounts still left
mounted
during schroot session stop phase (imager_run API).
Lazy umount fix will not resolve those mounts, but makes them visible
for debug.

Schroot mounts are controlled by fstab, which is filled from SCHROOT_MOUNTS
value. If there are some mounts performing manually during imager_run
call, it
should be also unmounted in the same call.

[cedric.h...@siemens.com]

On Mon, 2024-07-08 at 17:13 +0200, Jan Kiszka wrote:
> On 08.07.24 16:17, Gökhan Çetin wrote:
> > Hi everyone,
> >
> > Similarly, we have been observing do_image tasks fail sporadically
> > in
> > CI/CD pipelines for a while.
> > You can see a few examples below:
> >
> > INFO - | E: 10mount: umount:
> > /run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-
> > dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: target is
> > busy.
> > INFO - | E: 10mount: rmdir: failed to remove
> > '/var/run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-
> > dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b': Device
> > or resource busy
> > INFO - | E:
> > isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-
> > b359-48ad-91f4-2962f4bdf32b: Chroot setup failed: stage=setup-stop
> > INFO - NOTE: recipe .......: task do_image_swu: Failed
> > ERROR - ERROR: Task (/builds/.......bb:do_image_swu) failed with
> > exit
> > code '1'
> >
> > INFO - | E: 10mount: umount:
> > /run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-

> > 57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: target is

> > busy.
> > INFO - | E: 10mount: rmdir: failed to remove

> > '/var/run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-
> > 57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82': Device

> > or resource busy
> > INFO - | E:

> > isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-

> > 8c19-4fc8-9b5e-d2ccb3d67d82: Chroot setup failed: stage=setup-stop
> > INFO - NOTE: recipe ......: task do_image_ova: Failed
> > ERROR - ERROR: Task (/builds/.....bb:do_image_ova) failed with exit
> > code '1'
> >
> > Do you have any information regarding the root cause?
>
> Trying once more to pull Cedric in (I think I used the wrong address
> last time): Weren't you trying to resolve that with a lazy umount? Or
> was it for a different corner of isar?

Correct. On our side, this issue is seen sporadically on servers
managed by our IT where they have some proprietary / black-box services
scanning the file-system when new mounts are discovered...

Have not found an elegant way to fix or work-around this (and obviously
no luck in getting IT to remove their software or configure it
differently)

Cedric Hombourger
Siemens AG
http://www.siemens.com/

[Jan Kiszka]

Hehe, this is the point where moving to a VM will both be more secure
and faster.

Seriously, all we could do for such a scenario is adding some finite
retry loop. Messy.

[Anton Mikanovich]

Hello Gökhan,

I've found an issue with isar-apt usage without lock protection in
deb_dl_dir_export and imager_run implementations. This can influence on
umount
issues you've observed, so please try the fix from patchset:

[PATCH 0/2] Protect isar-apt usages in shell

[Anton Mikanovich]

Unfortunately this fix didn't help.
Even with it we've got the same issue (it looks like there is the first time
we've catched it in our CI):

ERROR: Task
(mc:qemuarm-bullseye:/workspace/build-auto/isar_next/1324/meta-test/recipes-core/images/isar-image-ci.bb:do_image_ext4)

failed with exit code '1'

E: 10mount: umount:
/run/schroot/mount/isar-jenkins-0c03d5d2-da53-444c-93f6-fc9bdfafc290-2097734-1868b26c-f678-4e99-9e9a-360c7c05fcd3:
target is busy.

E: 10mount: rmdir: failed to remove

'/var/run/schroot/mount/isar-jenkins-0c03d5d2-da53-444c-93f6-fc9bdfafc290-2097734-1868b26c-f678-4e99-9e9a-360c7c05fcd3':
Device or resource busy
E:
isar-jenkins-0c03d5d2-da53-444c-93f6-fc9bdfafc290-2097734-1868b26c-f678-4e99-9e9a-360c7c05fcd3:
Chroot setup failed: stage=setup-stop

[Gökhan Çetin]

Hi Anton,

I haven't had a chance to try the patch you mentioned yet.

But this fix has greatly reduced the error frequency on our side.

As Cedric mentioned, we also have managed servers and several IT services that might cause such issues.

That's why, I can say that we have achieved acceptable stability for now.

Thanks,

Gokhan

[Schultschik, Sven]

Hello everyone,

I saw that there where several patches in the past and it seemed
better, but we're phasing again issues with this problem in the CI/CD
pipeline.

We are currently on the 0.10 isar version.

2025-08-05 19:44:23 - INFO - | update-alternatives: using /bin/mt-
gnu to provide /bin/mt (mt) in auto mode
2025-08-05 19:44:23 - INFO - | Processing triggers for man-db
(2.11.2-2) ...
2025-08-05 19:44:23 - INFO - | 606473 blocks
2025-08-05 19:44:23 - INFO - | E: 10mount: umount:
/run/schroot/mount/isar-builder-422f34fc-0228-432e-89c2-dcb36197702b-
388848-c8bce6c5-6eb2-444d-bafc-b94069fd47dc/home/builder/audis-lpe-
rescue-debug-lpe9403-std/deploy: target is busy.
2025-08-05 19:44:23 - INFO - | E: 10mount: umount:
/run/schroot/mount/isar-builder-422f34fc-0228-432e-89c2-dcb36197702b-
388848-c8bce6c5-6eb2-444d-bafc-b94069fd47dc: target is busy.
2025-08-05 19:44:23 - INFO - | E: 10mount: rmdir: failed to remove
'/var/run/schroot/mount/isar-builder-422f34fc-0228-432e-89c2-
dcb36197702b-388848-c8bce6c5-6eb2-444d-bafc-b94069fd47dc': Device or
resource busy
2025-08-05 19:44:23 - INFO - | E: isar-builder-422f34fc-0228-432e-
89c2-dcb36197702b-388848-c8bce6c5-6eb2-444d-bafc-b94069fd47dc: Chroot
setup failed: stage=setup-stop
2025-08-05 19:44:23 - INFO - NOTE: recipe rescue-debug-1.0-r0: task
do_image_cpio: Failed
2025-08-05 19:44:23 - ERROR - ERROR: Task (/build/../work/lpe9403-
base/recipes-core/images/audis-lpe-rescue-debug.bb:do_image_cpio)

failed with exit code '1'

2025-08-05 21:06:41 - INFO - | Processing triggers for man-db
(2.11.2-2) ...
2025-08-05 21:06:41 - INFO - | tune2fs 1.47.0 (5-Feb-2023)
2025-08-05 21:06:41 - INFO - | E: 10mount: umount:
/run/schroot/mount/isar-builder-7acbebe3-8b26-4def-abd5-a48d19ef4673-
1166849-61509bf3-3372-41e4-ba7d-1d13acb5fa42: target is busy.
2025-08-05 21:06:41 - INFO - | E: 10mount: rmdir: failed to remove
'/var/run/schroot/mount/isar-builder-7acbebe3-8b26-4def-abd5-
a48d19ef4673-1166849-61509bf3-3372-41e4-ba7d-1d13acb5fa42': Device or
resource busy
2025-08-05 21:06:41 - INFO - | E: isar-builder-7acbebe3-8b26-4def-
abd5-a48d19ef4673-1166849-61509bf3-3372-41e4-ba7d-1d13acb5fa42: Chroot
setup failed: stage=setup-stop
2025-08-05 21:06:41 - INFO - NOTE: recipe image-debug-1.0-r0: task
do_image_ext4: Failed
2025-08-05 21:06:41 - ERROR - ERROR: Task (/build/../repo/recipes-
core/images/audis-lpe-image-debug.bb:do_image_ext4) failed with exit
code '1'

bg Sven