[PATCH 0/3] Rootfs install race fix for isar-apt packages
Anton Mikanovich
a race during isar-apt packages installation into rootfs. If some
package performs do_deploy_deb task during do_rootfs_install execution
it can remove its deb in the moment when it read by apt-get.
This happens because our existing logic for precaching debs under
isar-apt lock was not working for isar-apt packages, because apt-get
just do not cache packages from local repos in /var/cache/apt/archives.
To fix this, introduce a new rootfs install command which obtains all
the packages not been cached and download them manually.
Also add "--no-download" option to the final apt-get install call to
prevent masking such an issues in the future.
Anton Mikanovich (3):
rootfs: Allow locking on single install command
rootfs: Download isar-apt packages under isar-apt lock
rootfs: Deny packages download during install
meta/classes-recipe/rootfs.bbclass | 29 ++++++++++++++++++++++++++---
1 file changed, 26 insertions(+), 3 deletions(-)
--
2.34.1
Anton Mikanovich
(started with file:/) in /var/cache/apt/archives even with
--download-only option enabled. It results in isar-apt packages
download execution during rootfs_install_pkgs_install command, which is
not covered under isar-apt lock.
To put isar-apt packages in local cache under isar-apt lock new command
was introduced. It obtain all the packages not been cached and download
them manually.
This change fixes "file not found" issues during rootfs installation.
Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---
meta/classes-recipe/rootfs.bbclass | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index dc4e702c..81384340 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -363,6 +363,29 @@ rootfs_export_package_cache() {
deb_dl_dir_export ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
}
+ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_isar_download"
+rootfs_install_pkgs_isar_download[weight] = "50"
+rootfs_install_pkgs_isar_download[isar-apt-lock] = "acquire-before release-after"
+rootfs_install_pkgs_isar_download() {
+ mkdir -p "${WORKDIR}/dpkg"
+
+ # Use our own dpkg lock files rather than those in the rootfs since we are not root
+ # (this is safe as there are no concurrent apt/dpkg operations for that rootfs)
+ touch "${WORKDIR}/dpkg/lock" "${WORKDIR}/dpkg/lock-frontend"
+
+ # Command apt-get install do not cache packages from local repos
+ # We can obtain non cached package URIs by recalling install command here
+ # No need in export those files to dl_dir, so we can run it right after
+ rootfs_cmd --bind "${ROOTFSDIR}/var/cache/apt/archives" /var/cache/apt/archives \
+ --bind "${WORKDIR}/dpkg/lock" /var/lib/dpkg/lock \
+ --bind "${WORKDIR}/dpkg/lock-frontend" /var/lib/dpkg/lock-frontend \
+ --chdir "/var/cache/apt/archives" \
+ ${ROOTFSDIR} \
+ -- /usr/bin/sh -c "apt-get ${ROOTFS_APT_ARGS} --print-uris ${ROOTFS_PACKAGES} | \
+ sed -n \"s|^.*/\\(.*\\)_[^_]*_[^_]*\\.deb'.*|\\1|p\" | \
+ xargs -r apt-get download"
+}
+
ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}"
rootfs_install_clean_files[weight] = "2"
rootfs_install_clean_files() {
--
2.34.1
Anton Mikanovich
can require isar-apt locking. To allow this logic improve flag checking
to accept the list of codenames. Existing API and codenames
("acquire-before" and "release-after") were not changed.
So the usage of single task declaration now looks like:
rootfs_install_cmd[isar-apt-lock] = "acquire-before release-after"
meta/classes-recipe/rootfs.bbclass | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index 8b502a50..dc4e702c 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -427,13 +427,13 @@ python do_rootfs_install() {
for cmd in cmds:
progress_reporter.next_stage()
- if (d.getVarFlag(cmd, 'isar-apt-lock') or "") == "acquire-before":
+ if "acquire-before" in (d.getVarFlag(cmd, 'isar-apt-lock') or ""):
lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
shared=True)
bb.build.exec_func(cmd, d)
- if (d.getVarFlag(cmd, 'isar-apt-lock') or "") == "release-after":
+ if "release-after" in (d.getVarFlag(cmd, 'isar-apt-lock') or ""):
bb.utils.unlockfile(lock)
progress_reporter.finish()
finally:
--
2.34.1
Anton Mikanovich
downloading should be done. All the debs must be precached and already
placed in /var/cache/apt/archives, so apt-get can reuse them.
Currently any downloads in this command are really succeed because we
still have "network" flag on for using sudo, which masks all the
caching issues in previous rootfs install commands.
To prevent such a cases add "--no-download" option to apt-get.
meta/classes-recipe/rootfs.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index 81384340..f4580421 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -402,7 +402,7 @@ rootfs_install_pkgs_install[progress] = "custom:rootfs_progress.PkgsInstallProgr
rootfs_install_pkgs_install[network] = "${TASK_USE_SUDO}"
rootfs_install_pkgs_install() {
sudo -E chroot "${ROOTFSDIR}" \
- /usr/bin/apt-get ${ROOTFS_APT_ARGS} ${ROOTFS_PACKAGES}
+ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --no-download ${ROOTFS_PACKAGES}
}
ROOTFS_INSTALL_COMMAND += "rootfs_restore_initrd_tooling"
--
2.34.1
MOESSBAUER, Felix
> In some cases only one command from all the ROOTFS_INSTALL_COMMAND list
> can require isar-apt locking. To allow this logic improve flag checking
> to accept the list of codenames. Existing API and codenames
> ("acquire-before" and "release-after") were not changed.
>
> So the usage of single task declaration now looks like:
>
> rootfs_install_cmd[isar-apt-lock] = "acquire-before release-after"
Hi, does it even make sense to do an acquire release in two different
commands? To me this looks like a source for deadlocks.
Felix
>
> Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
> ---
> meta/classes-recipe/rootfs.bbclass | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
> index 8b502a50..dc4e702c 100644
> --- a/meta/classes-recipe/rootfs.bbclass
> +++ b/meta/classes-recipe/rootfs.bbclass
> @@ -427,13 +427,13 @@ python do_rootfs_install() {
> for cmd in cmds:
> progress_reporter.next_stage()
>
> - if (d.getVarFlag(cmd, 'isar-apt-lock') or "") == "acquire-before":
> + if "acquire-before" in (d.getVarFlag(cmd, 'isar-apt-lock') or ""):
> lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
> shared=True)
>
> bb.build.exec_func(cmd, d)
>
> - if (d.getVarFlag(cmd, 'isar-apt-lock') or "") == "release-after":
> + if "release-after" in (d.getVarFlag(cmd, 'isar-apt-lock') or ""):
> bb.utils.unlockfile(lock)
> progress_reporter.finish()
> finally:
> --
> 2.34.1
>
> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260610064812.4010511-2-amikan%40ilbers.de.
Anton Mikanovich
> On Wed, 2026-06-10 at 09:48 +0300, Anton Mikanovich wrote:
>> In some cases only one command from all the ROOTFS_INSTALL_COMMAND list
>> can require isar-apt locking. To allow this logic improve flag checking
>> to accept the list of codenames. Existing API and codenames
>> ("acquire-before" and "release-after") were not changed.
>>
>> So the usage of single task declaration now looks like:
>>
>> rootfs_install_cmd[isar-apt-lock] = "acquire-before release-after"
> Hi, does it even make sense to do an acquire release in two different
> commands? To me this looks like a source for deadlocks.
>
> Felix
This is done to be sure isar-apt will not changed between "apt-get update"
and "apt-get install" commands. This probably can be reordered and
optimized,
but it will be out of scope for the current patchset.
MOESSBAUER, Felix
Anton Mikanovich
MOESSBAUER, Felix
> downloading should be done. All the debs must be precached and already
> placed in /var/cache/apt/archives, so apt-get can reuse them.
>
> Currently any downloads in this command are really succeed because we
> still have "network" flag on for using sudo, which masks all the
> caching issues in previous rootfs install commands.
>
> To prevent such a cases add "--no-download" option to apt-get.
Hi, this at least breaks under rootless during
rootfs_install_pkgs_install on the sbom-chroot recipe.
I'm currently investigating this, but here is already a log showing the
issue. Could it be, that local packages (via file://) are never cached?
Best regards,
Felix
DEBUG: Executing python function sstate_task_prefunc
DEBUG: Python function sstate_task_prefunc finished
DEBUG: Executing python function do_rootfs_install
DEBUG: Executing shell function root_cleandirs
DEBUG: Shell function root_cleandirs finished
DEBUG: Executing shell function rootfs_prepare
DEBUG: Shell function rootfs_prepare finished
DEBUG: Executing shell function rootfs_configure_isar_apt
DEBUG: Shell function rootfs_configure_isar_apt finished
DEBUG: Executing shell function rootfs_configure_apt
DEBUG: Shell function rootfs_configure_apt finished
DEBUG: Executing shell function rootfs_disable_initrd_generation
DEBUG: Shell function rootfs_disable_initrd_generation finished
DEBUG: Executing python function rootfs_do_mounts
DEBUG: Python function rootfs_do_mounts finished
DEBUG: Executing shell function rootfs_install_pkgs_update
Ign:1 file:/isar-apt isar InRelease
Get:2 file:/isar-apt isar Release [5099 B]
Get:2 file:/isar-apt isar Release [5099 B]
Ign:3 file:/isar-apt isar Release.gpg
Get:4 file:/isar-apt isar/main Sources [2102 B]
Get:5 file:/isar-apt isar/main amd64 Packages [1512 B]
Reading package lists...
DEBUG: Shell function rootfs_install_pkgs_update finished
DEBUG: Executing shell function rootfs_install_resolvconf
DEBUG: Shell function rootfs_install_resolvconf finished
DEBUG: Executing shell function rootfs_import_package_cache
DEBUG: Shell function rootfs_import_package_cache finished
DEBUG: Executing shell function rootfs_install_pkgs_download
Reading package lists...
Building dependency tree...
Reading state information...
Starting pkgProblemResolver with broken count: 0
Starting 2 pkgProblemResolver with broken count: 0
Done
The following additional packages will be installed:
distro-info-data libexpat1 libffi8 libjs-jquery libjs-sphinxdoc
libjs-underscore libncursesw6 libpython3-stdlib libpython3.13-minimal
libpython3.13-stdlib libreadline8t64 libyaml-0-2 media-types netbase
python-apt-common python3 python3-apt python3-beartype python3-
boolean
python3-charset-normalizer python3-click python3-debian python3-
defusedxml
python3-license-expression python3-minimal python3-packageurl
python3-ply
python3-py-serializable python3-pyparsing python3-rdflib
python3-semantic-version python3-sortedcontainers python3-uritools
python3-xmltodict python3-yaml python3.13 python3.13-minimal
readline-common
Suggested packages:
python3-doc python3-tk python3-venv python-apt-doc python3-boolean-
doc
python-charset-normalizer-doc python-cyclonedx-lib-doc gpgv zstd
python-ply-doc python3-pkg-resources python-pyparsing-doc python-
rdflib-doc
python-semantic-version-doc python-sortedcontainers-doc python3.13-
venv
python3.13-doc binutils binfmt-support readline-doc
Recommended packages:
javascript-common libgpm2 lsb-release iso-codes python3-html5rdf
python3-lxml python3-networkx python3-orjson ca-certificates
The following NEW packages will be installed:
distro-info-data libexpat1 libffi8 libjs-jquery libjs-sphinxdoc
libjs-underscore libncursesw6 libpython3-stdlib libpython3.13-minimal
libpython3.13-stdlib libreadline8t64 libyaml-0-2 media-types netbase
python-apt-common python3 python3-apt python3-beartype python3-
boolean
python3-charset-normalizer python3-click python3-cyclonedx-lib
python3-debian python3-debsbom python3-defusedxml python3-license-
expression
python3-minimal python3-packageurl python3-ply python3-py-
serializable
python3-pyparsing python3-rdflib python3-semantic-version
python3-sortedcontainers python3-spdx-tools python3-uritools
python3-xmltodict python3-yaml python3.13 python3.13-minimal
readline-common
0 upgraded, 41 newly installed, 0 to remove and 0 not upgraded.
Need to get 512 kB/9561 kB of archives.
After this operation, 42.5 MB of additional disk space will be used.
Get:1 file:/isar-apt isar/main amd64 python3-debsbom all 0.8.1 [62.5
kB]
Get:2 file:/isar-apt isar/main amd64 python3-spdx-tools all 0.8.3 [79.4
kB]
Get:3 http://deb.debian.org/debian trixie/main amd64 distro-info-data
all 0.66+deb13u2 [6792 B]
Get:4 http://deb.debian.org/debian trixie/main amd64 python-apt-common
all 3.0.0 [68.2 kB]
Get:5 http://deb.debian.org/debian trixie/main amd64 python3-apt amd64
3.0.0 [164 kB]
Get:6 http://deb.debian.org/debian trixie/main amd64 python3-defusedxml
all 0.7.1-3 [43.4 kB]
Get:7 http://deb.debian.org/debian trixie/main amd64 python3-py-
serializable all 2.0.0-2 [29.1 kB]
Get:8 http://deb.debian.org/debian trixie/main amd64 python3-
sortedcontainers all 2.4.0-2 [31.9 kB]
Get:9 http://deb.debian.org/debian trixie/main amd64 python3-cyclonedx-
lib all 9.1.0-2 [168 kB]
Fetched 512 kB in 0s (5318 kB/s)
Download complete and in download only mode
DEBUG: Shell function rootfs_install_pkgs_download finished
DEBUG: Executing shell function rootfs_export_package_cache
DEBUG: Shell function rootfs_export_package_cache finished
DEBUG: Executing shell function rootfs_install_pkgs_isar_download
Starting pkgProblemResolver with broken count: 0
Starting 2 pkgProblemResolver with broken count: 0
Done
Err:1 file:/isar-apt isar/main amd64 python3-debsbom all 0.8.1
File not found - /isar-apt/pool/main/p/python3-debsbom/python3-
debsbom_0.8.1_all.deb (2: No such file or directory)
Err:2 file:/isar-apt isar/main amd64 python3-spdx-tools all 0.8.3
File not found - /isar-apt/pool/main/p/python3-spdx-tools/python3-
spdx-tools_0.8.3_all.deb (2: No such file or directory)
E: Failed to fetch file:/isar-apt/pool/main/p/python3-debsbom/python3-
debsbom_0.8.1_all.deb File not found - /isar-apt/pool/main/p/python3-
debsbom/python3-debsbom_0.8.1_all.deb (2: No such file or directory)
E: Failed to fetch file:/isar-apt/pool/main/p/python3-spdx-
tools/python3-spdx-tools_0.8.3_all.deb File not found - /isar-
apt/pool/main/p/python3-spdx-tools/python3-spdx-tools_0.8.3_all.deb (2:
No such file or directory)
WARNING: exit code 123 from a shell command.
DEBUG: Executing python function rootfs_do_umounts
DEBUG: Executing shell function rootfs_do_umounts_priv
DEBUG: Shell function rootfs_do_umounts_priv finished
DEBUG: Python function rootfs_do_umounts finished
DEBUG: Python function do_rootfs_install finished
>
> Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
> ---
> meta/classes-recipe/rootfs.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
> index 81384340..f4580421 100644
> --- a/meta/classes-recipe/rootfs.bbclass
> +++ b/meta/classes-recipe/rootfs.bbclass
> @@ -402,7 +402,7 @@ rootfs_install_pkgs_install[progress] = "custom:rootfs_progress.PkgsInstallProgr
> rootfs_install_pkgs_install[network] = "${TASK_USE_SUDO}"
> rootfs_install_pkgs_install() {
> sudo -E chroot "${ROOTFSDIR}" \
> - /usr/bin/apt-get ${ROOTFS_APT_ARGS} ${ROOTFS_PACKAGES}
> + /usr/bin/apt-get ${ROOTFS_APT_ARGS} --no-download ${ROOTFS_PACKAGES}
> }
>
> ROOTFS_INSTALL_COMMAND += "rootfs_restore_initrd_tooling"
> --
> 2.34.1
>
> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260610064812.4010511-4-amikan%40ilbers.de.
MOESSBAUER, Felix
wrote:
> On Wed, 2026-06-10 at 09:48 +0300, Anton Mikanovich wrote:
> > During the command rootfs_install_pkgs_install no actuall packages
> > downloading should be done. All the debs must be precached and already
> > placed in /var/cache/apt/archives, so apt-get can reuse them.
> >
> > Currently any downloads in this command are really succeed because we
> > still have "network" flag on for using sudo, which masks all the
> > caching issues in previous rootfs install commands.
> >
> > To prevent such a cases add "--no-download" option to apt-get.
>
> Hi, this at least breaks under rootless during
> rootfs_install_pkgs_install on the sbom-chroot recipe.
>
> I'm currently investigating this, but here is already a log showing the
> issue. Could it be, that local packages (via file://) are never cached?
Probably that's a rootless problem, as we explicitly have to mount the
isar-apt during rootfs_install_pkgs_isar_download. On non-rootless, we
still have the mount from a previous task.
Felix
> To view this discussion visit https://groups.google.com/d/msgid/isar-users/bf4b13309aefe0442e41a3ad76aed8b03ea8c046.camel%40siemens.com.
Anton Mikanovich
> Hi, this at least breaks under rootless during
> rootfs_install_pkgs_install on the sbom-chroot recipe.
>
> I'm currently investigating this, but here is already a log showing the
> issue. Could it be, that local packages (via file://) are never cached?
patchset about actually. And that's why
rootfs_install_pkgs_isar_download was
introduced (cache them manually).