[PATCH 0/2] Protect isar-apt usages in shell

[Anton Mikanovich]

Read operations on isar-apt should be covered by shared lock.
This was not implemented for deb_dl_dir_export and imager_run.
At least following CI error was caused by isar-apt race:

DEBUG: Executing shell function rootfs_export_package_cache
find: 'isar-apt/debian-bullseye-arm64/apt/debian-bullseye/pool/main/e/enable-fsck': No such file or directory

Find call inside deb_dl_dir_export was trying to look into enable-fsck
directory while is was deleted by deb_clean function of do_deploy_deb
task of enable-fsck package.

The same situation can potentially happen on imager_run call, causing
races during image preparations.

To protect isar-apt usages inside big shell functions introduce
lockrun.py helper for using with isar-apt lock.
More clear implementation will require tasks rewrite in python, which
is not an option for the big downstream projects.

Anton Mikanovich (2):
deb-dl-dir: Avoid multiple find run on export
meta: Cover bash isar-apt usages by isar.lock

meta/classes/deb-dl-dir.bbclass | 9 +++--
meta/classes/image-tools-extension.bbclass | 2 +
scripts/lockrun.py | 44 ++++++++++++++++++++++
3 files changed, 52 insertions(+), 3 deletions(-)
create mode 100755 scripts/lockrun.py

--
2.34.1

[Anton Mikanovich]

We are running find on isar-apt repo for every new deb in cache.
This can be optimised by collecting isar-apt content once and compare
deb name with this list later.
It can be also usefull for the future isar-apt lock protection.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---
meta/classes/deb-dl-dir.bbclass | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 8e0243fe..2d8739dc 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -89,6 +89,9 @@ deb_dl_dir_export() {
export rootfs="${1}"
export owner=$(id -u):$(id -g)
mkdir -p "${pc}"
+
+ isar_debs="\$(find '${REPO_ISAR_DIR}/${DISTRO}' -name '*.deb' -print)"
+
flock "${pc}".lock sudo -Es << 'EOSUDO'
set -e
printenv | grep -q BB_VERBOSE_LOGS && set -x
@@ -98,9 +101,8 @@ deb_dl_dir_export() {
while read p; do
# skip files from a previous export
[ -f "${pc}/${p##*/}" ] && continue
- # can not reuse bitbake function here, this is basically
- # "repo_contains_package"
- package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/})
+ # skip packages from isar-apt
+ package=$(echo "$isar_debs" | grep -F -m 1 "${p##*/}" | cat)
if [ -n "$package" ]; then
cmp --silent "$package" "$p" && continue
fi
--
2.34.1

[Anton Mikanovich]

Cover isar-apt repo usages by ${REPO_ISAR_DIR}/isar.lock to prevent any
conflicts or races.

Introduce lockrun.py helper script to execute bash code covered by
fcntl locks in bitbake compatible way. It means the same lock files as
used in lockfiles task flag can be used with lockrun.py.
Key -r is used for shared locking on read operations.

Usage examples:

${SCRIPTSDIR}/lockrun.py -r -f "${REPO_ISAR_DIR}/isar.lock" -c "true"

${SCRIPTSDIR}/lockrun.py -r -f "${REPO_ISAR_DIR}/isar.lock" -s <<EOF
true
EOF

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta/classes/deb-dl-dir.bbclass | 3 +-

meta/classes/image-tools-extension.bbclass | 2 +
scripts/lockrun.py | 44 ++++++++++++++++++++++

3 files changed, 48 insertions(+), 1 deletion(-)
create mode 100755 scripts/lockrun.py

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 2d8739dc..55e56c50 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -90,7 +90,8 @@ deb_dl_dir_export() {

export owner=$(id -u):$(id -g)
mkdir -p "${pc}"

- isar_debs="\$(find '${REPO_ISAR_DIR}/${DISTRO}' -name '*.deb' -print)"
+ isar_debs="$(${SCRIPTSDIR}/lockrun.py -r -f '${REPO_ISAR_DIR}/isar.lock' -c \
+ "find '${REPO_ISAR_DIR}/${DISTRO}' -name '*.deb' -print")"

flock "${pc}".lock sudo -Es << 'EOSUDO'
set -e

diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 46bdf78b..e8ace8f5 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -46,6 +46,7 @@ imager_run() {

E="${@ isar_export_proxies(d)}"
deb_dl_dir_import ${schroot_dir} ${distro}
+ ${SCRIPTSDIR}/lockrun.py -r -f "${REPO_ISAR_DIR}/isar.lock" -s <<EOAPT
schroot -r -c ${session_id} -d / -u root -- sh -c " \
apt-get update \
-o Dir::Etc::SourceList='sources.list.d/isar-apt.list' \
@@ -54,6 +55,7 @@ imager_run() {
apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
--allow-unauthenticated --allow-downgrades --download-only install \
${local_install}"
+EOAPT

deb_dl_dir_export ${schroot_dir} ${distro}
schroot -r -c ${session_id} -d / -u root -- sh -c " \
diff --git a/scripts/lockrun.py b/scripts/lockrun.py
new file mode 100755
index 00000000..58a598dd
--- /dev/null
+++ b/scripts/lockrun.py
@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+#
+# Helper script to use bitbake locks in shell
+# Copyright (c) 2024, ilbers GmbH
+
+import argparse
+import os
+import subprocess
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib'))
+
+from bb import utils
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('-f', '--file', help='Lock file name.', required=True)
+ parser.add_argument(
+ '-r', '--read', action="store_true", help='Use read (shared) locking.'
+ )
+ group = parser.add_mutually_exclusive_group(required=True)
+ group.add_argument('-c', '--command', help='Command(s) to execute.')
+ group.add_argument(
+ '-s',
+ '--shell',
+ action="store_true",
+ help='Execute commands from stdin.',
+ )
+ return parser.parse_args()
+
+
+if __name__ == "__main__":
+ args = parse_args()
+
+ with utils.fileslocked([args.file], shared=args.read):
+ if args.shell:
+ cmd = sys.stdin.read()
+ else:
+ cmd = args.command
+ try:
+ subprocess.run(cmd, check=True, shell=True)
+ except subprocess.CalledProcessError as e:
+ exit(e.returncode)
--
2.34.1

[Uladzimir Bely]

Applied to next.

--
Best regards,
Uladzimir.