[PATCH v2 1/5] dpkg-base: Fix enabling of cross build profile

22 views
Skip to first unread message

Jan Kiszka

unread,
May 15, 2024, 7:08:24 AM5/15/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
From: Jan Kiszka <jan.k...@siemens.com>

ISAR_CROSS_COMPILE might be set even if we are not cross-building a
package. Use an output of crossbuild.bbclass, BUILD_ARCH, to find out
if we are actually cross-building a package.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/classes/dpkg-base.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 30caedf9..93321976 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -215,7 +215,7 @@ dpkg_runbuild() {

def isar_deb_build_profiles(d):
deb_build_profiles = d.getVar('DEB_BUILD_PROFILES')
- if bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')):
+ if d.getVar('BUILD_ARCH') != d.getVar('DISTRO_ARCH'):
deb_build_profiles += ' cross'
return deb_build_profiles.strip()

--
2.35.3

Jan Kiszka

unread,
May 15, 2024, 7:08:24 AM5/15/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
This depends on [1] and replaces [2].

Jan

[1] https://patchwork.isar-build.org/project/isar/list/?series=1184
[2] https://patchwork.isar-build.org/project/isar/list/?series=1171

Jan Kiszka (5):
dpkg-base: Fix enabling of cross build profile
linux-custom: Model cross-built kbuild package separately
linux-custom: Resolve native source package differences
dpkg: Retrieve Debian source name from variable
dpkg-source: Build source package only once

meta/classes/dpkg-base.bbclass | 2 +-
meta/classes/dpkg-source.bbclass | 60 ++++++++++++++++---
meta/classes/dpkg.bbclass | 3 +-
.../linux/files/debian/control.tmpl | 12 +++-
.../linux/files/debian/isar/common.tmpl | 2 +
.../linux/files/debian/isar/install.tmpl | 6 +-
meta/recipes-kernel/linux/linux-custom.inc | 8 +--
7 files changed, 77 insertions(+), 16 deletions(-)

--
2.35.3

Jan Kiszka

unread,
May 15, 2024, 7:08:25 AM5/15/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
From: Jan Kiszka <jan.k...@siemens.com>

Rename to kbuild package containing tools to cross-build modules
linux-kbuild-<name>-<arch>-cross. The headers package now depends on
either the native kbuild or a cross variant, unconditionally. This
allows to have unstable source packages between native and cross and is
generally cleaner to describe the cross version.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/recipes-kernel/linux/files/debian/control.tmpl | 12 +++++++++++-
.../linux/files/debian/isar/common.tmpl | 2 ++
.../linux/files/debian/isar/install.tmpl | 6 +++++-
meta/recipes-kernel/linux/linux-custom.inc | 5 +----
4 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/meta/recipes-kernel/linux/files/debian/control.tmpl b/meta/recipes-kernel/linux/files/debian/control.tmpl
index 6f8f8afe..e0e932a0 100644
--- a/meta/recipes-kernel/linux/files/debian/control.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/control.tmpl
@@ -51,9 +51,19 @@ Description: Linux kernel debugging symbols for @KR@
all the necessary debug symbols for the kernel and its modules.

Package: linux-kbuild-${KERNEL_NAME_PROVIDED}
-Build-Profiles: <kbuild>
+Build-Profiles: <kbuild !cross>
Architecture: any
Depends: ${perl:Depends}, ${shlib:Depends}
Description: ${KERNEL_NAME_PROVIDED} Linux kbuild scripts and tools for @KR@
This package provides kernel kbuild scripts and tools for @KR@
This is useful for people who need to build external modules
+
+Package: linux-kbuild-${KERNEL_NAME_PROVIDED}-${DISTRO_ARCH}-cross
+Build-Profiles: <kbuild cross>
+Architecture: any
+Multi-Arch: foreign
+Depends: ${perl:Depends}, ${shlib:Depends}
+Conflicts: linux-kbuild-${KERNEL_NAME_PROVIDED}
+Description: ${KERNEL_NAME_PROVIDED} Linux kbuild scripts and tools for @KR@
+ This package provides kernel kbuild scripts and tools for @KR@
+ This is useful for people who need to build external modules
diff --git a/meta/recipes-kernel/linux/files/debian/isar/common.tmpl b/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
index e3a1d8a0..86af70c5 100644
--- a/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
@@ -11,6 +11,7 @@ KERNEL_PKG_KERN_HEADERS=linux-headers-${KERNEL_NAME_PROVIDED}
KERNEL_PKG_LIBC_HEADERS=linux-libc-dev
KERNEL_PKG_LIBC_HEADERS_CROSS=linux-libc-dev-${DISTRO_ARCH}-cross
KERNEL_PKG_KERN_KBUILD=linux-kbuild-${KERNEL_NAME_PROVIDED}
+KERNEL_PKG_KERN_KBUILD_CROSS=${KERNEL_PKG_KERN_KBUILD}-${DISTRO_ARCH}-cross

# Force creating debian package with valid host arch for -native build
# Use a cross build to comply with arch specific kernel defconfigs
@@ -32,6 +33,7 @@ deb_kern_hdr_dir=${deb_top_dir}/${KERNEL_PKG_KERN_HEADERS}
deb_libc_hdr_dir=${deb_top_dir}/${KERNEL_PKG_LIBC_HEADERS}
deb_libc_hdr_cross_dir=${deb_top_dir}/${KERNEL_PKG_LIBC_HEADERS_CROSS}
deb_kern_kbuild_dir=${deb_top_dir}/${KERNEL_PKG_KERN_KBUILD}
+deb_kern_kbuild_cross_dir=${deb_top_dir}/${KERNEL_PKG_KERN_KBUILD_CROSS}

# Array of packages to be generated
declare -A kern_pkgs
diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
index 77856aee..00011517 100644
--- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
@@ -35,7 +35,11 @@ do_install() {

if echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then
# Install kernel scripts and tools
- install_kbuild ${deb_kern_kbuild_dir}
+ if echo "${DEB_BUILD_PROFILES}" | grep -q "cross"; then
+ install_kbuild ${deb_kern_kbuild_cross_dir}
+ else
+ install_kbuild ${deb_kern_kbuild_dir}
+ fi
fi

if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 3fd0c6c2..c54c287f 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -111,9 +111,6 @@ BBCLASSEXTEND:append:cross-profile = " kbuildtarget"
# build only kernel with the default variant of the recipe
BUILD_PROFILES:cross-profile = "kernel"

-# Select correct kbuild package for isar cross-build
-HEADERS_DEPENDS:cross-profile = ", linux-kbuild-${KERNEL_NAME_PROVIDED}:${HOST_ARCH} | linux-kbuild-${KERNEL_NAME_PROVIDED}"
-
# -native: kbuild package for host
BUILD_PROFILES:class-native = "kbuild"
RECIPE_PROVIDES:class-native = " \
@@ -151,7 +148,7 @@ RECIPE_PROVIDES:remove:cross-profile = " \
linux-kbuild-${KERNEL_NAME_PROVIDED}"

# Append headers depends
-HEADERS_DEPENDS = ", linux-kbuild-${KERNEL_NAME_PROVIDED}"
+HEADERS_DEPENDS = ", linux-kbuild-${KERNEL_NAME_PROVIDED} | linux-kbuild-${KERNEL_NAME_PROVIDED}-${DISTRO_ARCH}-cross"
KERNEL_HEADERS_DEBIAN_DEPENDS:append = "${HEADERS_DEPENDS}"

# Append provides
--
2.35.3

Jan Kiszka

unread,
May 15, 2024, 7:08:26 AM5/15/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
From: Jan Kiszka <jan.k...@siemens.com>

Avoid building the source package multiple times, possibly even
inconsistently. This is achieved by delegating this task to to the base
package and installing the source package from isar-apt in the native
and compat package variants.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/classes/dpkg-source.bbclass | 44 ++++++++++++++++++++++++++++++--
1 file changed, 42 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
index 560f536b..6a36081b 100644
--- a/meta/classes/dpkg-source.bbclass
+++ b/meta/classes/dpkg-source.bbclass
@@ -18,7 +18,7 @@ do_dpkg_source() {
find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete
sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
}
-addtask dpkg_source after do_prepare_build before do_dpkg_build
+addtask dpkg_source after do_prepare_build

do_deploy_source[depends] += "isar-apt:do_cache_config"
do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
@@ -34,4 +34,44 @@ do_deploy_source() {
"${DSC_FILE}"
fi
}
-addtask deploy_source after do_dpkg_source before do_dpkg_build
+addtask deploy_source after do_dpkg_source
+
+do_dpkg_build[depends] += "${BPN}:do_deploy_source"
+
+SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt"
+
+do_fetch_common_source[depends] += "${SCHROOT_DEP} ${BPN}:do_deploy_source"
+do_fetch_common_source[network] = "${TASK_USE_SUDO}"
+do_fetch_common_source() {
+ schroot_create_configs
+ insert_mounts
+
+ session_id=$(schroot -q -b -c ${SBUILD_CHROOT})
+ echo "Started session: ${session_id}"
+
+ schroot_cleanup() {
+ schroot -q -f -e -c ${session_id} > /dev/null 2>&1
+ remove_mounts > /dev/null 2>&1
+ schroot_delete_configs
+ }
+ trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+ trap 'schroot_cleanup' EXIT
+
+ schroot -r -c ${session_id} -d / -u root -- \
+ apt-get update -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" -o Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"
+ schroot -r -c ${session_id} -d / -- \
+ sh -c '
+ cd /work
+ apt-get -y --download-only --only-source -o Acquire::Source-Symlinks="false" source ${DEBIAN_SOURCE}'
+
+ schroot -e -c ${session_id}
+ remove_mounts
+ schroot_delete_configs
+}
+addtask fetch_common_source after do_unpack
+
+def is_native_or_compat(d):
+ overrides = d.getVar('OVERRIDES').split(':')
+ return 'class-native' in overrides or 'class-compat' in overrides
+
+do_dpkg_build[depends] += "${@'${PN}:do_fetch_common_source' if is_native_or_compat(d) else ''}"
--
2.35.3

Jan Kiszka

unread,
May 15, 2024, 7:08:27 AM5/15/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
From: Jan Kiszka <jan.k...@siemens.com>

Rather than reading this multiple times back from the changelog, define
DEBIAN_SOURCE, by default BPN, and use that consistently. This will
allow to fetch sources built by the base recipe in native and compat
recipes later on without generating the changelog first.

It also permits to clean up related code. Specifically do_deploy_source
was expecting more than one source file (which was incorrect) and looked
in the wrong folder.

The equivalence between DEBIAN_SOURCE and the actual source field is
checked in do_dpkg_source and enforced by failing the build otherwise.

At this chance, also move global -maxdepth before -name in find
statements.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/classes/dpkg-source.bbclass | 16 +++++++++++-----
meta/classes/dpkg.bbclass | 3 +--
2 files changed, 12 insertions(+), 7 deletions(-)

diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
index 7fd5d2ed..560f536b 100644
--- a/meta/classes/dpkg-source.bbclass
+++ b/meta/classes/dpkg-source.bbclass
@@ -7,10 +7,15 @@ inherit dpkg-base

DPKG_SOURCE_EXTRA_ARGS ?= "-I"

+DEBIAN_SOURCE ?= "${BPN}"
+
do_dpkg_source() {
# Create a .dsc file from source directory to use it with sbuild
DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
- find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
+ if [ "${DEB_SOURCE_NAME}" != "${DEBIAN_SOURCE}" ]; then
+ bbfatal "DEBIAN_SOURCE (${DEBIAN_SOURCE}) not aligned with source name used in control files (${DEB_SOURCE_NAME})"
+ fi
+ find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete
sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
}
addtask dpkg_source after do_prepare_build before do_dpkg_build
@@ -20,12 +25,13 @@ do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
do_deploy_source[dirs] = "${S}"
do_deploy_source() {
repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
- "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
- find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
+ "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}"
+ DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc")
+ if [ -n "${DSC_FILE}" ]; then
repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
"${DEBDISTRONAME}" \
- "${package}"
- done
+ "${DSC_FILE}"
+ fi
}
addtask deploy_source after do_dpkg_source before do_dpkg_build
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 3fa9f604..3cc36c12 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -108,8 +108,7 @@ dpkg_runbuild() {
echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG}

- DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
- DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print)
+ DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)

sbuild -A -n -c ${SBUILD_CHROOT} \
--host=${PACKAGE_ARCH} --build=${BUILD_ARCH} ${profiles} \
--
2.35.3

Jan Kiszka

unread,
May 15, 2024, 7:08:28 AM5/15/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
From: Jan Kiszka <jan.k...@siemens.com>

When building the native package, KERNEL_FILE as defined in bitbake.conf
is wrong as it uses the package arch. Avoid that by redefining
KERNEL_FILE in the kernel recipe using KERNEL_ARCH.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/recipes-kernel/linux/linux-custom.inc | 3 +++
1 file changed, 3 insertions(+)

diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index c54c287f..b53d136c 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -175,6 +175,9 @@ def get_kernel_arch(d):

KERNEL_ARCH ??= "${@get_kernel_arch(d)}"

+# set KERNEL_FILE without depending on package arch used in bitbake.conf
+KERNEL_FILE:forcevariable = "${@ 'vmlinux' if d.getVar('KERNEL_ARCH') in ['mipsel', 'riscv', 'arm64'] else 'vmlinuz'}"
+
def config_fragments(d):
fragments = []
sources = d.getVar("SRC_URI").split()
--
2.35.3

Koch, Stefan

unread,
May 15, 2024, 9:47:52 AM5/15/24
to isar-...@googlegroups.com, Kiszka, Jan, Schmidt, Adriaan, ami...@ilbers.de, MOESSBAUER, Felix
Since applying this series, I get this build error:

ERROR: linux-audis-kbuildtarget-5.10.209-audis-rtp-r0 do_dpkg_build:
ExecutionError('/home/z003jsce/kbuildboards/build/tmp/work/audis-
bullseye-latest-arm64/linux-audis-kbuildtarget/5.10.209-audis-rtp-
r0/temp/run.dpkg_runbuild.77935', 1, None, None)
ERROR: Logfile of failure stored in:
/home/z003jsce/kbuildboards/build/tmp/work/audis-bullseye-latest-
arm64/linux-audis-kbuildtarget/5.10.209-audis-rtp-
r0/temp/log.do_dpkg_build.77935
Log data follows:
| DEBUG: Executing python function sstate_task_prefunc
| DEBUG: Python function sstate_task_prefunc finished
| DEBUG: Executing python function do_dpkg_build
| DEBUG: Executing shell function dpkg_schroot_create_configs
| Creating /etc/schroot/chroot.d/isar-builder-cbda93bb-87d8-4b8d-ae73-
2495897f4aaf-77935
| DEBUG: Shell function dpkg_schroot_create_configs finished
| DEBUG: Executing shell function dpkg_runbuild
| WARNING: Export of 'CCACHE_UMASK=000' detected, please migrate to
templates
| dpkg-parsechangelog: error: cannot open file
/home/z003jsce/kbuildboards/build/debian/changelog: No such file or
directory
| E: Could not parse
/home/z003jsce/kbuildboards/build/debian/changelog:
| WARNING: exit code 1 from a shell command.
| DEBUG: Executing shell function schroot_delete_configs
| Removing /etc/schroot/isar-builder-cbda93bb-87d8-4b8d-ae73-
2495897f4aaf-77935
| Removing /etc/schroot/chroot.d/isar-builder-cbda93bb-87d8-4b8d-ae73-
2495897f4aaf-77935
| DEBUG: Shell function schroot_delete_configs finished
| DEBUG: Python function do_dpkg_build finished
ERROR: Task
(virtual:kbuildtarget:/home/z003jsce/kbuildboards/build/../audis-
isar/recipes-kernel/linux/linux-audis.bb:do_dpkg_build) failed with
exit code '1'
NOTE: Tasks Summary: Attempted 97 tasks of which 0 didn't need to be
rerun and 1 failed.

Stefan
Stefan Koch
Siemens AG
www.siemens.com

Jan Kiszka

unread,
May 15, 2024, 1:40:26 PM5/15/24
to Koch, Stefan (DI PA DCP R&D 3), isar-...@googlegroups.com, Schmidt, Adriaan (T CED EDC-DE), ami...@ilbers.de, Moessbauer, Felix (T CED OES-DE)
Can you find a reproduction path with upstream? Or identify the required
modification for it to trigger this? I suspect some downstream kernel
recipe bits play an central role because we no longer run
dpkg-parsechangelog in upstream do_dpkg_build.

Jan

--
Siemens AG, Technology
Linux Expert Center

Schmidt, Adriaan

unread,
May 16, 2024, 2:06:16 AM5/16/24
to Kiszka, Jan, isar-users, MOESSBAUER, Felix, Koch, Stefan, Anton Mikanovich
Kiszka, Jan (T CED), Mittwoch, 15. Mai 2024 13:08:
> ISAR_CROSS_COMPILE might be set even if we are not cross-building a
> package. Use an output of crossbuild.bbclass, BUILD_ARCH, to find out
> if we are actually cross-building a package.
>
> Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
> ---
> meta/classes/dpkg-base.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 30caedf9..93321976 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -215,7 +215,7 @@ dpkg_runbuild() {
>
> def isar_deb_build_profiles(d):
> deb_build_profiles = d.getVar('DEB_BUILD_PROFILES')
> - if bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')):
> + if d.getVar('BUILD_ARCH') != d.getVar('DISTRO_ARCH'):

That's the question "are we _actually_ cross-compiling"?

Elsewhere (multiarch.bbclass, linux-custom.inc) we have that
check implemented as "ISAR_CROSS_COMPILE && DISTRO_ARCH != HOST_ARCH".

As a follow-up we should look into unifying this.
Maybe have crossvars.bbclass figure out how we're building, and set
one variable to be consumed by recipes, e.g.

ISAR_BUILDPROFILE="native|emulated|cross"

Or even add an override (:buildprofile-native, ...) to make it easier to use?

Adriaan

Jan Kiszka

unread,
May 16, 2024, 2:15:32 AM5/16/24
to Schmidt, Adriaan (T CED EDC-DE), isar-users, Moessbauer, Felix (T CED OES-DE), Koch, Stefan (DI PA DCP R&D 3), Anton Mikanovich
I don't disagree.

Jan Kiszka

unread,
May 16, 2024, 12:37:32 PM5/16/24
to Koch, Stefan (DI PA DCP R&D 3), isar-...@googlegroups.com, Schmidt, Adriaan (T CED EDC-DE), ami...@ilbers.de, Moessbauer, Felix (T CED OES-DE)
Were you able to clarify what is needed to trigger this? As this series
fixes the remaining issues of the kbuild changes (to my best knowledge),
it would be good to have them ready ASAP.

Koch, Stefan

unread,
May 17, 2024, 5:03:38 AM5/17/24
to isar-...@googlegroups.com, Kiszka, Jan, Schmidt, Adriaan, ami...@ilbers.de, MOESSBAUER, Felix
The issue seems to come only with commit:
dpkg-source: Build source package only once

But it still needs deeper investigation.


> Jan

Jan Kiszka

unread,
May 17, 2024, 8:21:32 AM5/17/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
Looks like I lost a good share of my related dependency improvements,
and some appear to be actually needed to avoid regressions. Just found
them again in stash, v3 will follow.

FWIW, they do NOT explain the issue Stefan reported. I tested against
isar-cip-core updated with this, and it works perfectly fine.

Jan Kiszka

unread,
May 17, 2024, 12:35:45 PM5/17/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
From: Jan Kiszka <jan.k...@siemens.com>

ISAR_CROSS_COMPILE might be set even if we are not cross-building a
package. Use an output of crossbuild.bbclass, BUILD_ARCH, to find out
if we are actually cross-building a package.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/classes/dpkg-base.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 30caedf9..93321976 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -215,7 +215,7 @@ dpkg_runbuild() {

def isar_deb_build_profiles(d):
deb_build_profiles = d.getVar('DEB_BUILD_PROFILES')
- if bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')):
+ if d.getVar('BUILD_ARCH') != d.getVar('DISTRO_ARCH'):

Jan Kiszka

unread,
May 17, 2024, 12:35:46 PM5/17/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
From: Jan Kiszka <jan.k...@siemens.com>

Rename to kbuild package containing tools to cross-build modules
linux-kbuild-<name>-<arch>-cross. The headers package now depends on
either the native kbuild or a cross variant, unconditionally. This
allows to have unstable source packages between native and cross and is
generally cleaner to describe the cross version.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 3fd0c6c2..c54c287f 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc

Jan Kiszka

unread,
May 17, 2024, 12:35:46 PM5/17/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
From: Jan Kiszka <jan.k...@siemens.com>

Rather than reading this multiple times back from the changelog, define
DEBIAN_SOURCE, by default BPN, and use that consistently. This will
allow to fetch sources built by the base recipe in native and compat
recipes later on without generating the changelog first.

It also permits to clean up related code. Specifically do_deploy_source
was expecting more than one source file (which was incorrect) and looked
in the wrong folder.

The equivalence between DEBIAN_SOURCE and the actual source field is
checked in do_dpkg_source and enforced by failing the build otherwise.

At this chance, also move global -maxdepth before -name in find
statements.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/classes/dpkg-source.bbclass | 16 +++++++++++-----
meta/classes/dpkg.bbclass | 3 +--
2 files changed, 12 insertions(+), 7 deletions(-)

diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
index 7fd5d2ed..560f536b 100644
--- a/meta/classes/dpkg-source.bbclass
+++ b/meta/classes/dpkg-source.bbclass
@@ -7,10 +7,15 @@ inherit dpkg-base

DPKG_SOURCE_EXTRA_ARGS ?= "-I"

+DEBIAN_SOURCE ?= "${BPN}"
+
do_dpkg_source() {
# Create a .dsc file from source directory to use it with sbuild
DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
- find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
+ if [ "${DEB_SOURCE_NAME}" != "${DEBIAN_SOURCE}" ]; then
+ bbfatal "DEBIAN_SOURCE (${DEBIAN_SOURCE}) not aligned with source name used in control files (${DEB_SOURCE_NAME})"
+ fi
+ find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete
sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
}
addtask dpkg_source after do_prepare_build before do_dpkg_build
@@ -20,12 +25,13 @@ do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
do_deploy_source[dirs] = "${S}"
do_deploy_source() {
repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
- "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
- find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
+ "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}"
+ DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc")
+ if [ -n "${DSC_FILE}" ]; then
repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
"${DEBDISTRONAME}" \
- "${package}"
- done
+ "${DSC_FILE}"
+ fi
}
addtask deploy_source after do_dpkg_source before do_dpkg_build
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 0e25eb76..804fe5a6 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -109,8 +109,7 @@ dpkg_runbuild() {

Jan Kiszka

unread,
May 17, 2024, 12:35:46 PM5/17/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
From: Jan Kiszka <jan.k...@siemens.com>

When building the native package, KERNEL_FILE as defined in bitbake.conf
is wrong as it uses the package arch. Avoid that by redefining
KERNEL_FILE in the kernel recipe using KERNEL_ARCH.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/recipes-kernel/linux/linux-custom.inc | 3 +++
1 file changed, 3 insertions(+)

diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index c54c287f..b53d136c 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc

Jan Kiszka

unread,
May 17, 2024, 12:35:46 PM5/17/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
This depends on [1] and [2] and replaces [3].

Changes in v3:
- avoid anything but source package fetching for non-base targets
- fix regression of kbuildtarget

Jan

[1] https://patchwork.isar-build.org/project/isar/list/?series=1184
[2] https://patchwork.isar-build.org/project/isar/list/?series=1188
[3] https://patchwork.isar-build.org/project/isar/list/?series=1171

Jan Kiszka (5):
dpkg-base: Fix enabling of cross build profile
linux-custom: Model cross-built kbuild package separately
linux-custom: Resolve native source package differences
dpkg: Retrieve Debian source name from variable
dpkg-source: Build source package only once

meta/classes/dpkg-base.bbclass | 6 +-
meta/classes/dpkg-prebuilt.bbclass | 7 +--
meta/classes/dpkg-source.bbclass | 56 ++++++++++++++++---
meta/classes/dpkg.bbclass | 6 +-
.../linux/files/debian/control.tmpl | 12 +++-
.../linux/files/debian/isar/common.tmpl | 2 +
.../linux/files/debian/isar/install.tmpl | 6 +-
meta/recipes-kernel/linux/linux-custom.inc | 8 +--
8 files changed, 79 insertions(+), 24 deletions(-)

--
2.35.3

Jan Kiszka

unread,
May 17, 2024, 12:35:47 PM5/17/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
From: Jan Kiszka <jan.k...@siemens.com>

Avoid building the source package multiple times, possibly even
inconsistently. This is achieved by delegating this task to to the base
package and installing the source package from isar-apt in the native
and compat package variants. Those derived packages will also no longer
trigger fetching, unpacking or prepare_build as everything is supposed
to be in the source package.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/classes/dpkg-base.bbclass | 4 +--
meta/classes/dpkg-prebuilt.bbclass | 7 ++----
meta/classes/dpkg-source.bbclass | 40 ++++++++++++++++++++++++++++--
meta/classes/dpkg.bbclass | 3 ++-
4 files changed, 44 insertions(+), 10 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 93321976..789d6c74 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -78,7 +78,7 @@ addtask adjust_git after do_unpack before do_patch
do_adjust_git[lockfiles] += "${DL_DIR}/git/isar.lock"

inherit patch
-addtask patch after do_adjust_git before do_dpkg_build
+addtask patch after do_adjust_git

SRC_APT ?= ""

@@ -191,7 +191,7 @@ do_prepare_build() {
true
}

-addtask prepare_build after do_patch do_transform_template before do_dpkg_build
+addtask prepare_build after do_patch do_transform_template
# If Isar recipes depend on each other, they typically need the package
# deployed to isar-apt
do_local_isarapt[depends] += "isar-apt:do_cache_config"
diff --git a/meta/classes/dpkg-prebuilt.bbclass b/meta/classes/dpkg-prebuilt.bbclass
index 8135fc81..7092670b 100644
--- a/meta/classes/dpkg-prebuilt.bbclass
+++ b/meta/classes/dpkg-prebuilt.bbclass
@@ -16,11 +16,8 @@ python do_unpack:prepend() {
d.setVar('SRC_URI', ' '.join(src_uri))
}

-# break dependencies on do_patch, etc... but still support sstate caching
-deltask dpkg_build
-addtask dpkg_build after do_unpack before do_deploy_deb
-# break inherited (from dpkg-base) dependency on sbuild_chroot
-do_dpkg_build[depends] = ""
+# also breaks inherited (from dpkg-base) dependency on sbuild_chroot
+do_dpkg_build[depends] = "${PN}:do_unpack"
do_dpkg_build() {
true
}
diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
index 560f536b..d6ab5aad 100644
--- a/meta/classes/dpkg-source.bbclass
+++ b/meta/classes/dpkg-source.bbclass
@@ -18,7 +18,7 @@ do_dpkg_source() {
find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete
sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
}
-addtask dpkg_source after do_prepare_build before do_dpkg_build
+addtask dpkg_source after do_prepare_build

do_deploy_source[depends] += "isar-apt:do_cache_config"
do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
@@ -34,4 +34,40 @@ do_deploy_source() {
+
+do_dpkg_build[depends] += "${@'${PN}:do_dpkg_source' if '${PN}' == '${BPN}' else '${PN}:do_fetch_common_source'}"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 804fe5a6..d92ff68c 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -24,9 +24,10 @@ def expand_sbuild_pt_additions(d):
cmds += 'sbuild_export ' + var + ' "' + varval + '"\n'
return cmds

-do_prepare_build:append() {
+do_get_reference_env() {
env > ${DPKG_PREBUILD_ENV_FILE}
}
+addtask get_reference_env before do_dpkg_build

# cp -n results in nonzero exit code starting from coreutils 9.2
# and starting from 9.3 we can use --update=none for the same behaviour
--
2.35.3

Jan Kiszka

unread,
May 17, 2024, 12:57:53 PM5/17/24
to isar-users, Felix Moessbauer, stefa...@siemens.com, Adriaan Schmidt, Anton Mikanovich
From: Jan Kiszka <jan.k...@siemens.com>

This ensures that kbuildtarget case is checked.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

I must admit that this test code is, well, untested.

testsuite/cibuilder.py | 1 +
1 file changed, 1 insertion(+)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 3e151566..ebe79468 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -143,6 +143,7 @@ class CIBuilder(Test):
f.write('IMAGE_INSTALL += "kselftest"\n')
if cross:
f.write('ISAR_CROSS_COMPILE = "1"\n')
+ f.write('IMAGE_INSTALL:append:hikey = " linux-headers-${KERNEL_NAME}"\n')
if debsrc_cache:
f.write('BASE_REPO_FEATURES = "cache-deb-src"\n')
if offline:
--
2.35.3

Koch, Stefan

unread,
May 21, 2024, 6:29:50 AM5/21/24
to isar-...@googlegroups.com, Kiszka, Jan, Schmidt, Adriaan, ami...@ilbers.de, MOESSBAUER, Felix
Successful compiled.

Thanks.

Stefan

Anton Mikanovich

unread,
May 21, 2024, 6:35:43 AM5/21/24
to Koch, Stefan, isar-...@googlegroups.com, Kiszka, Jan, Schmidt, Adriaan, MOESSBAUER, Felix
21/05/2024 13:29, Koch, Stefan wrote:
> Successful compiled.
>
> Thanks.
>
> Stefan

This one also is the first version passed fast CI.
Will try full also.

Jan Kiszka

unread,
May 21, 2024, 6:41:21 AM5/21/24
to Anton Mikanovich, Koch, Stefan, isar-...@googlegroups.com, Schmidt, Adriaan, MOESSBAUER, Felix
Great to hear!

We are running this over in isar-cip-core as well [1], though with less
coverage than full CI here. But there it is currently the last blocker
for going into a release.

Jan

[1]
https://gitlab.com/cip-project/cip-core/isar-cip-core/-/pipelines/1298866474

Jan Kiszka

unread,
May 27, 2024, 5:16:20 AM5/27/24
to Anton Mikanovich, Schmidt, Adriaan, Koch, Stefan, MOESSBAUER, Felix, isar-...@googlegroups.com
Any news?

Jan

Uladzimir Bely

unread,
May 27, 2024, 7:05:07 AM5/27/24
to Jan Kiszka, Anton Mikanovich, Schmidt, Adriaan, Koch, Stefan, MOESSBAUER, Felix, isar-...@googlegroups.com
Hello.

We have the patchet passed both fast and full CI, so it could be
merged.

--
Best regards,
Uladzimir.



Jan Kiszka

unread,
May 28, 2024, 1:30:30 AM5/28/24
to Uladzimir Bely, Anton Mikanovich, Schmidt, Adriaan, Koch, Stefan, MOESSBAUER, Felix, isar-...@googlegroups.com
And when do you plan to?

Uladzimir Bely

unread,
May 28, 2024, 1:39:01 AM5/28/24
to Koch, Stefan, isar-...@googlegroups.com, Kiszka, Jan
Applied to next, thanks.

--
Best regards,
Uladzimir.

MOESSBAUER, Felix

unread,
Aug 20, 2025, 6:52:19 AMAug 20
to isar-...@googlegroups.com, Kiszka, Jan, Schmidt, Adriaan, ami...@ilbers.de, cedric.h...@siemens.com, Bezdeka, Florian, Koch, Stefan
On Fri, 2024-05-17 at 18:35 +0200, Jan Kiszka wrote:
Hi, while debugging another issue related to DPKG_ARCH=all, I noticed
that the do_fetch_common_source task has a hidden dependency to having
the source package in isar apt. While this is modeled in bitbake (via
depends to ${BPN}:do_deploy_source), the source package is not there
anymore when running repo_del_package on the -native package.

This happens because the clean task on the -native package calls out to
repo_del_package, which also removes the source package. This deletion
remains hidden to bitbake, hence the ${PN}:do_dpkg_source is not
executed again and the do_fetch_common_source fails, as the source
package is not there anymore.

I'm wondering if there is a way to tell reprepro to not delete source
packages. This could be used to then only delete the source package in
the non-native case. An alternative solution would be to let the -
native clean task depend on the non-native clean task.

Felix

> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 804fe5a6..d92ff68c 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -24,9 +24,10 @@ def expand_sbuild_pt_additions(d):
>              cmds += 'sbuild_export ' + var + ' "' + varval + '"\n'
>      return cmds
>  
> -do_prepare_build:append() {
> +do_get_reference_env() {
>      env > ${DPKG_PREBUILD_ENV_FILE}
>  }
> +addtask get_reference_env before do_dpkg_build
>  
>  # cp -n results in nonzero exit code starting from coreutils 9.2
>  # and starting from 9.3 we can use --update=none for the same
> behaviour

--
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

Reply all
Reply to author
Forward
0 new messages