[PATCH 0/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

[Cedric Hombourger]

With sources being built only once, do_prepare_build (and its dependencies)
are skipped when building -compat or -native packages. This unfortunately
means that do_install is not executed and is causing the package build
to fail as the install step is not finding any files in ${PP}/image. To
avoid changing how do_install is scheduled in the normal case, it is simply
added as an extra dependency to do_dpkg_build to make sure it gets executed
prior to the build of the binary package.

This patch depends on https://groups.google.com/g/isar-users/c/Ys54mk6_bRM

Cedric Hombourger (1):
dpkg-raw: make sure do_install gets called for -compat/-native builds

meta/classes/dpkg-raw.bbclass | 6 ++++++
1 file changed, 6 insertions(+)

--
2.39.2

[Cedric Hombourger]

With sources being built only once, do_prepare_build (and its dependencies)
are skipped when building -compat or -native packages. This unfortunately
means that do_install is not executed and is causing the package build
to fail as the install step is not finding any files in ${PP}/image. To
avoid changing how do_install is scheduled in the normal case, it is simply
added as an extra dependency to do_dpkg_build to make sure it gets executed
prior to the build of the binary package.

Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---

meta/classes/dpkg-raw.bbclass | 6 ++++++
1 file changed, 6 insertions(+)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index dd7b761f..94aba1aa 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -15,6 +15,12 @@ do_install() {
do_install[cleandirs] = "${D}"
addtask install after do_patch do_transform_template before do_prepare_build

+# make sure do_install gets executed when building -compat / -native packages
+# are built since do_prepare_build would be skipped for those (sources are
+# built only once). To avoid changing how do_install is scheduled in the normal
+# case, it is simply added as an extra dependency to do_dpkg_build
+do_dpkg_build[depends] += "${PN}:do_install"
+
do_prepare_build[cleandirs] += "${S}/debian"
do_prepare_build() {
cd ${D}
--
2.39.2

[Jan Kiszka]

On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote:
> With sources being built only once, do_prepare_build (and its dependencies)
> are skipped when building -compat or -native packages. This unfortunately
> means that do_install is not executed and is causing the package build
> to fail as the install step is not finding any files in ${PP}/image. To
> avoid changing how do_install is scheduled in the normal case, it is simply
> added as an extra dependency to do_dpkg_build to make sure it gets executed
> prior to the build of the binary package.

What is the use case for raw-compat and -native? Are you actually having
a case with different binaries then?

>
> Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
> ---
> meta/classes/dpkg-raw.bbclass | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
> index dd7b761f..94aba1aa 100644
> --- a/meta/classes/dpkg-raw.bbclass
> +++ b/meta/classes/dpkg-raw.bbclass
> @@ -15,6 +15,12 @@ do_install() {
> do_install[cleandirs] = "${D}"
> addtask install after do_patch do_transform_template before do_prepare_build
>
> +# make sure do_install gets executed when building -compat / -native packages
> +# are built since do_prepare_build would be skipped for those (sources are
> +# built only once). To avoid changing how do_install is scheduled in the normal
> +# case, it is simply added as an extra dependency to do_dpkg_build
> +do_dpkg_build[depends] += "${PN}:do_install"
> +

So, this class would then violate the rule of shared, identical "source"
packages? Does not feel correct.

> do_prepare_build[cleandirs] += "${S}/debian"
> do_prepare_build() {
> cd ${D}

Jan

--
Siemens AG, Technology
Linux Expert Center

[cedric.h...@siemens.com]

On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote:
> On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote:
> > With sources being built only once, do_prepare_build (and its
> > dependencies)
> > are skipped when building -compat or -native packages. This
> > unfortunately
> > means that do_install is not executed and is causing the package
> > build
> > to fail as the install step is not finding any files in
> > ${PP}/image. To
> > avoid changing how do_install is scheduled in the normal case, it
> > is simply
> > added as an extra dependency to do_dpkg_build to make sure it gets
> > executed
> > prior to the build of the binary package.
>
> What is the use case for raw-compat and -native? Are you actually
> having
> a case with different binaries then?

For my specific case, I had wished -native would not be built because
my dpkg-raw package has DPKG_ARCH set to "all". In that case, we should
not have a separate build for -native

The issue will occur for any "dpkg" package being built for -native
when it DEPENDS on a "dpkg-raw" package because our native class
automatically changes all dependencies to -native

While most "dpkg-raw" packages should have DPKG_ARCH = "all", many
recipes did not bother setting this (in most downstream layers that I
have seen) and will end up with different binary packages in their -
native builds when HOST_ARCH != DISTRO_ARCH

To reproduce the issue that I have seen (and which should be fixed
IMO):

1. bitbake mc:qemuarm64-bookworm:example-raw
2. bitbake mc:qemuarm64-bookworm:example-raw-native

PS: I am planning to investigate how to suppress the duplicate build of
"dpkg-raw" package that has DPKG_ARCH set to "all" when a dependent
"dpkg" package is built for both variants

I believe the issue demonstrated with the above scenario needs fixing
(the error message you otherwise get as a user is rather cryptic)

No same sources are used between non-native and -native. See below

Tasks for example-raw:

$ cat build-test-raw-native/tmp/work/debian-bookworm-arm64/example-
raw/0.3-r0/temp/log.task_order
20240912-143135.942912 do_fetch (171631): log.do_fetch.171631
20240912-143135.967641 do_get_reference_env (171636):
log.do_get_reference_env.171636
20240912-143144.778512 do_unpack (171697): log.do_unpack.171697
20240912-143145.280183 do_adjust_git (171767): log.do_adjust_git.171767
20240912-143145.369652 do_transform_template (171780):
log.do_transform_template.171780
20240912-143145.477501 do_patch (171793): log.do_patch.171793
20240912-143145.964197 do_install (171849): log.do_install.171849
20240912-143146.108066 do_prepare_build (171883):
log.do_prepare_build.171883
20240912-143146.247279 do_dpkg_source (172015):
log.do_dpkg_source.172015
20240912-143146.303464 do_local_isarapt (172169):
log.do_local_isarapt.172169
20240912-143146.517836 do_deploy_source (172513):
log.do_deploy_source.172513
20240912-143457.684464 do_dpkg_build (195129): log.do_dpkg_build.195129
20240912-143527.279878 do_deploy_deb (197150): log.do_deploy_deb.197150


Tasks for example-raw-native (with my changes):

$ cat build-test-raw-native/tmp/work/debian-bookworm-arm64/example-raw-
native/0.3-r0/temp/log.task_order
20240912-143632.541485 do_fetch (197337): log.do_fetch.197337
20240912-143632.548091 do_get_reference_env (197338):
log.do_get_reference_env.197338
20240912-143632.555547 do_local_isarapt (197341):
log.do_local_isarapt.197341
20240912-143639.819570 do_unpack (197499): log.do_unpack.197499
20240912-143640.047376 do_adjust_git (197743): log.do_adjust_git.197743
20240912-143640.160769 do_transform_template (198038):
log.do_transform_template.198038
20240912-143640.267071 do_patch (198156): log.do_patch.198156
20240912-143640.593967 do_install (198222): log.do_install.198222
20240912-143742.934586 do_fetch_common_source (210951):
log.do_fetch_common_source.210951
20240912-143743.720108 do_dpkg_build (211215): log.do_dpkg_build.211215
20240912-143748.362105 do_deploy_deb (212659): log.do_deploy_deb.212659

Without my changes, do_install would not be executed and do_dpkg_build
would fail miserably.

run.do_dpkg_build will find the source package (more specifically the
.dsc file) in the work tree and use it

I therefore believe that identical sources are used between -native and
non-native

Am I missing something?

>
> >  do_prepare_build[cleandirs] += "${S}/debian"
> >  do_prepare_build() {
> >         cd ${D}
>
> Jan
>

--

Cedric Hombourger
Siemens AG
www.siemens.com

[Jan Kiszka]

Then stub-out the other variants, make them depend on the BPN package,
but do not further foster their build.

Jan

[cedric.h...@siemens.com]

My "dpkg" recipe is clean, it has a DEPENDS clause with only BPN names
But we have multiarch [1] suffixing each of our DEPENDS with -native
when built for -native

Or maybe I am not understanding your proposal? I am pretty sure we have
a bug in Isar. I can build a test case if the illustration with
example-raw is not sufficient / convincing

[1]
https://github.com/ilbers/isar/blob/master/meta/classes/multiarch.bbclass#L73

[Jan Kiszka]

Yes, this is an issue in isar. But I would address it there by not doing
anything in <raw>-native/compat targets, only making them depend on
their base target. IIRC, we already have such a case deep in the kernel
build.

Rebuilding the same thing is wrong.

[cedric.h...@siemens.com]

Agreed but :)

There are two cases to consider

1) dpkg-raw package has DPKG_ARCH = "all" => shall not rebuild for -
native

2) dpkg-raw produces an architecture-dependent package (Architecture:
any) => we would need to build both -native and non-native

I hit 1) and I am investigating further (getting myself familiar with
the multiarch code) and 2) is IMO addressed with this patch

Actually, when I first hit the issue, my dpkg-raw package did not have
DPKG_ARCH set to "all" hence my first instinct was to set it to avoid
the extra build

[cedric.h...@siemens.com]

For 1) and if my reading of multiarch is correct, the following should
do the trick:

diff --git a/meta/classes/multiarch.bbclass
b/meta/classes/multiarch.bbclass
index bb0f7983..8aa2de64 100644
--- a/meta/classes/multiarch.bbclass
+++ b/meta/classes/multiarch.bbclass
@@ -10,7 +10,7 @@ python() {
d.appendVar('BBCLASSEXTEND', ' compat')

# build native separately only when it differs from the target
variant
- if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
+ if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH') or
d.getVar('DPKG_ARCH') == 'all':
pn = d.getVar('PN')
if not pn.endswith('-native') and not pn.endswith('-compat'):
provides = (d.getVar('PROVIDES') or '').split()


This appears to work in the quick test I did but will perform more
tests

[Jan Kiszka]

No, your patch is even for that case wrong because it enforces source
package rebuilds that are no longer expected and supported - same base
target, same source package.

Jan

>
> Actually, when I first hit the issue, my dpkg-raw package did not have
> DPKG_ARCH set to "all" hence my first instinct was to set it to avoid
> the extra build
>
>>
>> Jan
>>
>

--

[Jan Kiszka]

At least for debianized packages (like dpkg-raw) and for -native. Will
probably not address -compat dependencies. Yeah, now it's getting
complicated.

[cedric.h...@siemens.com]

Are you sure? My understanding of the code and testing (with changes)
is showing that do_dpkg_source is only called once as as expected for
the non-native variant, for the -native case, do_fetch_common_source is
called: the source package to build -native is pulled from isar-apt

It is in fact one of the reason this patch is dependent on [2]. Without
this, the build will try to get its inputs from /home/builder/example-
raw/image because sources used to build -native were created when the
non-native package was built but would have been put under
/home/builder/example-raw-native/ without [2]

[2] https://groups.google.com/g/isar-users/c/md3gmDvhEPQ

>
> Jan
>
> >
> > Actually, when I first hit the issue, my dpkg-raw package did not
> > have
> > DPKG_ARCH set to "all" hence my first instinct was to set it to
> > avoid
> > the extra build
> >
> > >
> > > Jan
> > >
> >
>

--

[cedric.h...@siemens.com]

if DPKG_ARCH == all, wouldn't we expect the recipe to have PROVIDES set
to "example-raw-compat example-raw-native" since whatever flavor would
result in an _all.deb package being generated?

[Cedric Hombourger]

Binary-independent packages (DPKG_ARCH == all) do not require separate
builds for our special -compat and -native targets: extend PROVIDES for
these cases. It should be noted that DPKG_ARCH is always set for all
dpkg variants and defaults to "any" (only recipes setting DPKG_ARCH to
all explicitly will be impacted).

Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---

meta/classes/multiarch.bbclass | 34 ++++++++++++++++++++++++----------
1 file changed, 24 insertions(+), 10 deletions(-)

diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
index bb0f7983..9df8eefd 100644
--- a/meta/classes/multiarch.bbclass
+++ b/meta/classes/multiarch.bbclass
@@ -5,25 +5,39 @@

inherit compat
python() {
+ pn = d.getVar('PN')
+ archDiffers = d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH')
+ archIsAll = d.getVar('DPKG_ARCH') == 'all'
+
+ def pn_multiarch_target(pn):
+ return pn.endswith('-native') or pn.endswith('-compat')
+
+ def extend_provides(pn, provides, d):
+ if not pn_multiarch_target(pn):
+ all_provides = (d.getVar('PROVIDES') or '').split()
+ for p in all_provides:
+ if not pn_multiarch_target(p):
+ d.appendVar('PROVIDES', f' {p}-{provides}')
+ d.appendVar('PROVIDES', f' {pn}-{provides}')
+
# provide compat only when we can build it
if isar_can_build_compat(d):
- d.appendVar('BBCLASSEXTEND', ' compat')
+ # but do not build separately if architecture-independent
+ if archIsAll:
+ extend_provides(pn, 'compat', d)
+ else:
+ d.appendVar('BBCLASSEXTEND', ' compat')

# build native separately only when it differs from the target variant
- if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):

- pn = d.getVar('PN')
- if not pn.endswith('-native') and not pn.endswith('-compat'):
- provides = (d.getVar('PROVIDES') or '').split()
- for p in provides:
- d.appendVar('PROVIDES', f' {p}-native')
- d.appendVar('PROVIDES', f' {pn}-native')
- else:
+ if archIsAll is False and archDiffers:
d.appendVar('BBCLASSEXTEND', ' native')
+ else:
+ extend_provides(pn, 'native', d)

# drop own -native build dependencies at recipe level if building natively
# and not for the builder architecture
depends = d.getVar('DEPENDS')
- if depends is not None and d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH') \
+ if depends is not None and archDiffers \
and not bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')):
new_deps = []
for dep in depends.split():
--
2.39.2

[Cedric Hombourger]

The intent of the dpkg-raw class is to easily package configuration and data
files into a Debian package. Packages to be compiled should really use other
dpkg classes where support for cross-compilation and multiarch is provided
and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes and a
deprecation warning will be raised if overriden.

Despite this change, Isar may still build dpkg-raw packages a second time
when pulled by a -compat or -native package. multiarch is changed to check
if DPKG_ARCH was changed to all and will add -compat and -native to the
PROVIDES of the package (this condition will be checked for all dpkg variants)

It shall be noted that setting DPKG_ARCH in a recipe is a contract, the
user declares that generated package(s) are either architecture dependent
or independent.

Changes since v2:
* v1 used a completely different approach where we were ensuring that
do_install was called whether sources were built or pulled from
isar_apt (do_fetch_common_source) since override_dh_install is
pulling contents of dpkg-raw packages outside of the source tree
(${PP}/image)

Cedric Hombourger (2):
multiarch: avoid separate builds when DPKG_ARCH is all
dpkg-raw: change DPKG_ARCH to all

RECIPE-API-CHANGELOG.md | 11 +++++++++++
meta/classes/dpkg-raw.bbclass | 8 ++++++++
meta/classes/multiarch.bbclass | 34 ++++++++++++++++++++++++----------
3 files changed, 43 insertions(+), 10 deletions(-)

--
2.39.2

[Cedric Hombourger]

Packages created by dpkg-raw are really for configuration and data files
and therefore architecture independent. DPKG_ARCH is now set to all and
a warning will be produced if changed by the recipe (this will later be
turned into an error).

Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---

RECIPE-API-CHANGELOG.md | 11 +++++++++++
meta/classes/dpkg-raw.bbclass | 8 ++++++++

2 files changed, 19 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 21c558d2..d40827d0 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -646,3 +646,14 @@ recipe would use the following setting:
```
HEADERS_INSTALL_EXTRA += "nvidia"
```
+
+### Architecture for dpkg-raw packages
+
+The intent of the dpkg-raw class is to easily package configuration and data
+files into a Debian package. Packages to be compiled should really use other
+dpkg classes where support for cross-compilation and multiarch is provided
+and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes and a
+deprecation warning will be raised if overriden.
+
+This change fixes an issue where a `dpkg` package is built for `-compat`
+or `-native` and `DEPENDS` on a `dpkg-raw` package.
diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index dd7b761f..cbaf298c 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -7,6 +7,9 @@ inherit dpkg

D = "${WORKDIR}/image"

+# Create a binary-indep package
+DPKG_ARCH = "all"
+
# Populate folder that will be picked up as package
do_install() {
bbnote "Put your files for this package in $""{D}"
@@ -17,6 +20,11 @@ addtask install after do_patch do_transform_template before do_prepare_build

do_prepare_build[cleandirs] += "${S}/debian"
do_prepare_build() {

+ if [ "${DPKG_ARCH}" != "all" ]; then
+ bbwarn "support for building binary-arch packages with " \
+ "dpkg-raw is deprecated and will be removed"
+ fi
+
cd ${D}
find . -maxdepth 1 ! -name .. -and ! -name . -and ! -name debian | \
sed 's:^./::' > ${S}/debian/${PN}.install
--
2.39.2

[Jan Kiszka]

Could split this patch into 2, the first introducing these helpers
without changing the current logic, and as second one adding the actual
logic of this patch? All this is complex, and it would greatly help with
reviewing when changes are finer-grained.

[Jan Kiszka]

On 17.09.24 09:00, 'Cedric Hombourger' via isar-users wrote:
> Packages created by dpkg-raw are really for configuration and data files
> and therefore architecture independent. DPKG_ARCH is now set to all and
> a warning will be produced if changed by the recipe (this will later be
> turned into an error).

Well, this is generally correct, but not always. We may also package a
prebuilt arch-dependent thing here, and then the result would no longer
be "all". Such cases should remain expressible correctly.

That said, changing the default is not a bad move.

Jan

[Cedric Hombourger]

-native / -compat do not always apply and some corner cases were
found. Introduce some helper functions to refactor the existing
code without changing its semantics for now.

Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---

meta/classes/multiarch.bbclass | 24 +++++++++++++++++-------
1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
index bb0f7983..802686d5 100644
--- a/meta/classes/multiarch.bbclass
+++ b/meta/classes/multiarch.bbclass
@@ -5,25 +5,35 @@

inherit compat
python() {

+ archDiffers = d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH')
+

+ def pn_multiarch_target(pn):
+ return pn.endswith('-native') or pn.endswith('-compat')
+
+ def extend_provides(pn, provides, d):
+ if not pn_multiarch_target(pn):
+ all_provides = (d.getVar('PROVIDES') or '').split()
+ for p in all_provides:
+ if not pn_multiarch_target(p):
+ d.appendVar('PROVIDES', f' {p}-{provides}')
+ d.appendVar('PROVIDES', f' {pn}-{provides}')
+

# provide compat only when we can build it
if isar_can_build_compat(d):

d.appendVar('BBCLASSEXTEND', ' compat')

# build native separately only when it differs from the target variant
- if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):

+ if archDiffers is False:

pn = d.getVar('PN')
- if not pn.endswith('-native') and not pn.endswith('-compat'):
- provides = (d.getVar('PROVIDES') or '').split()
- for p in provides:
- d.appendVar('PROVIDES', f' {p}-native')
- d.appendVar('PROVIDES', f' {pn}-native')

+ if pn_multiarch_target(pn) is False:

+ extend_provides(pn, 'native', d)

else:

d.appendVar('BBCLASSEXTEND', ' native')

# drop own -native build dependencies at recipe level if building natively
# and not for the builder architecture
depends = d.getVar('DEPENDS')
- if depends is not None and d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH') \
+ if depends is not None and archDiffers \
and not bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')):
new_deps = []
for dep in depends.split():

--
2.34.1

[Cedric Hombourger]

Binary-independent packages (DPKG_ARCH == all) do not require separate
builds for our special -compat and -native targets: extend PROVIDES for
these cases. It should be noted that DPKG_ARCH is always set for all
dpkg variants and defaults to "any" (only recipes setting DPKG_ARCH to
"all" explicitly will be impacted).

Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---

meta/classes/multiarch.bbclass | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
index 802686d5..9df8eefd 100644
--- a/meta/classes/multiarch.bbclass
+++ b/meta/classes/multiarch.bbclass
@@ -5,7 +5,9 @@

inherit compat
python() {
+ pn = d.getVar('PN')

archDiffers = d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH')

+ archIsAll = d.getVar('DPKG_ARCH') == 'all'

def pn_multiarch_target(pn):

return pn.endswith('-native') or pn.endswith('-compat')

@@ -20,15 +22,17 @@ python() {

# provide compat only when we can build it
if isar_can_build_compat(d):

- d.appendVar('BBCLASSEXTEND', ' compat')
+ # but do not build separately if architecture-independent
+ if archIsAll:
+ extend_provides(pn, 'compat', d)
+ else:

+ d.appendVar('BBCLASSEXTEND', ' compat')

# build native separately only when it differs from the target variant

- if archDiffers is False:
- pn = d.getVar('PN')
- if pn_multiarch_target(pn) is False:
- extend_provides(pn, 'native', d)

- else:
+ if archIsAll is False and archDiffers:

d.appendVar('BBCLASSEXTEND', ' native')

+ else:

+ extend_provides(pn, 'native', d)

# drop own -native build dependencies at recipe level if building natively
# and not for the builder architecture

--
2.34.1

[Cedric Hombourger]

The intent of the dpkg-raw class is to easily package configuration and data

files into a Debian package. Packages to be compiled should really use other

dpkg classes where support for cross-compilation and multiarch is provided

and tested. `DPKG_ARCH` is now defaulting to `all` for `dpkg-raw` recipes.

Despite this change, Isar may still build dpkg-raw packages a second time
when pulled by a -compat or -native package. multiarch is changed to check

if DPKG_ARCH is "all" and will add -compat and -native to the PROVIDES of

the package (this condition will be checked for all dpkg variants)

It shall be noted that setting DPKG_ARCH in a recipe is a contract, the
user declares that generated package(s) are either architecture dependent
or independent.

Changes in v3:
* Do not warn if DPKG_ARCH is not "all" as users may use this class
to create a Debian package from a prebuilt binaries (shared libraries,
applications, etc.)
* Set DEBIAN_MULTI_ARCH to "foreign" in dpkg-raw if DPKG_ARCH is "all"
* Introduce helpers in a separate commit to ease reviews (a "fast"
ci_build run was used to validate this intermediate step).

Changes in v2:

* v1 used a completely different approach where we were ensuring that
do_install was called whether sources were built or pulled from
isar_apt (do_fetch_common_source) since override_dh_install is
pulling contents of dpkg-raw packages outside of the source tree
(${PP}/image)

Cedric Hombourger (3):
multiarch: introduce some helpers to later handle corner cases

multiarch: avoid separate builds when DPKG_ARCH is all
dpkg-raw: change DPKG_ARCH to all

RECIPE-API-CHANGELOG.md | 10 ++++++++++
meta/classes/dpkg-raw.bbclass | 4 ++++
meta/classes/multiarch.bbclass | 34 ++++++++++++++++++++++++----------
3 files changed, 38 insertions(+), 10 deletions(-)

--
2.34.1

[Cedric Hombourger]

Packages created by dpkg-raw are really for configuration and data files
and therefore architecture independent. DPKG_ARCH is now set to "all" and

DEBIAN_MULTI_ARCH will default to "foreign" unless DPKG_ARCH is changed
(it would then be reset to "no").

Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---

RECIPE-API-CHANGELOG.md | 10 ++++++++++
meta/classes/dpkg-raw.bbclass | 4 ++++

2 files changed, 14 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 21c558d2..57b2205a 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -646,3 +646,13 @@ recipe would use the following setting:

```
HEADERS_INSTALL_EXTRA += "nvidia"
```
+
+### Architecture for dpkg-raw packages
+
+The intent of the dpkg-raw class is to easily package configuration and data
+files into a Debian package. Packages to be compiled should really use other
+dpkg classes where support for cross-compilation and multiarch is provided

+and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes.
+
+This change fixes an issue where a `dpkg` package is built for `-compat` or
+`-native` and `DEPENDS` on a `dpkg-raw` package.
diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index dd7b761f..ac70d5cc 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -7,6 +7,10 @@ inherit dpkg

D = "${WORKDIR}/image"

+# Create a binary-indep package
+DPKG_ARCH = "all"

+DEBIAN_MULTI_ARCH = "${@ 'foreign' if '${DPKG_ARCH}' == 'all' else 'no' }"

+
# Populate folder that will be picked up as package
do_install() {
bbnote "Put your files for this package in $""{D}"

--
2.34.1

[Uladzimir Bely]

On Thu, 2024-09-26 at 05:39 +0200, 'Cedric Hombourger' via isar-users
wrote:

Applied to next, thanks.

--
Best regards,
Uladzimir.

[Jan Kiszka]

Sorry, forgotten to review. We will need fixes on top now.

Jan

[Jan Kiszka]

On 26.09.24 05:39, 'Cedric Hombourger' via isar-users wrote:
> -native / -compat do not always apply and some corner cases were
> found. Introduce some helper functions to refactor the existing
> code without changing its semantics for now.
>
> Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
> ---
> meta/classes/multiarch.bbclass | 24 +++++++++++++++++-------
> 1 file changed, 17 insertions(+), 7 deletions(-)
>
> diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
> index bb0f7983..802686d5 100644
> --- a/meta/classes/multiarch.bbclass
> +++ b/meta/classes/multiarch.bbclass
> @@ -5,25 +5,35 @@
>
> inherit compat
> python() {
> + archDiffers = d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH')
> +
> + def pn_multiarch_target(pn):
> + return pn.endswith('-native') or pn.endswith('-compat')
> +
> + def extend_provides(pn, provides, d):
> + if not pn_multiarch_target(pn):

Redundant? You are already testing this...

> + all_provides = (d.getVar('PROVIDES') or '').split()
> + for p in all_provides:
> + if not pn_multiarch_target(p):
> + d.appendVar('PROVIDES', f' {p}-{provides}')
> + d.appendVar('PROVIDES', f' {pn}-{provides}')
> +
> # provide compat only when we can build it
> if isar_can_build_compat(d):
> d.appendVar('BBCLASSEXTEND', ' compat')
>
> # build native separately only when it differs from the target variant
> - if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
> + if archDiffers is False:
> pn = d.getVar('PN')
> - if not pn.endswith('-native') and not pn.endswith('-compat'):
> - provides = (d.getVar('PROVIDES') or '').split()
> - for p in provides:
> - d.appendVar('PROVIDES', f' {p}-native')
> - d.appendVar('PROVIDES', f' {pn}-native')
> + if pn_multiarch_target(pn) is False:

...here. And you are using different styles ("not" vs "is False").

But these lines get reworked again with patch 2 which at least resolves
the duplicate test. But check again if you are using consistent styles.

> + extend_provides(pn, 'native', d)
> else:
> d.appendVar('BBCLASSEXTEND', ' native')
>
> # drop own -native build dependencies at recipe level if building natively
> # and not for the builder architecture
> depends = d.getVar('DEPENDS')
> - if depends is not None and d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH') \
> + if depends is not None and archDiffers \
> and not bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')):
> new_deps = []
> for dep in depends.split():

Jan

[Jan Kiszka]

On 26.09.24 05:39, 'Cedric Hombourger' via isar-users wrote:
> Packages created by dpkg-raw are really for configuration and data files
> and therefore architecture independent. DPKG_ARCH is now set to "all" and
> DEBIAN_MULTI_ARCH will default to "foreign" unless DPKG_ARCH is changed
> (it would then be reset to "no").
>
> Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
> ---
> RECIPE-API-CHANGELOG.md | 10 ++++++++++
> meta/classes/dpkg-raw.bbclass | 4 ++++
> 2 files changed, 14 insertions(+)
>
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 21c558d2..57b2205a 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -646,3 +646,13 @@ recipe would use the following setting:
> ```
> HEADERS_INSTALL_EXTRA += "nvidia"
> ```
> +
> +### Architecture for dpkg-raw packages
> +
> +The intent of the dpkg-raw class is to easily package configuration and data
> +files into a Debian package. Packages to be compiled should really use other
> +dpkg classes where support for cross-compilation and multiarch is provided
> +and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes.

Not fully true. In rare cases, it can also be used for pre-compiled
binaries. This sections should state that the /common/ case is 'all' and
therefore the /default/ is set to 'all'.

> +
> +This change fixes an issue where a `dpkg` package is built for `-compat` or
> +`-native` and `DEPENDS` on a `dpkg-raw` package.

This is misleading. One can still tune DPKG_ARCH and run into such
cases. Warn about them, don't claim they are fixed.

> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
> index dd7b761f..ac70d5cc 100644
> --- a/meta/classes/dpkg-raw.bbclass
> +++ b/meta/classes/dpkg-raw.bbclass
> @@ -7,6 +7,10 @@ inherit dpkg
>
> D = "${WORKDIR}/image"
>
> +# Create a binary-indep package
> +DPKG_ARCH = "all"

Weak assignment, please.

> +DEBIAN_MULTI_ARCH = "${@ 'foreign' if '${DPKG_ARCH}' == 'all' else 'no' }"
> +
> # Populate folder that will be picked up as package
> do_install() {
> bbnote "Put your files for this package in $""{D}"

Jan

[Cedric Hombourger]

Fixes: 13ef1d48

Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---

meta/classes/multiarch.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
index 9df8eefd..babdfbd4 100644
--- a/meta/classes/multiarch.bbclass
+++ b/meta/classes/multiarch.bbclass
@@ -29,7 +29,7 @@ python() {

d.appendVar('BBCLASSEXTEND', ' compat')

# build native separately only when it differs from the target variant

- if archIsAll is False and archDiffers:
+ if not archIsAll and archDiffers:

d.appendVar('BBCLASSEXTEND', ' native')

else:
extend_provides(pn, 'native', d)
--
2.39.5

[Cedric Hombourger]

Fixes: 0816ae6e

Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---

meta/classes/dpkg-raw.bbclass | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index ac70d5cc..a7bf204a 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -7,9 +7,9 @@ inherit dpkg

D = "${WORKDIR}/image"

-# Create a binary-indep package
-DPKG_ARCH = "all"
-DEBIAN_MULTI_ARCH = "${@ 'foreign' if '${DPKG_ARCH}' == 'all' else 'no' }"
+# Default to creating a binary-indep package
+DPKG_ARCH ??= "all"
+DEBIAN_MULTI_ARCH ??= "${@ 'foreign' if '${DPKG_ARCH}' == 'all' else 'no' }"

# Populate folder that will be picked up as package
do_install() {

--
2.39.5

[Cedric Hombourger]

Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---

RECIPE-API-CHANGELOG.md | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 57b2205a..8c1e8fc8 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -649,10 +649,14 @@ HEADERS_INSTALL_EXTRA += "nvidia"

### Architecture for dpkg-raw packages

-The intent of the dpkg-raw class is to easily package configuration and data
-files into a Debian package. Packages to be compiled should really use other
-dpkg classes where support for cross-compilation and multiarch is provided
-and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes.
+The primary use-case of the dpkg-raw class is to easily package configuration
+and data files into a Debian package: the target architecture will now default
+to "all". It may also be used to package binaries that were built outside of
+Isar: such recipes may still override `DPKG_ARCH` to `"any"`.

This change fixes an issue where a `dpkg` package is built for `-compat` or

-`-native` and `DEPENDS` on a `dpkg-raw` package.
+`-native` and `DEPENDS` on a `dpkg-raw` package with `DPKG_ARCH` set to `"all"`.
+Some issues remain with `dpkg-raw` packages: Isar will advertise -native and
+-compat variants even though such recipes can only produce packages for
+`${DISTRO_ARCH}` or `"all"`. That corner case needs to be further discussed
+and will require further changes.
--
2.39.5

[Jan Kiszka]

On 04.10.24 15:11, 'Cedric Hombourger' via isar-users wrote:
> Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
> ---
> RECIPE-API-CHANGELOG.md | 14 +++++++++-----
> 1 file changed, 9 insertions(+), 5 deletions(-)
>
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 57b2205a..8c1e8fc8 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -649,10 +649,14 @@ HEADERS_INSTALL_EXTRA += "nvidia"
>
> ### Architecture for dpkg-raw packages
>
> -The intent of the dpkg-raw class is to easily package configuration and data
> -files into a Debian package. Packages to be compiled should really use other
> -dpkg classes where support for cross-compilation and multiarch is provided
> -and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes.
> +The primary use-case of the dpkg-raw class is to easily package configuration
> +and data files into a Debian package: the target architecture will now default
> +to "all". It may also be used to package binaries that were built outside of
> +Isar: such recipes may still override `DPKG_ARCH` to `"any"`.
>

...to 'any' or whatever reflects the package content accurately.

> This change fixes an issue where a `dpkg` package is built for `-compat` or
> -`-native` and `DEPENDS` on a `dpkg-raw` package.
> +`-native` and `DEPENDS` on a `dpkg-raw` package with `DPKG_ARCH` set to `"all"`.
> +Some issues remain with `dpkg-raw` packages: Isar will advertise -native and
> +-compat variants even though such recipes can only produce packages for
> +`${DISTRO_ARCH}` or `"all"`. That corner case needs to be further discussed
> +and will require further changes.

The fact that -native/-compat stub targets are provided for 'all'
packages is not the issue. The issue is limited to DPKG_ARCH != all.

[Cedric Hombourger]

Fixes: 13ef1d48

Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---

[Cedric Hombourger]

Fixes: 0816ae6e

Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---

[Cedric Hombourger]

Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---

RECIPE-API-CHANGELOG.md | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 57b2205a..8eeaf325 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -649,10 +649,19 @@ HEADERS_INSTALL_EXTRA += "nvidia"

### Architecture for dpkg-raw packages

-The intent of the dpkg-raw class is to easily package configuration and data
-files into a Debian package. Packages to be compiled should really use other
-dpkg classes where support for cross-compilation and multiarch is provided
-and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes.
+The primary use-case of the dpkg-raw class is to easily package configuration
+and data files into a Debian package: the target architecture will now default
+to "all". It may also be used to package binaries that were built outside of

+Isar: such recipes may still override `DPKG_ARCH` to `"any"` or a specific
+architecture matching binaries to be included in the payload of the package.

This change fixes an issue where a `dpkg` package is built for `-compat` or
-`-native` and `DEPENDS` on a `dpkg-raw` package.
+`-native` and `DEPENDS` on a `dpkg-raw` package with `DPKG_ARCH` set to `"all"`.

+Some issues remain with `dpkg-raw` packages targetting a specific architecture:
+Isar will advertise -native and -compat variants even though such recipes can
+only produce packages for that architecture and not what could possibly expect
+for -native or -compat. If we consider a dpkg-raw recipe generating an `arm64`
+package on an `amd64` host: you would expect the -native variant to produce
+an `amd64` package and -compat an 'armhf` package: it will however remain
+`arm64` and build of dependent recipes (image or dpkg) may fail because of
+the architecture mismatch.
--
2.39.5

[Cedric Hombourger]

Changes in v2:
Attempt to address feedback [1] from Jan in the RECIPE-API-CHANGELOG
First two patches (code changes) are the same.

[1] https://lists.isar-build.org/isar-users/62b2d7bb-25cf-4c42...@siemens.com/T/#u

Cedric Hombourger (3):
dpkg-raw: use weak assignments for DPKG_ARCH and DEBIAN_MULTI_ARCH
multiarch: use "if not" vs "if cond is False" for consistency
RECIPE-API-CHANGELOG.md: clarify scope of recent multiarch/dpkg-arch
changes

RECIPE-API-CHANGELOG.md | 19 ++++++++++++++-----
meta/classes/dpkg-raw.bbclass | 6 +++---
meta/classes/multiarch.bbclass | 2 +-
3 files changed, 18 insertions(+), 9 deletions(-)

--
2.39.5

[Uladzimir Bely]

On Fri, 2024-10-04 at 16:17 +0200, 'Cedric Hombourger' via isar-users
wrote:

Hello all.

The patchset is tested and ready to be merged. Since it's technically a
set of fixes for already merged things, we'd like to apply it to next
as soon as possible, if there are no objections.

--
Best regards,
Uladzimir.

[Uladzimir Bely]

On Fri, 2024-10-04 at 16:17 +0200, 'Cedric Hombourger' via isar-users
wrote:

[Uladzimir Bely]

On Thu, 2024-09-26 at 05:39 +0200, 'Cedric Hombourger' via isar-users
wrote:

Playing with isar-apt I've found an issue that existed for a long time
but with this patch it became easier to find.

For the packages with "all" architecture there are no source packages
found in isar-apt.

Steps to reproduce:

1. Run `bitbake libhello hello example-raw sshd-regen-keys`.
2. Check isar-apt contents (e.g. `build/tmp/deploy/isar-apt/debian-
bookworm-amd64/apt/debian-bookworm/pool/main/`)

With current `next` we have in isar-apt:

```
libhello dsc + deb
hello dsc + deb
example-raw only deb
sshd-regen-keys only deb
```

With this patch reverted (and some followed patches):

```
libhello dsc + deb
hello dsc + deb
example-raw dsc + deb
sshd-regen-keys only deb
```

Package `sshd-regen-keys` is still affected since it explicitly sets
DPKG_ARCH = "all" in the recipe.

Debugging shows that source package goes to the isar-apt but later
is wrongly removed by `repo_del_package` from `repository.bbclass` due
to "all" architecture.

>  # Populate folder that will be picked up as package
>  do_install() {
>   bbnote "Put your files for this package in $""{D}"
> --
> 2.34.1
>

--
Best regards,
Uladzimir.

[Uladzimir Bely]

@Cedric

Oh, while I was writing this email you seem to have sent a patchset
that fixes the issue. Thanks, will check it.

> >  # Populate folder that will be picked up as package
> >  do_install() {
> >   bbnote "Put your files for this package in $""{D}"
> > --
> > 2.34.1
> >
>
> --
> Best regards,
> Uladzimir.
>

--
Best regards,
Uladzimir.

[cedric.h...@siemens.com]

On Fri, 2024-11-22 at 09:07 +0300, Uladzimir Bely wrote:

See https://groups.google.com/g/isar-users/c/NrVHusKyNTA for a fix
Looks like good timing :)

Cedric Hombourger
Siemens AG
www.siemens.com