isar-bootstrap

[ydi...@free.fr]

Hello,

While reading the isar-bootstrap recipe to get the gist of it, I
stumbled on what looks like just a typo, but could possibly be more
than that:

I was trying to get a grasp on DISTRO_APT_SOURCES handling, and noticed
a spurious match for "distro_apt_sources" in DISTRO_APT_PREFERENCES handling.
While this possibly was meant to read "distro_apt_preferences" instead,
it raises the question of whether it is useful to add all of this to SRC_URI
if not adding it did not cause any problem (it does not appear to be used
by any conf in isar itself, but I guess it is/has been used in downstream
layers).

This brought me to where this gets used (both _SOURCES and _PREFERENCES in fact),
ie. do_apt_config_prepare: the whole point of SRC_URI is to direct the fetcher
to download/copy source files into $WORKDIR, and then later tasks (unpack/patch
in OE) take files from there. But here, $WORKDIR is not used, and instead
aggregate_files() uses resolve_file() on them. At first sight this seems
to break the bitbake abstractions (and seems unnecessarily complicated at
the same time). Am I missing something, or is it worth for me to try patching ?

On a related note, I assume that aggregate_files() was meant to support distro
versions predating preferences.d/ and sources.list.d/ - would it be correct
to use the latter instead to simply things further ? Most recent is preferences.d/,
released with 0.7.22 in 2009, and stretch which seems to be the oldest supported
distro has 1.4.


The chroot-setup.sh script there also seems to do things that are already
handled by the debootstrap (policy.d and start-stop-daemon), except for
the upstart support. Even if someone still uses upstart (which has not been
developped since 2014), probably the rest could be omitted ?


Best regards,
--
Yann

[Henning Schild]

Hi,

Am Tue, 26 Oct 2021 01:33:51 +0200 (CEST)
schrieb ydi...@free.fr:

> Hello,
>
> While reading the isar-bootstrap recipe to get the gist of it, I
> stumbled on what looks like just a typo, but could possibly be more
> than that:
>
> I was trying to get a grasp on DISTRO_APT_SOURCES handling, and
> noticed a spurious match for "distro_apt_sources" in
> DISTRO_APT_PREFERENCES handling. While this possibly was meant to
> read "distro_apt_preferences" instead,

Indeed, that looks weird.
https://github.com/ilbers/isar/blob/ceb7e21154fc4862f704bb5c7739e87a26db6eb3/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc#L67

> it raises the question of
> whether it is useful to add all of this to SRC_URI if not adding it
> did not cause any problem (it does not appear to be used by any conf
> in isar itself, but I guess it is/has been used in downstream layers).
>
> This brought me to where this gets used (both _SOURCES and
> _PREFERENCES in fact), ie. do_apt_config_prepare: the whole point of
> SRC_URI is to direct the fetcher to download/copy source files into
> $WORKDIR, and then later tasks (unpack/patch in OE) take files from
> there. But here, $WORKDIR is not used, and instead aggregate_files()
> uses resolve_file() on them. At first sight this seems to break the
> bitbake abstractions (and seems unnecessarily complicated at the same
> time). Am I missing something, or is it worth for me to try patching
> ?

Not sure i get your point. In general we have to deal with two worlds
in Isar. There is the debian-world and the bitbake world and they are
not too tightly integrated at some points.
Meaning we have debian fetching stuff and bitbake fetching stuff. We
have two namespace for packages which leads to weird statements like

# comma-sep
DEBIAN_DEPENDS = "debian-binary-package-a, debian-binary-package-b"
# space-sep
DEPENDS = "recipe-for-binpkg-a recipe-for-binpkg-b"

That two world can not be easily resolved and is found in several
places, the one i pointed out being maybe the nasties one. No need to
comment on a possible idea, it has been tried and is more complex than
i made it look.

That said you can not see everything in our code, the files placed
carefully will be used by debootstrap and later by apt.

And the two variables are lists of files which are meant to be
customizeable. That way you can switch to another mirror, add more
repos (ppa in ubunut speach) and configure what to take from where or
what to possibly pin (prefs)

> On a related note, I assume that aggregate_files() was meant to
> support distro versions predating preferences.d/ and sources.list.d/
> - would it be correct to use the latter instead to simply things
> further ? Most recent is preferences.d/, released with 0.7.22 in
> 2009, and stretch which seems to be the oldest supported distro has
> 1.4.

I do not know, maybe also to be able to delete things again after the
build. Because you want other configuration at build-time than you want
on product delivery. i.e. build against a "frozen" mirror but deliver
with a public mirror as update source

>
> The chroot-setup.sh script there also seems to do things that are
> already handled by the debootstrap (policy.d and start-stop-daemon),
> except for the upstart support. Even if someone still uses upstart
> (which has not been developped since 2014), probably the rest could
> be omitted ?
>

Thanks for the feedback. But i guess it might be a little much mixed up
in one email. So i mainly answered to give you more information, not
because i want to go and write patches.
But maybe you at some point might want to.

regards,
Henning

> Best regards,

[Anton Mikanovich]

26.10.2021 02:33, ydi...@free.fr wrote:
> Hello,
>
> While reading the isar-bootstrap recipe to get the gist of it, I
> stumbled on what looks like just a typo, but could possibly be more
> than that:
>
> I was trying to get a grasp on DISTRO_APT_SOURCES handling, and noticed
> a spurious match for "distro_apt_sources" in DISTRO_APT_PREFERENCES handling.

> While this possibly was meant to read "distro_apt_preferences" instead.

This was already fixed by `[PATCH v2] isar-bootstrap: Fix SRC_URI
inclusion` and will be merged into next soon.

--
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

[ydi...@free.fr]

Hi,

Anton wrote:
> > I was trying to get a grasp on DISTRO_APT_SOURCES handling, and
> > noticed
> > a spurious match for "distro_apt_sources" in DISTRO_APT_PREFERENCES
> > handling.
> > While this possibly was meant to read "distro_apt_preferences"

> > instead.

Cool :)

Henning wrote:
> This was already fixed by `[PATCH v2] isar-bootstrap: Fix SRC_URI
> inclusion` and will be merged into next soon.

At first sight I don't find this last part too problematic, this looks
like the standard
double source/binary namespace, and DEBIAN_DEPENDS rather looks like
a Yocto RDEPENDS (though processed), and DEPENDS a Debian Build-Depends.
(indeed Yocto packaging does generate Depends from RDEPENDS when building
a deb or ipk)

> No need to
> comment on a possible idea, it has been tried and is more complex
> than i made it look.

I guess this refers to the double-downloader problem, I had my share
of such things some time ago [0] (and finally found a rather acceptable
solution based on (IIRC) the meta-rust approach). I'd be interested to
learn about the approaches you tried.

> That said you can not see everything in our code, the files placed
> carefully will be used by debootstrap and later by apt.

Sure, such things do not lay themselves on (e)paper in one day, and
that can make them more difficult for newcomers to see all the
implications of a given construct - and also makes it easy to break
things.

> And the two variables are lists of files which are meant to be
> customizeable. That way you can switch to another mirror, add more
> repos (ppa in ubunut speach) and configure what to take from where or
> what to possibly pin (prefs)

Yes, I realized this and was not suggesting to drop them and use SRC_URI
instead (although the idea did cross my mind ;). This remark was purely
from the bitbake side, and I realized thinking about it later that some
assumptions of mine may have been wrong, I'll have to check this later.

> > On a related note, I assume that aggregate_files() was meant to
> > support distro versions predating preferences.d/ and
> > sources.list.d/
> > - would it be correct to use the latter instead to simply things
> > further ? Most recent is preferences.d/, released with 0.7.22 in
> > 2009, and stretch which seems to be the oldest supported distro has
> > 1.4.
>
> I do not know, maybe also to be able to delete things again after the
> build. Because you want other configuration at build-time than you
> want
> on product delivery. i.e. build against a "frozen" mirror but deliver
> with a public mirror as update source

OK I see.

>
> >
> > The chroot-setup.sh script there also seems to do things that are
> > already handled by the debootstrap (policy.d and
> > start-stop-daemon),
> > except for the upstart support. Even if someone still uses upstart
> > (which has not been developped since 2014), probably the rest could
> > be omitted ?
> >
>
> Thanks for the feedback. But i guess it might be a little much mixed
> up in one email.

Yes, I had to refrain from bringing too many subjects already :)

> So i mainly answered to give you more information, not
> because i want to go and write patches.
> But maybe you at some point might want to.

Yes, though I will need more time to get a full picture.

[0] https://patchwork.openembedded.org/patch/178697/

Best regards,
--
Yann

[Henning Schild]

Am Tue, 26 Oct 2021 20:58:16 +0200 (CEST)
schrieb ydi...@free.fr:

Yes that is pretty much it. But Isar currently does not try to help
here. In practice it is less of "big deal" just very confusing to new
Isar users when they have to say things twice in different lingo.

> > No need to
> > comment on a possible idea, it has been tried and is more complex
> > than i made it look.
>
> I guess this refers to the double-downloader problem, I had my share
> of such things some time ago [0] (and finally found a rather
> acceptable solution based on (IIRC) the meta-rust approach). I'd be
> interested to learn about the approaches you tried.

No i was talking about the src vs binary namespace, which in fact you
have in Debian and OE alike. And Isar has 4 such namespaces i guess.
But as pointed out that is less of a practical problem than a
"confusion topic". If you go mostly debian and litte Isar (like 95/5)
it is really just an academic thing and not worth too much effort.

For the download Isar goes the pragmatic way and lets debian fetch what
it wants. With a few exceptions ... i.e. there is only one "global
apt-get update" so you have to hope that you can apt-get install what
that initial run created your external database for. In practice that
does not fail too ofter ... or you have to clean build again.

If you really need to pin debian down to what it fetches, because for
some reason (like repro build) you need your own mirror. In fact Isar
spits out a partial debian mirror after an "online" build (base-apt).
That can be used for consecutive offline builds, or as a base for
consecutive "online" builds with custom DISTRO_APT_SOURCES.

While snapshots.debian.org can be used as DISTRO_APT_SOURCES mirror in
theory ... in practice it has rate-limiting in place. So you might
succeed in a manual build that you retry over and over (or a small
image), but in CI without caching ... you will never get a big image to
build. That rate-limiting issue will need to be discussed with
snapshots, we are not the first ones to have issues with it.
But i personally would tell people to simply not freeze if they can,
and the ones that need to freeze i would in fact tell to get a full
debian mirror of their own, instead of a partial one produced by isars
base-apt.
As an OSS project you might see less of a need of freezing, tracking in
fact is a security feature ... and debian will not do much more than
security on their stable distros.

regards,
Henning

[ydi...@free.fr]

This is more of a concern for reproducibility of the build process
at package level. Probably this was not published very widely, but
there has been work on Debian package reproducibility in the context of
Qubes already, including a solution to the snapshots.d.o problem:

https://forum.qubes-os.org/t/reproducible-builds-for-debian-a-big-step-forward/6800

Best regards,
--
Yann

[Henning Schild]

Am Tue, 26 Oct 2021 22:48:53 +0200 (CEST)
schrieb ydi...@free.fr:

Cool stuff! I will give that service a try and look into where the tool
debrebuild comes into play. A tool that might in itself add some value.

But to me it all seems like a workaround of the rate-limiting on
snapshots. A problem that can maybe be solved with funding, in case it
is a cost-problem. In Siemens people end up with their own
implementations of snapshots, which in sum likely costs more than
funding the real one for public non-limited use (together with other
stakeholders).
Still getting funding right and stable might be harder than any costly
workaround downstream.
I think i also heard trust arguments against snapshots, but those might
just be pointless. Even if it is not-Debian (is it?) there will be gpg
in place, but then you still need to trust that you get what you asked
for because if a snapshot holds back a security update gpg would not be
able to tell.

But yes, good stuff! I bet that can be used in Isar somehow, but maybe
only in your layers. You might not want the load from others on your
infra. And the amd64 limitation is kind of severe. We have arm64
becoming more and more relevant and even riscv working for early
prototypes. And i686 and arm32 floating around in places for some time
to be maintained.

Henning

> Best regards,

[ydi...@free.fr]

----- Mail original -----
> De: "Henning Schild" <henning...@siemens.com>
> À: ydi...@free.fr
> Cc: "Anton Mikanovich" <ami...@ilbers.de>, isar-...@googlegroups.com
> Envoyé: Mardi 26 Octobre 2021 23:15:45
> Objet: Re: isar-bootstrap

Yes it is started as an independent service (from a DD, as snapshots.debian.net)
but has been an official one for some time now.
And it hosts the original Debian-signed indices, so that's just
"trusting Debian" at all levels.

> but then you still need to trust that you get what you
> asked
> for because if a snapshot holds back a security update gpg would not
> be
> able to tell.
>
> But yes, good stuff! I bet that can be used in Isar somehow, but
> maybe
> only in your layers. You might not want the load from others on your
> infra. And the amd64 limitation is kind of severe. We have arm64
> becoming more and more relevant and even riscv working for early
> prototypes. And i686 and arm32 floating around in places for some
> time
> to be maintained.

Well, that limitation I guess is mostly that it's the only arch used in
Qubes today, I assume it should work with other archs too, as long as
someone funds a server :)

[Jan Kiszka]

Missing non-amd64 support is exactly the reason why we are not yet
broadly using that alternative snapshot - too many of our use cases are
armhf or arm64 (extremely few are still i386). Compared to that, [1] is
then only a minor issue.

Reproducibility is a strategic topic for us, just not yet one with
highest urgency. However, people start to look into this in Isar context
already (see e.g. [2]).

Jan

[1] https://github.com/fepitre/debian-snapshot/issues/2
[2] https://groups.google.com/g/isar-users/c/GXYNmEpn-ns/m/tb7MKx3FAAAJ

--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux