[PATCH 1/2] fix: lazy expand ISAR_APT_SNAPSHOT_DATE[flag]

[Felix Moessbauer]

The ISAR_APT_SNAPSHOT_DATE variable controls the timestamp of the
sources on the snapshot mirror. This variable further is a shortcut for
ISAR_APT_SNAPSHOT_TIMESTAMP that avoids letting the user compute a unix
timestamp of the desired date. However, while
ISAR_APT_SNAPSHOT_TIMESTAMP is always available via bitbake conf, the
_DATE variant is only available after computing it in do_bootstrap.

In 394e954, support was added to just set the timestamp / date for the
security distribution by extending the _TIMESTAMP and _DATE variables
with a flag. However, the logic that evaluates the variable in
DISTRO_APT_SNAPSHOT_PREMIRROR is defined in the distro conf, which is
parsed earlier than the logic in do_bootstrap. By that, depending on the
context the variable might not be set leading to incorrect substitutions
of the apt-sources. This has not been observed in actual images, but in
the devshell and when printing the variable with bitbake.

Anyways, the value of the variable should not depend on the parsing
order. To fix this, we now do not expand the variable until using it.

Fixes: 394e9540 ("snapshots: add option to use separate timestamp ...")
Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
meta/conf/distro/debian-common.conf | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
index 8da17d6f..92daddc3 100644
--- a/meta/conf/distro/debian-common.conf
+++ b/meta/conf/distro/debian-common.conf
@@ -45,6 +45,6 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:bullseye = " usrmerge"

# snapshot mirror for reproducible builds
DISTRO_APT_SNAPSHOT_PREMIRROR ??= " \
- deb.debian.org/(debian-security)/? snapshot.debian.org/archive/\1/${@d.getVarFlag('ISAR_APT_SNAPSHOT_DATE', 'security')}\n \
+ deb.debian.org/(debian-security)/? snapshot.debian.org/archive/\1/${@d.getVarFlag('ISAR_APT_SNAPSHOT_DATE', 'security', expand=False)}\n \
deb.debian.org/(.*)/? snapshot.debian.org/archive/\1/${ISAR_APT_SNAPSHOT_DATE}\n \
"
--
2.51.0

[Felix Moessbauer]

The final snapshot timestamp can be specified either by setting
ISAR_APT_SNAPSHOT_TIMESTAMP or by setting the otherwise derived variable
ISAR_APT_SNAPSHOT_DATE. However, the corresponding distribution specific
flag (e.g. [security]) always reads the value from the TIMESTAMP
component, hence if only ISAR_APT_SNAPSHOT_DATE is set, the date for the
security distribution is still computed from the
ISAR_APT_SNAPSHOT_TIMESTAMP which itself is derived from the source date
epoch.

By that, the timestamp of the security component was wrong if only
ISAR_APT_SNAPSHOT_DATE is set but not ISAR_APT_SNAPSHOT_DATE[security].

We fix it by tracking where the value comes from and set the flagged
version according to the parent value if not set.

Fixes: 394e9540 ("snapshots: add option to use separate timestamp ...")

Reported-by: Clara Kowalsky <clara.k...@siemens.com>

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-isar/conf/distro/ubuntu-common.inc | 2 +-
meta/classes/bootstrap.bbclass | 11 +++++++++--
meta/conf/distro/debian-common.conf | 4 ++--
3 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc
index 88a3b4f0..f1e8a1d6 100644
--- a/meta-isar/conf/distro/ubuntu-common.inc
+++ b/meta-isar/conf/distro/ubuntu-common.inc
@@ -44,4 +44,4 @@ SYSTEMD_BOOTLOADER_INSTALL:focal = "systemd:${DISTRO_ARCH}"
SYSTEMD_BOOTLOADER_INSTALL:jammy = "systemd:${DISTRO_ARCH}"

# snapshot mirror for reproducible builds

-DISTRO_APT_SNAPSHOT_PREMIRROR ??= "(http|https)://archive.ubuntu.com/(.*) https://snapshot.ubuntu.com/\2/${ISAR_APT_SNAPSHOT_DATE}\n"
+DISTRO_APT_SNAPSHOT_PREMIRROR ??= "(http|https)://archive.ubuntu.com/(.*) https://snapshot.ubuntu.com/\2/${ISAR_APT_SNAPSHOT_DATE_INTERNAL}\n"
diff --git a/meta/classes/bootstrap.bbclass b/meta/classes/bootstrap.bbclass
index 64702d5d..03014cb8 100644
--- a/meta/classes/bootstrap.bbclass
+++ b/meta/classes/bootstrap.bbclass
@@ -27,8 +27,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO_ARCH')}"
-ISAR_APT_SNAPSHOT_DATE ?= "${@ get_isar_apt_snapshot_date(d)}"
-ISAR_APT_SNAPSHOT_DATE[security] ?= "${@ get_isar_apt_snapshot_date(d, 'security')}"
+# use internal variables to track where the value comes from
+ISAR_APT_SNAPSHOT_DATE_INTERNAL ?= "${@ get_isar_apt_snapshot_date(d)}"
+ISAR_APT_SNAPSHOT_DATE_INTERNAL[security] ?= "${@ get_isar_apt_snapshot_date(d, 'security')}"

python () {
distro_bootstrap_keys = (d.getVar("DISTRO_BOOTSTRAP_KEYS") or "").split()
@@ -104,6 +105,12 @@ def parse_aptsources_list_line(source_list_line):

def get_isar_apt_snapshot_date(d, dist=None):
import time
+ snapshot_date = d.getVar('ISAR_APT_SNAPSHOT_DATE')
+ if dist:
+ snapshot_date = d.getVarFlag('ISAR_APT_SNAPSHOT_DATE', dist) or snapshot_date
+ if snapshot_date:
+ return snapshot_date
+ # no ISAR_APT_SNAPSHOT_DATE set, compute from timestamp
source_date_epoch = d.getVar('ISAR_APT_SNAPSHOT_TIMESTAMP')
if dist:
source_date_epoch = d.getVarFlag('ISAR_APT_SNAPSHOT_TIMESTAMP', dist) or source_date_epoch
diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
index 92daddc3..cd72837a 100644

--- a/meta/conf/distro/debian-common.conf
+++ b/meta/conf/distro/debian-common.conf
@@ -45,6 +45,6 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:bullseye = " usrmerge"

# snapshot mirror for reproducible builds
DISTRO_APT_SNAPSHOT_PREMIRROR ??= " \

- deb.debian.org/(debian-security)/? snapshot.debian.org/archive/\1/${@d.getVarFlag('ISAR_APT_SNAPSHOT_DATE', 'security', expand=False)}\n \
- deb.debian.org/(.*)/? snapshot.debian.org/archive/\1/${ISAR_APT_SNAPSHOT_DATE}\n \
+ deb.debian.org/(debian-security)/? snapshot.debian.org/archive/\1/${@d.getVarFlag('ISAR_APT_SNAPSHOT_DATE_INTERNAL', 'security', expand=False)}\n \
+ deb.debian.org/(.*)/? snapshot.debian.org/archive/\1/${ISAR_APT_SNAPSHOT_DATE_INTERNAL}\n \
"
--
2.51.0

[Florian Bezdeka]

On Mon, 2025-10-06 at 16:12 +0200, 'Felix Moessbauer' via isar-users
wrote:

> The ISAR_APT_SNAPSHOT_DATE variable controls the timestamp of the
> sources on the snapshot mirror. This variable further is a shortcut for
> ISAR_APT_SNAPSHOT_TIMESTAMP that avoids letting the user compute a unix
> timestamp of the desired date. However, while
> ISAR_APT_SNAPSHOT_TIMESTAMP is always available via bitbake conf, the
> _DATE variant is only available after computing it in do_bootstrap.
>
> In 394e954, support was added to just set the timestamp / date for the
> security distribution by extending the _TIMESTAMP and _DATE variables

^^^^^^^^^^^^^^^^^^^^
Wording: distribution -> repository?

According to https://wiki.debian.org/SourcesList this seems the right
term here.

> with a flag. However, the logic that evaluates the variable in
> DISTRO_APT_SNAPSHOT_PREMIRROR is defined in the distro conf, which is
> parsed earlier than the logic in do_bootstrap. By that, depending on the
> context the variable might not be set leading to incorrect substitutions
> of the apt-sources. This has not been observed in actual images, but in
> the devshell and when printing the variable with bitbake.
>
> Anyways, the value of the variable should not depend on the parsing
> order. To fix this, we now do not expand the variable until using it.

Agree.

>
> Fixes: 394e9540 ("snapshots: add option to use separate timestamp ...")
> Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
> ---
> meta/conf/distro/debian-common.conf | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
> index 8da17d6f..92daddc3 100644
> --- a/meta/conf/distro/debian-common.conf
> +++ b/meta/conf/distro/debian-common.conf
> @@ -45,6 +45,6 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:bullseye = " usrmerge"
>
> # snapshot mirror for reproducible builds
> DISTRO_APT_SNAPSHOT_PREMIRROR ??= " \
> - deb.debian.org/(debian-security)/? snapshot.debian.org/archive/\1/${@d.getVarFlag('ISAR_APT_SNAPSHOT_DATE', 'security')}\n \
> + deb.debian.org/(debian-security)/? snapshot.debian.org/archive/\1/${@d.getVarFlag('ISAR_APT_SNAPSHOT_DATE', 'security', expand=False)}\n \
> deb.debian.org/(.*)/? snapshot.debian.org/archive/\1/${ISAR_APT_SNAPSHOT_DATE}\n \
> "
> --
> 2.51.0
>

> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251006141227.1017941-1-felix.moessbauer%40siemens.com.

[MOESSBAUER, Felix]

On Tue, 2025-10-07 at 12:39 +0200, Florian Bezdeka wrote:
> On Mon, 2025-10-06 at 16:12 +0200, 'Felix Moessbauer' via isar-users
> wrote:
> > The ISAR_APT_SNAPSHOT_DATE variable controls the timestamp of the
> > sources on the snapshot mirror. This variable further is a shortcut for
> > ISAR_APT_SNAPSHOT_TIMESTAMP that avoids letting the user compute a unix
> > timestamp of the desired date. However, while
> > ISAR_APT_SNAPSHOT_TIMESTAMP is always available via bitbake conf, the
> > _DATE variant is only available after computing it in do_bootstrap.
> >
> > In 394e954, support was added to just set the timestamp / date for the
> > security distribution by extending the _TIMESTAMP and _DATE variables
> ^^^^^^^^^^^^^^^^^^^^
> Wording: distribution -> repository?
>
> According to https://wiki.debian.org/SourcesList this seems the right
> term here.

But there they also state:

Apt downloads and installs packages from one or more software
repositories (sometimes called suites, distributions, versions, or even
sources).

As we currently call this dist / distribution in the source code, I
prefer to also use that term in the commit message.

If a debian policy states the name, I would change it, but just because
it is mentioned in the wiki it not enough to require the change.

Felix

--
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

[Florian Bezdeka]

On Tue, 2025-10-07 at 14:39 +0000, Moessbauer, Felix (FT RPD CED OES-

DE) wrote:
> On Tue, 2025-10-07 at 12:39 +0200, Florian Bezdeka wrote:
> > On Mon, 2025-10-06 at 16:12 +0200, 'Felix Moessbauer' via isar-users
> > wrote:
> > > The ISAR_APT_SNAPSHOT_DATE variable controls the timestamp of the
> > > sources on the snapshot mirror. This variable further is a shortcut for
> > > ISAR_APT_SNAPSHOT_TIMESTAMP that avoids letting the user compute a unix
> > > timestamp of the desired date. However, while
> > > ISAR_APT_SNAPSHOT_TIMESTAMP is always available via bitbake conf, the
> > > _DATE variant is only available after computing it in do_bootstrap.
> > >
> > > In 394e954, support was added to just set the timestamp / date for the
> > > security distribution by extending the _TIMESTAMP and _DATE variables
> > ^^^^^^^^^^^^^^^^^^^^
> > Wording: distribution -> repository?
> >
> > According to https://wiki.debian.org/SourcesList this seems the right
> > term here.
>
> But there they also state:
>
> Apt downloads and installs packages from one or more software
> repositories (sometimes called suites, distributions, versions, or even
> sources).

Examples:

Distribution: Debian
Suites: trixie trixie-updates
Components: main non-free-firmware

Repository: The package feed for one specific combination of the
definitions above.

Wording is important here as this bug might trigger security
implications. We might not have build what the user expected. So we
have to clearly communicate what the problem was - by using the right
wording.

[MOESSBAUER, Felix]

I leave it up to the maintainers to change the commit message.
Again, if you can cite the location in the debian policy where this is
stated I will immediately change it. But if that is not specified,
there is no "right wording".

Felix

[Zhihang Wei]

Both patches were applied to next, thanks.