[PATCH] meta: Allow to specify the snapshot date in the upstream format.

[alexander...@siemens.com]

From: Alexander Heinisch <alexander...@siemens.com>

Currently, it is only possible to specify a unix timestamp to resolve
the snapshot version used. Since this is not very user friendly, most
downstream projects put comments like `# 20240702T082400Z` next to the
snapshot assignment `ISAR_APT_SNAPSHOT_TIMESTAMP = ...`.

This patch makes it easier for users to specify the snapshot version
in a more natural way by `APT_SNAPSHOT_DATE = 20240702T082400Z`

Signed-off-by: Alexander Heinisch <alexander...@siemens.com>
---
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 12f32ff0..125554e1 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -34,7 +34,7 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
FILESEXTRAPATHS:append = ":${BBPATH}"
-APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}"
+APT_SNAPSHOT_DATE ?= "${@ get_apt_snapshot_date(d)}"

inherit deb-dl-dir

--
2.43.0

[MOESSBAUER, Felix]

On Mon, 2024-10-07 at 12:47 +0200, alexander...@siemens.com
wrote:

> From: Alexander Heinisch <alexander...@siemens.com>
>
> Currently, it is only possible to specify a unix timestamp to resolve
> the snapshot version used. Since this is not very user friendly, most
> downstream projects put comments like `# 20240702T082400Z` next to
> the
> snapshot assignment `ISAR_APT_SNAPSHOT_TIMESTAMP = ...`.
>
> This patch makes it easier for users to specify the snapshot version
> in a more natural way by `APT_SNAPSHOT_DATE = 20240702T082400Z`

Thanks!

Acked-by: Felix Moessbauer <felix.mo...@siemens.com>

>
> Signed-off-by: Alexander Heinisch <alexander...@siemens.com>
> ---
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index 12f32ff0..125554e1 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -34,7 +34,7 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if
> bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
>  BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if
> bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
>  BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if
> bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else
> 'BASE_DISTRO')}"
>  FILESEXTRAPATHS:append = ":${BBPATH}"
> -APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}"
> +APT_SNAPSHOT_DATE ?= "${@ get_apt_snapshot_date(d)}"
>  
>  inherit deb-dl-dir
>  

--

Siemens AG, Technology
Linux Expert Center

[Jan Kiszka]

Nicely minimal invasive - but now you should also document this, just
like ISAR_APT_SNAPSHOT_TIMESTAMP. And naming should be aligned.

Jan

[Heinisch, Alexander]

> Nicely minimal invasive - but now you should also document this, just like > ISAR_APT_SNAPSHOT_TIMESTAMP. And naming should be aligned.

Sure. Just wanted to get feedback early, before spending much time on docs 😊.

[Jan Kiszka]

On 07.10.24 13:21, Heinisch, Alexander (T CED SES-AT) wrote:
>> Nicely minimal invasive - but now you should also document this, just like > ISAR_APT_SNAPSHOT_TIMESTAMP. And naming should be aligned.
>
> Sure. Just wanted to get feedback early, before spending much time on docs 😊.

It will definitely help with the kconfig approach, avoid non-working
conversion attempts there.

Let's just rename the var to ISAR_APT_SNAPSHOT_DATE and also clarify in
the doc that to masks ISAR_APT_SNAPSHOT_TIMESTAMP when both are provided.

Jan

[alexander...@siemens.com]

From: Alexander Heinisch <alexander...@siemens.com>

Currently, it is only possible to specify a unix timestamp to resolve
the snapshot version used. Since this is not very user friendly, most
downstream projects put comments like `# 20240702T082400Z` next to the
snapshot assignment `ISAR_APT_SNAPSHOT_TIMESTAMP = ...`.

This patch makes it easier for users to specify the snapshot version
in a more natural way by `APT_SNAPSHOT_DATE = 20240702T082400Z`

Signed-off-by: Alexander Heinisch <alexander...@siemens.com>
---

doc/user_manual.md | 3 ++-
kas/opt/Kconfig | 2 +-
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 706c74bf..bcfea5a0 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -435,7 +435,8 @@ Some other variables include:
- `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional.
`ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional.
- `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros.
- - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten.
+ - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten. (Consider `APT_SNAPSHOT_DATE` for a more user friendly format)
+ - `APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g. `20240702T082400Z`) of the apt snapshot. Automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP`
- `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping.
- `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS.
- `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable.
diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig
index 65482e3a..802bb1a0 100644
--- a/kas/opt/Kconfig
+++ b/kas/opt/Kconfig
@@ -44,7 +44,7 @@ config USE_APT_SNAPSHOT
bool "Use the distros snapshot mirror"
depends on !MIRROR_DEBIAN && (DEBIAN_BUSTER || DEBIAN_BULLSEYE || DEBIAN_BOOKWORM || DEBIAN_TRIXIE || DEBIAN_SID || UBUNTU_FOCAL || UBUNTU_JAMMY)
help
- Use a snapshot mirror for the selected distribution. The date is defined by ISAR_APT_SNAPSHOT_TIMESTAMP.
+ Use a snapshot mirror for the selected distribution. The date is defined by APT_SNAPSHOT_DATE.

config KAS_INCLUDE_APT_SNAPSHOT
string

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 12f32ff0..125554e1 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -34,7 +34,7 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
FILESEXTRAPATHS:append = ":${BBPATH}"
-APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}"
+APT_SNAPSHOT_DATE ?= "${@ get_apt_snapshot_date(d)}"

inherit deb-dl-dir

--
2.43.0

[Heinisch, Alexander]

> It will definitely help with the kconfig approach, avoid non-working conversion attempts there.
>
> Let's just rename the var to ISAR_APT_SNAPSHOT_DATE and also clarify in the doc that to masks ISAR_APT_SNAPSHOT_TIMESTAMP when both are provided.

Renaming APT_SNAPSHOT_DATE makes sense!, also won't break downstream! Will add this.

BR Alexander

[Anton Mikanovich]

07/10/2024 14:27, alexander.heinisch via isar-users wrote:
> From: Alexander Heinisch <alexander...@siemens.com>
>
> Currently, it is only possible to specify a unix timestamp to resolve
> the snapshot version used. Since this is not very user friendly, most
> downstream projects put comments like `# 20240702T082400Z` next to the
> snapshot assignment `ISAR_APT_SNAPSHOT_TIMESTAMP = ...`.
>
> This patch makes it easier for users to specify the snapshot version
> in a more natural way by `APT_SNAPSHOT_DATE = 20240702T082400Z`
>
> Signed-off-by: Alexander Heinisch <alexander...@siemens.com>

Please mark later patchset versions with v3, v4, etc, to make it easier
to track changes between patchset versions.

[alexander...@siemens.com]

From: Alexander Heinisch <alexander...@siemens.com>

Currently, it is only possible to specify a unix timestamp to resolve
the snapshot version used. Since this is not very user friendly, most
downstream projects put comments like `# 20240702T082400Z` next to the
snapshot assignment `ISAR_APT_SNAPSHOT_TIMESTAMP = ...`.

This patch makes it easier for users to specify the snapshot version

in a more natural way by `ISAR_APT_SNAPSHOT_DATE = 20240702T082400Z`

Signed-off-by: Alexander Heinisch <alexander...@siemens.com>
---
v2: Added documentation
v3: Renamed APT_SNAPSHOT_DATE to ISAR_APT_SNAPSHOT_DATE

doc/user_manual.md | 3 ++-
kas/opt/Kconfig | 2 +-

meta-isar/conf/distro/ubuntu-common.inc | 2 +-
meta/conf/distro/debian-common.conf | 2 +-
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++--
5 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 706c74bf..cf3c3b12 100644

--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -435,7 +435,8 @@ Some other variables include:
- `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional.
`ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional.
- `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros.
- - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten.

+ - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user friendly format)
+ - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g. `20240702T082400Z`) of the apt snapshot. Overrides `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP`

- `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping.
- `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS.
- `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable.
diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig

index 65482e3a..1e56c29f 100644

--- a/kas/opt/Kconfig
+++ b/kas/opt/Kconfig
@@ -44,7 +44,7 @@ config USE_APT_SNAPSHOT
bool "Use the distros snapshot mirror"
depends on !MIRROR_DEBIAN && (DEBIAN_BUSTER || DEBIAN_BULLSEYE || DEBIAN_BOOKWORM || DEBIAN_TRIXIE || DEBIAN_SID || UBUNTU_FOCAL || UBUNTU_JAMMY)
help
- Use a snapshot mirror for the selected distribution. The date is defined by ISAR_APT_SNAPSHOT_TIMESTAMP.

+ Use a snapshot mirror for the selected distribution. The date is defined by ISAR_APT_SNAPSHOT_DATE.

config KAS_INCLUDE_APT_SNAPSHOT
string
diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc
index 232bee35..be7fd047 100644
--- a/meta-isar/conf/distro/ubuntu-common.inc
+++ b/meta-isar/conf/distro/ubuntu-common.inc
@@ -37,4 +37,4 @@ SYSTEMD_BOOTLOADER_INSTALL:focal = "systemd:${DISTRO_ARCH}"
SYSTEMD_BOOTLOADER_INSTALL:jammy = "systemd:${DISTRO_ARCH}"

# snapshot mirror for reproducible builds
-DISTRO_APT_SNAPSHOT_PREMIRROR ??= "(http|https)://archive.ubuntu.com/(.*) https://snapshot.ubuntu.com/\2/${APT_SNAPSHOT_DATE}/\n"
+DISTRO_APT_SNAPSHOT_PREMIRROR ??= "(http|https)://archive.ubuntu.com/(.*) https://snapshot.ubuntu.com/\2/${ISAR_APT_SNAPSHOT_DATE}/\n"
diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
index 8c6a3f63..65b5b083 100644
--- a/meta/conf/distro/debian-common.conf
+++ b/meta/conf/distro/debian-common.conf
@@ -40,4 +40,4 @@ COMPAT_DISTRO_ARCH:amd64 = "i386"
COMPAT_DISTRO_ARCH:arm64 = "armhf"

# snapshot mirror for reproducible builds
-DISTRO_APT_SNAPSHOT_PREMIRROR ??= "deb.debian.org/(.*) snapshot.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n"
+DISTRO_APT_SNAPSHOT_PREMIRROR ??= "deb.debian.org/(.*) snapshot.debian.org/archive/\1/${ISAR_APT_SNAPSHOT_DATE}/\n"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 12f32ff0..92974eb5 100644

--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -34,7 +34,7 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
FILESEXTRAPATHS:append = ":${BBPATH}"
-APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}"

+ISAR_APT_SNAPSHOT_DATE ?= "${@ get_isar_apt_snapshot_date(d)}"

inherit deb-dl-dir

@@ -110,7 +110,7 @@ def parse_aptsources_list_line(source_list_line):

return [type, options, source, suite, components]

-def get_apt_snapshot_date(d):
+def get_isar_apt_snapshot_date(d):
import time
source_date_epoch = d.getVar('ISAR_APT_SNAPSHOT_TIMESTAMP')
return time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch)))
--
2.43.0

[Jan Kiszka]

Reviewed-by: Jan Kiszka <jan.k...@siemens.com>

Thanks,

[Uladzimir Bely]

On Mon, 2024-10-07 at 14:27 +0200, alexander.heinisch via isar-users
wrote:

Applied to next, thanks.

--
Best regards,
Uladzimir.