[PATCH 0/5] Make ext4 images reproducible

[venkat...@toshiba-tsip.com]

From: venkata pyla <venkat...@toshiba-tsip.com>

Hi,

The following series of patches will help to fix reproducible build
issues in ext4 file systems.

Verified with following steps:

```

$ git clone http://github.com/ilbers/isar.git
$ cd isar
$ . isar-init-build-env ../build

configure image parameters
$ cd isar
$ cat <<EOT >> conf/local.conf
SOURCE_DATE_EPOCH = "$(git log -1 --pretty=%ct)"
IMAGE_FSTYPES = "wic"
WIC_DEPLOY_PARTITIONS = "1"
EOT

# First Build
$ cd ../build
$ sudo rm -rf tmp sstate-cache
$ bitbake mc:qemuamd64-bookworm:isar-image-base
$ mv tmp/deploy/images/qemuamd64/ ../qemuamd64-1

# Second Build
$ sudo rm -rf tmp sstate-cache
$ bitbake mc:qemuamd64-bookworm:isar-image-base
$ mv tmp/deploy/images/qemuamd64/ ../qemuamd64-2

# compare ext4 images with diffoscope
$ diffoscope qemuamd64-1/isar-image-base-debian-bookworm-qemuamd64.wic.p2 \
qemuamd64-2/isar-image-base-debian-bookworm-qemuamd64.wic.p2

```

venkata pyla (5):
linux-module: Use debug-prefix-map to build modules deterministically
wic: Set file system uuid to ext4 partitions
wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible
ext4 images
wic: honor the SOURCE_DATE_EPOCH in case of updated fstab
wic: make ext2/3/4 images reproducible

.../lib/wic/canned-wks/common-isar.wks.inc | 2 +-
.../lib/wic/canned-wks/de0-nano-soc.wks.in | 2 +-
.../lib/wic/canned-wks/efi-plus-pcbios.wks | 2 +-
.../scripts/lib/wic/canned-wks/hikey.wks | 2 +-
.../lib/wic/canned-wks/multipart-efi.wks | 4 +-
.../lib/wic/canned-wks/nanopi-neo-efi.wks.in | 2 +-
.../lib/wic/canned-wks/nanopi-neo.wks.in | 2 +-
.../scripts/lib/wic/canned-wks/rpi-sdimg.wks | 2 +-
.../wic/canned-wks/sdimage-efi-sb-debian.wks | 2 +-
.../lib/wic/canned-wks/sdimage-efi-sd.wks | 2 +-
.../lib/wic/canned-wks/sdimage-efi.wks | 2 +-
.../lib/wic/canned-wks/sifive-fu540.wks | 2 +-
.../canned-wks/starfive-visionfive2.wks.in | 2 +-
.../lib/wic/canned-wks/stm32mp15x.wks.in | 2 +-
.../linux-module/files/debian/rules.tmpl | 5 ++-
scripts/lib/wic/partition.py | 37 ++++++++++++++++++-
scripts/lib/wic/plugins/imager/direct.py | 3 ++
scripts/lib/wic/plugins/source/rootfs.py | 2 +-
18 files changed, 59 insertions(+), 18 deletions(-)

--
2.20.1

[venkat...@toshiba-tsip.com]

From: venkata pyla <venkat...@toshiba-tsip.com>

The custom linux modules are built non-determinstically due to the
reason that the kernel build includes absolute paths in the debug
information which is not necessary and can be trimmed using
`debug-prefix-map` option.

Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com>
---
meta/recipes-kernel/linux-module/files/debian/rules.tmpl | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/linux-module/files/debian/rules.tmpl b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
index 7d950e38..20d5f670 100755
--- a/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
+++ b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
@@ -45,11 +45,14 @@ KERNEL_DEP := $(shell dpkg-query -W -f '$${Depends}' ${KERNEL_HEADERS_PKG} | sed
KDIR := $(shell dpkg -L $(KERNEL_DEP) | grep "/lib/modules/.*/build")
endif

+KCFLAGS := "-fdebug-prefix-map=$(PWD)=."
+KAFLAGS := "-fdebug-prefix-map=$(PWD)=."
+
override_dh_auto_clean:
$(MAKE) -C $(KDIR) M=$(PWD) clean

override_dh_auto_build:
- $(MAKE) -C $(KDIR) M=$(PWD) $(PARALLEL_MAKE) modules
+ $(MAKE) -C $(KDIR) M=$(PWD) $(PARALLEL_MAKE) KCFLAGS="$(KCFLAGS)" KAFLAGS="$(KAFLAGS)" modules
ifneq ($(filter pkg.sign,$(DEB_BUILD_PROFILES)),)
find . -name "*.ko" -print -exec $(KDIR)/scripts/sign-file ${SIGNATURE_HASHFN} ${SIGNATURE_KEYFILE} ${SIGNATURE_CERTFILE} {} \;
endif
--
2.20.1

[venkat...@toshiba-tsip.com]

From: venkata pyla <venkat...@toshiba-tsip.com>

Adding pre configured UUID's to the file systems will helps to build
reproducible images.

Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com>
---

meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc | 2 +-
meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in | 2 +-
meta-isar/scripts/lib/wic/canned-wks/efi-plus-pcbios.wks | 2 +-
meta-isar/scripts/lib/wic/canned-wks/hikey.wks | 2 +-
meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks | 4 ++--
meta-isar/scripts/lib/wic/canned-wks/nanopi-neo-efi.wks.in | 2 +-
meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in | 2 +-
meta-isar/scripts/lib/wic/canned-wks/rpi-sdimg.wks | 2 +-
.../scripts/lib/wic/canned-wks/sdimage-efi-sb-debian.wks | 2 +-
meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks | 2 +-
meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks | 2 +-
meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks | 2 +-
.../scripts/lib/wic/canned-wks/starfive-visionfive2.wks.in | 2 +-
meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in | 2 +-
14 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc
index fe028b50..71885cb4 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc
+++ b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc
@@ -1,3 +1,3 @@
# This file is included into 3 canned wks files from this directory
part /boot --source bootimg-pcbios-isar --ondisk sda --label boot --active --align 1024
-part / --source rootfs --use-uuid --fstype=ext4 --label platform --align 1024 --exclude-path=boot --mkfs-extraopts "-T default"
+part / --source rootfs --use-uuid --fstype=ext4 --label platform --align 1024 --exclude-path=boot --mkfs-extraopts "-T default" --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
diff --git a/meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in b/meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in
index 0794a2f3..06154e01 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in
+++ b/meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in
@@ -5,6 +5,6 @@

part --source rawcopy --sourceparams "file=/usr/lib/u-boot/${MACHINE}/${U_BOOT_BIN}" --system-id 0xa2 --align 1

-part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --active
+part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --active --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002

bootloader --append "rw rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/efi-plus-pcbios.wks b/meta-isar/scripts/lib/wic/canned-wks/efi-plus-pcbios.wks
index 03928915..321be10b 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/efi-plus-pcbios.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/efi-plus-pcbios.wks
@@ -1,6 +1,6 @@
# Example to show how to create an efi + pcbios image
# Note, that the loader argument is mandatory. But systemd-boot also works.
part /boot --source bootimg-biosplusefi-isar --sourceparams="loader=grub-efi" --label boot --active --align 1024
-part / --source rootfs --ondisk sda --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --exclude-path=boot
+part / --source rootfs --ondisk sda --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --exclude-path=boot --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002

bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks
index 0c966e65..27447e2d 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks
@@ -5,6 +5,6 @@

part /boot --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 --align 1024

-part / --source rootfs --ondisk mmcblk1 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/
+part / --source rootfs --ondisk mmcblk1 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/ --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002

bootloader --ptable gpt --timeout=3
diff --git a/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks b/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks
index b3abb3d0..ac8ebea9 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks
@@ -1,8 +1,8 @@
# Example showing how to split a single rootfs across partitions
part /boot --ondisk sda --source bootimg-efi-isar --sourceparams="loader=grub-efi" --label boot --active --align 1024
# when excluding paths, just specify --exclude-path once and separate paths using spaces
-part / --ondisk sda --source rootfs --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --exclude-path boot home --extra-space 1G
+part / --ondisk sda --source rootfs --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --exclude-path boot home --extra-space 1G --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
# put home last to support auto-expand of this partition
-part /home --ondisk sda --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024
+part /home --ondisk sda --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120004

bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo-efi.wks.in b/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo-efi.wks.in
index 7171a23e..4baf0642 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo-efi.wks.in
+++ b/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo-efi.wks.in
@@ -10,6 +10,6 @@
part u-boot --source rawcopy --sourceparams "file=/usr/lib/u-boot/nanopi_neo/u-boot-sunxi-with-spl.bin" --no-table --align 128

part /boot --use-uuid --source bootimg-efi-isar --sourceparams="loader=systemd-boot" --label efi --part-type EF00 --align 1024
-part / --use-uuid --source rootfs --fstype=ext4 --mkfs-extraopts "-T default" --label platform --active --align 1024 --exclude-path boot
+part / --use-uuid --source rootfs --fstype=ext4 --mkfs-extraopts "-T default" --label platform --active --align 1024 --exclude-path boot --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002

bootloader --ptable gpt --timeout 0 --append "rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in b/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in
index af5b6f08..cfad430e 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in
+++ b/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in
@@ -5,6 +5,6 @@

part u-boot --source rawcopy --sourceparams "file=/usr/lib/u-boot/nanopi_neo/u-boot-sunxi-with-spl.bin" --no-table --align 8

-part / --source rootfs-u-boot --ondisk mmcblk2 --fstype ext4 --mkfs-extraopts "-T default" --sourceparams "builtin_dt=yes" --label platform --align 1024 --active
+part / --source rootfs-u-boot --ondisk mmcblk2 --fstype ext4 --mkfs-extraopts "-T default" --sourceparams "builtin_dt=yes" --label platform --align 1024 --active --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002

bootloader --append "rw rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/rpi-sdimg.wks b/meta-isar/scripts/lib/wic/canned-wks/rpi-sdimg.wks
index e5a3668b..de1b742e 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/rpi-sdimg.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/rpi-sdimg.wks
@@ -1,5 +1,5 @@
part /boot --source bootimg-partition --ondisk mmcblk0 --fstype=vfat --label boot --active --align 4096 --size 10

-part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --mkfs-extraopts "-T default" --label root --align 4096 --exclude-path=boot
+part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --mkfs-extraopts "-T default" --label root --align 4096 --exclude-path=boot --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002

bootloader
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sb-debian.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sb-debian.wks
index cd99ebca..bb2da5ac 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sb-debian.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sb-debian.wks
@@ -5,6 +5,6 @@

part /boot --source bootimg-efi-isar --sourceparams "loader=grub-efi,use-debian-sb-stub=true" --ondisk sda --label efi --part-type EF00 --align 1024

-part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/
+part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/ --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002

bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
index 7fe2953f..6dc9c6c4 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
@@ -4,7 +4,7 @@

part /boot --source bootimg-efi-isar --sourceparams "loader=systemd-boot" --ondisk sda --label efi --part-type EF00 --align 1024

-part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/
+part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/ --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002

include expand-padding.wks.inc

diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
index 80c3a535..8c864f2f 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
@@ -4,7 +4,7 @@

part /boot --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk sda --label efi --part-type EF00 --align 1024

-part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/
+part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/ --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002

include expand-padding.wks.inc

diff --git a/meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks b/meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks
index 9cd0e662..ef99edd7 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks
@@ -5,6 +5,6 @@

part --source rawcopy --sourceparams "file=/usr/lib/opensbi/sifive-fu540/fw_payload.bin" --ondisk mmcblk0 --fixed-size 8M --align 1 --part-type 2e54b353-1271-4842-806f-e436d6af6985

-part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --mkfs-extraopts "-T default" --label platform --active --align 1024 --sourceparams "no_initrd=yes,script_prepend=setenv fdtfile sifive/hifive-unleashed-a00.dtb"
+part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --mkfs-extraopts "-T default" --label platform --active --align 1024 --sourceparams "no_initrd=yes,script_prepend=setenv fdtfile sifive/hifive-unleashed-a00.dtb" --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002

bootloader --ptable gpt --append "rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/starfive-visionfive2.wks.in b/meta-isar/scripts/lib/wic/canned-wks/starfive-visionfive2.wks.in
index 0f3d76a9..c265c752 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/starfive-visionfive2.wks.in
+++ b/meta-isar/scripts/lib/wic/canned-wks/starfive-visionfive2.wks.in
@@ -18,6 +18,6 @@ part --source rawcopy --sourceparams "file=/usr/share/visionfive2-u-boot-firmwar
# EFI
part /boot --source bootimg-efi-isar --sourceparams "loader=systemd-boot" --use-uuid --label efi --part-type EF00 --align 1024
# rootfs
-part / --source rootfs --use-uuid --fstype ext4 --mkfs-extraopts "-T default" --label image --active --align 1024 --exclude-path=boot/
+part / --source rootfs --use-uuid --fstype ext4 --mkfs-extraopts "-T default" --label image --active --align 1024 --exclude-path=boot/ --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002

bootloader --ptable gpt --timeout=0 --append "console=tty0 console=ttyS0,115200 earlycon rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in b/meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in
index 5d96f65f..6e651292 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in
+++ b/meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in
@@ -11,6 +11,6 @@ part teeh --part-name teeh --source rawcopy --sourceparams "file=/usr/lib/optee-
part teed --part-name teed --source rawcopy --sourceparams "file=/usr/lib/optee-os/${MACHINE}/tee-pageable_v2.stm32" --fstype=ext4 --fsoptions "noauto" --part-type 0x8301 --fixed-size 1024K
part teex --part-name teex --source rawcopy --sourceparams "file=/usr/lib/optee-os/${MACHINE}/tee-pager_v2.stm32" --fstype=ext4 --fsoptions "noauto" --part-type 0x8301 --fixed-size 256K

-part / --source rootfs-u-boot --fstype ext4 --mkfs-extraopts "-T default" --label root --align 1024 --active --use-uuid
+part / --source rootfs-u-boot --fstype ext4 --mkfs-extraopts "-T default" --label root --align 1024 --active --use-uuid --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002

bootloader --ptable gpt --append="rootwait rw console=ttySTM0,115200"
--
2.20.1

[venkat...@toshiba-tsip.com]

From: venkata pyla <venkat...@toshiba-tsip.com>

E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file system.
hash_seed: creates reproducible directory indexes in the file system.

Reference commit in e2fsprogs: e1f7100643a46456be107b33098f6034b0835e6d

Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com>
---

scripts/lib/wic/partition.py | 11 +++++++++++
1 file changed, 11 insertions(+)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index e50871b8..90b2c037 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -280,6 +280,17 @@ class Partition():

extraopts = self.mkfs_extraopts or "-F -i 8192"

+ if os.getenv('SOURCE_DATE_EPOCH'):
+ sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
+ pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" % (sde_time, pseudo)
+
+ # Set hash_seed to generate deterministic directory indexes
+ namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-2f2fdf33d460")
+ if self.fsuuid:
+ namespace = uuid.UUID(self.fsuuid)
+ hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
+ extraopts += " -E hash_seed=%s" % hash_seed
+
label_str = ""
if self.label:
label_str = "-L %s" % self.label
--
2.20.1

[venkat...@toshiba-tsip.com]

From: venkata pyla <venkat...@toshiba-tsip.com>

In case user requested to build a binary repeatable package,
it's required to honor the SOURCE_DATE_EPOCH environment
variable. So forcefully set mtime inside all the routines
which modify fstab in case it is updated.

Taken from openembedded-core: 99719a3712a88dce8450994d995803e126e49115

Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com>
---

scripts/lib/wic/partition.py | 7 ++++++-
scripts/lib/wic/plugins/imager/direct.py | 3 +++
scripts/lib/wic/plugins/source/rootfs.py | 2 +-
3 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 90b2c037..44b1277e 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -305,6 +305,11 @@ class Partition():
f.write("cd etc\n")
f.write("rm fstab\n")
f.write("write %s fstab\n" % (self.updated_fstab_path))

+ if os.getenv('SOURCE_DATE_EPOCH'):

+ fstab_time = int(os.getenv('SOURCE_DATE_EPOCH'))
+ for time in ["atime", "mtime", "ctime"]:
+ f.write("set_inode_field fstab %s %s\n" % (time, hex(fstab_time)))
+ f.write("set_inode_field fstab %s_extra 0\n" % (time))
debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs)
exec_native_cmd(debugfs_cmd, native_sysroot)

@@ -364,7 +369,7 @@ class Partition():
exec_native_cmd(mcopy_cmd, native_sysroot)

if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update:
- mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path)
+ mcopy_cmd = "mcopy -m -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path)
exec_native_cmd(mcopy_cmd, native_sysroot)

chmod_cmd = "chmod 644 %s" % rootfs
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index c44159b2..4d8f9216 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -149,6 +149,9 @@ class DirectPlugin(ImagerPlugin):
self.updated_fstab_path = os.path.join(self.workdir, "fstab")
with open(self.updated_fstab_path, "w") as f:
f.writelines(fstab_lines)

+ if os.getenv('SOURCE_DATE_EPOCH'):

+ fstab_time = int(os.getenv('SOURCE_DATE_EPOCH'))
+ os.utime(self.updated_fstab_path, (fstab_time, fstab_time))

def _full_path(self, path, name, extention):
""" Construct full file path to a file we generate. """
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index fc06312e..e29f3a4c 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -224,7 +224,7 @@ class RootfsPlugin(SourcePlugin):
if part.update_fstab_in_rootfs and part.has_fstab and not part.no_fstab_update:
fstab_path = os.path.join(new_rootfs, "etc/fstab")
# Assume that fstab should always be owned by root with fixed permissions
- install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path)
+ install_cmd = "install -m 0644 -p %s %s" % (part.updated_fstab_path, fstab_path)
if new_pseudo:
pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
else:
--
2.20.1

[venkat...@toshiba-tsip.com]

From: venkata pyla <venkat...@toshiba-tsip.com>

Ext2/3/4 FS contains not only mtime, but also ctime, atime and crtime.
Currently, all the files are being added into the rootfs image using
mkfs -d functionality which affects all the timestamps excluding mtime.
This patch ensures these timestamps inside the FS image equal to
the SOURCE_DATE_EPOCH if it is set.

taken from openembedded-core: 75d2dd0ea7790db2e8ee921784ca373abff2df65

Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com>
---

scripts/lib/wic/partition.py | 29 ++++++++++++++++++++++++-----
1 file changed, 24 insertions(+), 5 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 44b1277e..e7641c9e 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -305,17 +305,36 @@ class Partition():

f.write("cd etc\n")
f.write("rm fstab\n")
f.write("write %s fstab\n" % (self.updated_fstab_path))

- if os.getenv('SOURCE_DATE_EPOCH'):
- fstab_time = int(os.getenv('SOURCE_DATE_EPOCH'))
- for time in ["atime", "mtime", "ctime"]:
- f.write("set_inode_field fstab %s %s\n" % (time, hex(fstab_time)))
- f.write("set_inode_field fstab %s_extra 0\n" % (time))

debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs)
exec_native_cmd(debugfs_cmd, native_sysroot)

mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)

+ if os.getenv('SOURCE_DATE_EPOCH'):

+ sde_time = hex(int(os.getenv('SOURCE_DATE_EPOCH')))
+ debugfs_script_path = os.path.join(cr_workdir, "debugfs_script")
+ files = []
+ for root, dirs, others in os.walk(rootfs_dir):
+ base = root.replace(rootfs_dir, "").rstrip(os.sep)
+ files += [ "/" if base == "" else base ]
+ files += [ base + "/" + n for n in dirs + others ]
+ with open(debugfs_script_path, "w") as f:
+ f.write("set_current_time %s\n" % (sde_time))
+ if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update:
+ f.write("set_inode_field /etc/fstab mtime %s\n" % (sde_time))
+ f.write("set_inode_field /etc/fstab mtime_extra 0\n")
+ for file in set(files):
+ for time in ["atime", "ctime", "crtime"]:
+ f.write("set_inode_field \"%s\" %s %s\n" % (file, time, sde_time))
+ f.write("set_inode_field \"%s\" %s_extra 0\n" % (file, time))
+ for time in ["wtime", "mkfs_time", "lastcheck"]:
+ f.write("set_super_value %s %s\n" % (time, sde_time))
+ for time in ["mtime", "first_error_time", "last_error_time", "kbytes_written"]:
+ f.write("set_super_value %s 0\n" % (time))
+ debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs)
+ exec_native_cmd(debugfs_cmd, native_sysroot)
+
self.check_for_Y2038_problem(rootfs, native_sysroot)

def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
--
2.20.1

[MOESSBAUER, Felix]

On Thu, 2023-12-07 at 21:11 +0530, venkat...@toshiba-tsip.com wrote:
> From: venkata pyla <venkat...@toshiba-tsip.com>
>
> The custom linux modules are built non-determinstically due to the
> reason that the kernel build includes absolute paths in the debug
> information which is not necessary and can be trimmed using
> `debug-prefix-map` option.
>
> Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com>

Hi,

thanks for this. It is basically the same patch I send a year ago for
the kernel recipe.

@Maintainers: This patch needs to be serialized with this one:
"add support for non-default modules dir in linux-module".

Acked-by: Felix Moessbauer <felix.mo...@siemens.com>

Best regards,
Felix

[MOESSBAUER, Felix]

On Thu, 2023-12-07 at 21:11 +0530, venkat...@toshiba-tsip.com wrote:

> From: venkata pyla <venkat...@toshiba-tsip.com>
>
> Hi,
>
> The following series of patches will help to fix reproducible build
> issues in ext4 file systems.
>
> Verified with following steps:
>
> ```
>
> $ git clone http://github.com/ilbers/isar.git
> $ cd isar
> $ . isar-init-build-env ../build
>
> configure image parameters
> $ cd isar
> $ cat <<EOT >> conf/local.conf
> SOURCE_DATE_EPOCH = "$(git log -1 --pretty=%ct)"
> IMAGE_FSTYPES = "wic"
> WIC_DEPLOY_PARTITIONS = "1"
> EOT
>
> # First Build
> $ cd ../build
> $ sudo rm -rf tmp sstate-cache
> $ bitbake mc:qemuamd64-bookworm:isar-image-base
> $ mv tmp/deploy/images/qemuamd64/ ../qemuamd64-1
>
> # Second Build
> $ sudo rm -rf tmp sstate-cache

Hi,

just out of curiosity: Did you also have a look a building with sstate
cache? Are there reproducibility issues?

In the past I fixed a couple of these, but I'm not 100% sure if we
still have some (esp. around the rootfs caching).

Happy Coding!
Felix

[MOESSBAUER, Felix]

On Thu, 2023-12-07 at 21:11 +0530, venkat...@toshiba-tsip.com wrote:

> From: venkata pyla <venkat...@toshiba-tsip.com>
>
> E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file
> system.
> hash_seed: creates reproducible directory indexes in the file system.
>
> Reference commit in e2fsprogs:
> e1f7100643a46456be107b33098f6034b0835e6d

Hi!

Did you also send this patch to OE?
We need to integrate it there as well, as this is the basis for the
ISAR plugins.

Best regards,
Felix

[Venkat...@toshiba-tsip.com]

I just checked, with sstate cache the images are reproducible as well.
In my steps I have cleaned the sstate cache, because it should not hide any issues due to caching.

> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to isar-users+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-
> users/3397cb2d254bc8a2e45c47b5a4094e34c85bec32.camel%40siemens.c
> om.

[Venkat...@toshiba-tsip.com]

> -----Original Message-----
> From: 'MOESSBAUER, Felix' via isar-users <isar-...@googlegroups.com>
> Sent: Friday, December 8, 2023 4:04 PM
> To: isar-...@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
> <Venkat...@toshiba-tsip.com>
> Cc: Kiszka, Jan <jan.k...@siemens.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro...@toshiba.co.jp>; dinesh
> kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh...@toshiba-tsip.com>

> Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to
> generate reproducible ext4 images
>

> On Thu, 2023-12-07 at 21:11 +0530, venkat...@toshiba-tsip.com wrote:
> > From: venkata pyla <venkat...@toshiba-tsip.com>
> >
> > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file
> > system.
> > hash_seed: creates reproducible directory indexes in the file system.
> >
> > Reference commit in e2fsprogs:
> > e1f7100643a46456be107b33098f6034b0835e6d
>
> Hi!
>
> Did you also send this patch to OE?
> We need to integrate it there as well, as this is the basis for the ISAR plugins.

Not shared yet, I will send to OE as well.

> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> isar-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-

> users/813c16c90a28fd387d5dbdb51aa02f98b1ebfa57.camel%40siemens.com.

[Florian Bezdeka]

On Fri, 2023-12-08 at 18:14 +0000, Venkat...@toshiba-tsip.com wrote:
> > -----Original Message-----
> > From: 'MOESSBAUER, Felix' via isar-users <isar-...@googlegroups.com>
> > Sent: Friday, December 8, 2023 4:04 PM
> > To: isar-...@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
> > <Venkat...@toshiba-tsip.com>
> > Cc: Kiszka, Jan <jan.k...@siemens.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
> > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro...@toshiba.co.jp>; dinesh
> > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh...@toshiba-tsip.com>
> > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to
> > generate reproducible ext4 images
> >
> > On Thu, 2023-12-07 at 21:11 +0530, venkat...@toshiba-tsip.com wrote:
> > > From: venkata pyla <venkat...@toshiba-tsip.com>
> > >
> > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file
> > > system.
> > > hash_seed: creates reproducible directory indexes in the file system.
> > >
> > > Reference commit in e2fsprogs:
> > > e1f7100643a46456be107b33098f6034b0835e6d
> >
> > Hi!
> >
> > Did you also send this patch to OE?
> > We need to integrate it there as well, as this is the basis for the ISAR plugins.
>
> Not shared yet, I will send to OE as well.

We should follow "upstream first" here. Once OE has accepted the patch
we will inherit it during next OE -> Isar update cycle.

Maintainers: Do not apply it to Isar directly.

>
> >
> > Best regards,
> > Felix
> >
> > >

[Venkat...@toshiba-tsip.com]

I have shared this patch to OE [1], hope I will get feedback from the community and will support to fix this issue in the OE itself.

[1] https://lists.openembedded.org/g/openembedded-core/message/192192

>
> Maintainers: Do not apply it to Isar directly.
>
> >
> > >
> > > Best regards,
> > > Felix
> > >
> > > >
>

> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to isar-users+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-

> users/f2563bb179d011f22de10d212600f6ef4c0cb284.camel%40siemens.com
> .

[Venkat...@toshiba-tsip.com]

This patch [1] has been merged now in upstream OE-core,
now the required patches should be taken to ISAR, is there any timeline to bring the OE-core changes to ISAR or should we apply the required changes to ISAR manually?

[1] https://github.com/openembedded/openembedded-core/commit/bb822ab75de0020572058090439b93cc56bbf7e0

> >
> > Maintainers: Do not apply it to Isar directly.
> >
> > >
> > > >
> > > > Best regards,
> > > > Felix
> > > >
> > > > >
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to isar-users+...@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/isar-
> > users/f2563bb179d011f22de10d212600f6ef4c0cb284.camel%40siemens.com
> > .
>
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> isar-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-

> users/OS3PR01MB68566626DF8F78E4F73D2A45A48EA%40OS3PR01MB6856.jpn
> prd01.prod.outlook.com.

[Florian Bezdeka]

That's great! Thanks for taking care.

I added two guys from Ilbers, maybe they have an update pending already
or can comment on the timeline. Anton, Uladzimir any comments?

In general it's a good idea to come up with a patch to speed things up.

The last update was done in 64af29bcc4b0 ("wic: Update to the latest
revision"). Might be used as inspiration.

Florian

[Anton Mikanovich]

28/12/2023 10:47, Florian Bezdeka wrote:
> On Thu, 2023-12-28 at 06:02 +0000, Venkat...@toshiba-tsip.com wrote:
>> This patch [1] has been merged now in upstream OE-core,
>> now the required patches should be taken to ISAR, is there any timeline to bring the OE-core changes to ISAR or should we apply the required changes to ISAR manually?
>>
>> [1] https://github.com/openembedded/openembedded-core/commit/bb822ab75de0020572058090439b93cc56bbf7e0
> That's great! Thanks for taking care.
>
> I added two guys from Ilbers, maybe they have an update pending already
> or can comment on the timeline. Anton, Uladzimir any comments?
>
> In general it's a good idea to come up with a patch to speed things up.
>
> The last update was done in 64af29bcc4b0 ("wic: Update to the latest
> revision"). Might be used as inspiration.
>
> Florian

Hello,

We are going to make v0.10 release soon and probably can take care of
externals
(like WIC, OE libs and Bitbake) update right after that.

[Adithya....@toshiba-tsip.com]

Hello Anton,

Hope this message finds you well.

Just wanted to follow up on a previous mail regarding wic and other related updates from openembedded-core. You had mentioned about the updates from openembedded-core to be handled right after the release of ISAR v0.10

Would like to understand if there are any plans for wic and other related updates from openembedded-core.

Thanks and Regards,
Adithya Balakumar

-----Original Message-----
From: Anton Mikanovich <ami...@ilbers.de>
Sent: Thursday, December 28, 2023 3:40 PM
To: Florian Bezdeka <florian...@siemens.com>; pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkat...@toshiba-tsip.com>; felix.mo...@siemens.com; isar-...@googlegroups.com; Uladzimir Bely <ub...@ilbers.de>
Cc: jan.k...@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro...@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh...@toshiba-tsip.com>; venkat...@gmail.com; balakumar adithya(ＴＳＩＰ TEUR) <Adithya....@toshiba-tsip.com>
Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Anton Mikanovich]

13/03/2024 12:01, Adithya....@toshiba-tsip.com wrote:
> Hello Anton,
>
> Hope this message finds you well.
>
> Just wanted to follow up on a previous mail regarding wic and other related updates from openembedded-core. You had mentioned about the updates from openembedded-core to be handled right after the release of ISAR v0.10
>
> Would like to understand if there are any plans for wic and other related updates from openembedded-core.
>
> Thanks and Regards,
> Adithya Balakumar

Hello Adithya,

We've checked and the next OE LTS using bitbake 2.8 should be released
in April.
We could start preparations for the bitbake upgrade and update OE after
that.
Would that suit your needs?

[Adithya....@toshiba-tsip.com]

Hello Anton,

Thanks for the quick update. Updating OE after the update to bitbake 2.8 should be fine.

Thanks and Regards,
Adithya Balakumar

-----Original Message-----
From: Anton Mikanovich <ami...@ilbers.de>
Sent: Wednesday, March 13, 2024 9:30 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya....@toshiba-tsip.com>; florian...@siemens.com; venkata1.pyla <venkat...@toshibaap.onmicrosoft.com>; felix.mo...@siemens.com; isar-...@googlegroups.com; ub...@ilbers.de
Cc: jan.k...@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro...@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh...@toshiba-tsip.com>; venkat...@gmail.com
Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[MOESSBAUER, Felix]

On Thu, 2023-12-07 at 21:11 +0530, venkat...@toshiba-tsip.com wrote:

Hi, while reworking the SDE in ISAR, I stumbled upon this as well.
This patch only covers the .wic part, but we need a similar patch for
the IMAGE_CMD:ext4 as well. I tried to mimic the pattern there, but I
was not able to make the .ext4 build reproducible (the diff indicated
that the inodes are still shuffled around). This makes me wonder if
mke2fs.ext4 even supports producing a reproducible rootfs.

Have you been able to create a bit-by-bit identical .wic of an ext4
partition?

Best regards,
Felix

>          label_str = ""
>          if self.label:
>              label_str = "-L %s" % self.label

--

Siemens AG, Technology
Linux Expert Center

[MOESSBAUER, Felix]

On Wed, 2024-03-13 at 18:00 +0200, Anton Mikanovich wrote:
> 13/03/2024 12:01, Adithya....@toshiba-tsip.com wrote:
> > Hello Anton,
> >
> > Hope this message finds you well.
> >
> > Just wanted to follow up on a previous mail regarding wic and other
> > related updates from openembedded-core. You had mentioned about the
> > updates from openembedded-core to be handled right after the
> > release of ISAR v0.10
> >
> > Would like to understand if there are any plans for wic and other
> > related updates from openembedded-core.
> >
> > Thanks and Regards,
> > Adithya Balakumar
>
> Hello Adithya,
>
> We've checked and the next OE LTS using bitbake 2.8 should be
> released
> in April.

Hi, bitbake 2.8 has been released last week.

> We could start preparations for the bitbake upgrade and update OE
> after
> that.

It would be great if we could have bitbake 2.8 in a timely manner. This
release contains many of our contributed fixes (e.g. for reproducible
builds, better estimation of available CPUs on large servers).

Felix

> Would that suit your needs?
>

[Jan Kiszka]

Any impact on recipe syntax etc. again?

Jan

[MOESSBAUER, Felix]

I just did a quick test of bitbake 2.8 in ISAR and that (almost) worked
out of the box. Only required change was to move the
kbuildtarget.bbclass into the meta/classes dir.

Probably the only relevant changes are around the handling of PR:

- The PR value can no longer be set from the recipe file name - this
was rarely used, but in any case is no longer supported.
- PE and PR are no longer included in the work directory path
(WORKDIR). This may break some tool assumptions about directory paths,
but those should really be querying paths from the build system (or not
poking into WORKDIR externally).

https://docs.yoctoproject.org/4.3.4/migration-guides/migration-4.3.html#versioning-changes

Felix

[Anton Mikanovich]

Hello,

I've already started working on Bitbake and OE libs update.
As you've already mentioned there are no much changes, so hope will send
it soon.

[Adithya Balakumar]

Hi Felix,

I saw your mail regarding your attempt to make ext4 filesystem images from IMAGE_CMD:ext4 reproducible. 

If you don't mind, could you briefly explain what was the problem you faced in achieving this? 

I am also trying to understand on how to achieve the same.

Thanks and Regards,

Adithya Balakumar