[PATCH v1 0/1] Set hash-seed for empty ext partition

[Adithya Balakumar]

ext filesystems require a hash_seed to generate deterministic directory
indexes for reproducible builds. This is handled by wic in the case where
the rootfs plugin is used but this not handled in the case when an empty
ext4 partition is deployed.

This patch is also shared with openembedded-core and also accepted.
Upstream commit: 0202fb594fb05098cb8d8b6088e63beb40b5906e

Adithya Balakumar (1):
wic/partition.py: Set hash_seed for empty ext partition

scripts/lib/wic/partition.py | 37 ++++++++++++++++++++++--------------
1 file changed, 23 insertions(+), 14 deletions(-)

--
2.39.2

[Adithya Balakumar]

From: Adithya Balakumar <adithya....@toshiba-tsip.com>

Although setting hash_seed is handled for the rootfs plugin case, but
this is missed when deploying an empty ext partition.

Signed-off-by: Adithya Balakumar <adithya....@toshiba-tsip.com>
---

scripts/lib/wic/partition.py | 37 ++++++++++++++++++++++--------------
1 file changed, 23 insertions(+), 14 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 795707ec..bf2c34d5 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -284,19 +284,8 @@ class Partition():

extraopts = self.mkfs_extraopts or "-F -i 8192"

- if os.getenv('SOURCE_DATE_EPOCH'):
- sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
- if pseudo:
- pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s " % (sde_time, pseudo)
- else:
- pseudo = "export E2FSPROGS_FAKE_TIME=%s; " % sde_time
-
- # Set hash_seed to generate deterministic directory indexes
- namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-2f2fdf33d460")
- if self.fsuuid:
- namespace = uuid.UUID(self.fsuuid)
- hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
- extraopts += " -E hash_seed=%s" % hash_seed
+ # use hash_seed to generate reproducible ext4 images
+ (extraopts, pseudo) = self.get_hash_seed_ext4(extraopts, pseudo)

label_str = ""
if self.label:
@@ -344,6 +333,23 @@ class Partition():

self.check_for_Y2038_problem(rootfs, native_sysroot)

+ def get_hash_seed_ext4(self, extraopts, pseudo):
+ if os.getenv('SOURCE_DATE_EPOCH'):
+ sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
+ if pseudo:
+ pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s " % (sde_time, pseudo)
+ else:
+ pseudo = "export E2FSPROGS_FAKE_TIME=%s; " % sde_time
+
+ # Set hash_seed to generate deterministic directory indexes
+ namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-2f2fdf33d460")
+ if self.fsuuid:
+ namespace = uuid.UUID(self.fsuuid)
+ hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
+ extraopts += " -E hash_seed=%s" % hash_seed
+
+ return (extraopts, pseudo)
+
def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
native_sysroot, pseudo):
"""
@@ -437,13 +443,16 @@ class Partition():

extraopts = self.mkfs_extraopts or "-i 8192"

+ # use hash_seed to generate reproducible ext4 images
+ (extraopts, pseudo) = self.get_hash_seed_ext4(extraopts, None)
+
label_str = ""
if self.label:
label_str = "-L %s" % self.label

mkfs_cmd = "mkfs.%s -F %s %s -U %s %s" % \
(self.fstype, extraopts, label_str, self.fsuuid, rootfs)
- exec_native_cmd(mkfs_cmd, native_sysroot)
+ exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)

self.check_for_Y2038_problem(rootfs, native_sysroot)

--
2.39.2

[Jan Kiszka]

Shouldn't we sync to a wic revision >= the one containing this?

Jan

--
Siemens AG, Technology
Linux Expert Center

[Anton Mikanovich]

25/06/2024 10:37, Jan Kiszka wrote:
> Shouldn't we sync to a wic revision >= the one containing this?
>
> Jan
>

Hello Jan,

This is our preferred process by default, but this commit is in oe-core
master
branch only for now. We should probably wait for release or backporting.

We are currently using the latest Scarthgap LTS version 5.0 branch.

[Jan Kiszka]

Maybe propose the commit for 5.0-LTS branch and then sync that in? But
if you are fine with taking this backport, I will not hold you back. It
will be valuable for us, that is for sure.

[Adithya....@toshiba-tsip.com]

Hi Anton and Jan,

Thanks for the feedback. Let me propose this patch to Scarthgap also. And once accepted in Scarthgap, I will share the patch with isar again.

Thanks,
Adithya

[Adithya Balakumar]

ext filesystems require a hash_seed to generate deterministic directory indexes
for reproducible builds. This is handled by wic in the case where the rootfs
plugin is used but this not handled in the case when an empty ext4 partition
is deployed.

This patch is also shared with openembedded-core and also accepted.

master branch commit: 0202fb594fb05098cb8d8b6088e63beb40b5906e
scarthgap branch commit: a4e7334a4b87f1bb1947f6b10f71ddb445c91d0c

Changes since v1:
There is no change in the patch from v1. Just that this patch is also applied
to the scarthgap branch in OE-core.

Adithya Balakumar (1):
wic/partition.py: Set hash_seed for empty ext partition

scripts/lib/wic/partition.py | 37 ++++++++++++++++++++++--------------
1 file changed, 23 insertions(+), 14 deletions(-)

--
2.39.2

[Adithya Balakumar]

From: Adithya Balakumar <adithya....@toshiba-tsip.com>

Although setting hash_seed is handled for the rootfs plugin case, but
this is missed when deploying an empty ext partition.

Signed-off-by: Adithya Balakumar <adithya....@toshiba-tsip.com>
---

scripts/lib/wic/partition.py | 37 ++++++++++++++++++++++--------------
1 file changed, 23 insertions(+), 14 deletions(-)

[MOESSBAUER, Felix]

Hi, isn't that always true since we (and OE) nowadays always set this
variable? Why do we need special handling in this case?

Felix

--

[Florian Bezdeka]

On Thu, 2024-07-04 at 16:40 +0530, Adithya Balakumar wrote:

> ext filesystems require a hash_seed to generate deterministic directory indexes
> for reproducible builds. This is handled by wic in the case where the rootfs
> plugin is used but this not handled in the case when an empty ext4 partition
> is deployed.
>
> This patch is also shared with openembedded-core and also accepted.
> master branch commit: 0202fb594fb05098cb8d8b6088e63beb40b5906e
> scarthgap branch commit: a4e7334a4b87f1bb1947f6b10f71ddb445c91d0c

Thanks for upstreaming!

Instead of applying a single patch I would vote for triggering a
complete wic update.

[Adithya....@toshiba-tsip.com]

Hi Felix,

Thanks for the feedback. You are right, there is no need for a check for existence of the SOURCE_DATE_EPOCH environment variable since it is always available. I see this check in quite a few places in the wic source code upstream (OE) as well.

Probably this could be addressed in a separate patch to upstream and then subsequently to isar as well. What do you think?

Thanks and Regards,
Adithya Balakumar

[Adithya....@toshiba-tsip.com]

Hi Florian,

As I remember, ISAR just updated wic for the latest LTS release of OE (scarthgap) back in May, and this is the only patch to the scarthgap branch that is wic related since then.

If I understand correctly, ISAR updates OE related libraries / dependencies for every LTS release in OE. Please correct me if I am mistaken.

Thanks and Regards,
Adithya Balakumar

-----Original Message-----
From: Florian Bezdeka <florian...@siemens.com>
Sent: Thursday, July 4, 2024 9:14 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya....@toshiba-tsip.com>; isar-...@googlegroups.com; ami...@ilbers.de

[MOESSBAUER, Felix]

On Fri, 2024-07-05 at 07:34 +0000, Adithya....@toshiba-tsip.com
wrote:

> Hi Felix,
>
> Thanks for the feedback. You are right, there is no need for a check
> for existence of the SOURCE_DATE_EPOCH environment variable since it
> is always available. I see this check in quite a few places in the
> wic source code upstream (OE) as well.
>
> Probably this could be addressed in a separate patch to upstream and
> then subsequently to isar as well. What do you think?

Ok, fine for me.

Felix