[PATCH v1 0/1] Use oe.utils.directory_size() instead of du for calculating rootfs size

[Adithya Balakumar]

Currently the "du" command is used to calculate the rootfs size, and sometimes the
reported size of the rootfs by du varies across multiple builds (usually by a few kb).
But the rootfs contents are reproducible.

Although this does not affect any functionality, but it does affect reproducibility
of artifacts that rely on the calculated rootfs size.

There are 2 scenarios (that I noticed) where this is affects reproducibility:
1. In the case where no size parameter is set for a rootfs partition in the wks file,
wic uses the calculated rootfs size.
2. ext4 filesystem images with the IMAGE_CMD of isar, relies on rootfs size to create image.

Some of the files/directories in the rootfs has different number of blocks allocated.
The files/directories affected are not always the same across builds.

Using the "du" command to measure the size of the rootfs directory in such a case, yields
different values for the ROOTFS_SIZE.

This patch introduces the use of an alternate function from oe.utils to calculate the
directory size. OE-Core uses the same function to calculate the directory size (See [1])

Based on multiple build runs, usage of this function seems to give consistent size results.

Some details on the issue is mentioned in the below thread in the ISAR mailing list (See [2])

[1] https://github.com/openembedded/openembedded-core/commit/d8f1f3a6b024a2ae6631d1ce25421e8d94b69a12
[2] https://groups.google.com/g/isar-users/c/Ll7t4G41Lfo/m/gvxR4hYBAwAJ

Adithya Balakumar (1):
image.bbclass: use oe.utils.directory_size() instead of du

meta/classes/image.bbclass | 7 ++-----
1 file changed, 2 insertions(+), 5 deletions(-)

--
2.39.5

[Adithya Balakumar]

Currently the "du" command is used to calculate the rootfs size, and
sometimes the reported size of the rootfs by "du" varies across multiple
builds (usually by a few kb). But the rootfs contents are reproducible.

Although this does not affect any functionality, but it does affect
reproducibility of artifacts that rely on the calculated rootfs size.

Some of the files/directories in the rootfs has different number of

blocks allocated and "du" estimates the directory size based on the
allocated blocks. The files/directories affected are not always the
same across builds.

OE-Core uses a custom function to calculate directory size (See [1])
and this patch introduces the use of that function from oe.utils to
calculate the directory size.

[1] https://github.com/openembedded/openembedded-core/commit/d8f1f3a6b024a2ae6631d1ce25421e8d94b69a12

Signed-off-by: Adithya Balakumar <Adithya....@toshiba-tsip.com>
---

meta/classes/image.bbclass | 7 ++-----
1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 982523b0..3e70fe83 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -85,13 +85,10 @@ inherit image-account-extension
ROOTFS_EXTRA ?= "64"

def get_rootfs_size(d):
- import subprocess
+ import subprocess, oe.utils
rootfs_extra = int(d.getVar("ROOTFS_EXTRA"))

- output = subprocess.check_output(
- ["sudo", "du", "-xs", "--block-size=1k", d.getVar("IMAGE_ROOTFS")]
- )
- base_size = int(output.split()[0])
+ base_size = int(oe.utils.directory_size(d.getVar("IMAGE_ROOTFS")) / 1024)

return base_size + rootfs_extra * 1024

--
2.39.5

[Uladzimir Bely]

Applied to next, thanks.

--
Best regards,
Uladzimir.

[MOESSBAUER, Felix]

On Fri, 2024-11-29 at 11:58 +0530, Adithya Balakumar wrote:

Hi, the subprocess import is not needed anymore.

> rootfs_extra = int(d.getVar("ROOTFS_EXTRA"))
>
> - output = subprocess.check_output(
> - ["sudo", "du", "-xs", "--block-size=1k", d.getVar("IMAGE_ROOTFS")]
> - )
> - base_size = int(output.split()[0])
> + base_size = int(oe.utils.directory_size(d.getVar("IMAGE_ROOTFS")) / 1024)

This unfortunately breaks if the calling user is not able to read all
files in this rootfs. Actually a single file with perms 600 and owner
root breaks this and by that the imaging.

I don't have a better solution yet, though.
I'm also wondering why this only came up now. It seems like it only
breaks under rootless.

Felix

>
> return base_size + rootfs_extra * 1024
>
> --
> 2.39.5
>
>

> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/isar-users/20241129062827.527177-2-Adithya.Balakumar%40toshiba-tsip.com.