[isar 0/1] Fix non-reproducible issue due to localepurge

[venkat...@toshiba-tsip.com]

From: venkata pyla <venkat...@toshiba-tsip.com>

Hello all,

Replying to this mail conversation,
(https://groups.google.com/g/isar-users/c/JCgpWU_pFJU/m/IwlYTmzQBQAJ)

For the problem "debconf cache non-reproducible", I figured out a solution in
isar system itself.

The problem is that, in isar system, in-order to remove unwanted locales from
the system image, it is installing the localepurge package after its
configuration is copied,
due to this when the package post-installation script triggered the ucf
check, it creates a debconf cache entry with the new configuration
change (which is non-reproducible).

This would be avoided if localpurge package is installed first and then copy
required configuration to remove unwanted to locales from the system image.

Created below patch to fix non-reproducible problem, please review and provide
your feedback.

venkata pyla (1):
image-locales-extension: Do localepurge configuration after
installation

meta/classes/image-locales-extension.bbclass | 33 ++++++++++----------
1 file changed, 17 insertions(+), 16 deletions(-)

--
2.20.1

[venkat...@toshiba-tsip.com]

From: venkata pyla <venkat...@toshiba-tsip.com>

The current seqence of copying the localepurge configuration first and
then installing localepurge package is causing the debconf cache entry,
that is because the package postinstall script assume the configuration
is already present and checks for configuration change and updates the
debconf cache entry.

With the above sequence of steps it is causing unnecessary debconf cache
entry and which is causing non-reproducible system image creation.

To avoid this, the sequence is altered to install the package first and
then configure necessary locales to remove.

Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com>
---

meta/classes/image-locales-extension.bbclass | 33 ++++++++++----------
1 file changed, 17 insertions(+), 16 deletions(-)

diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass
index 25af540..2db6d4a 100644
--- a/meta/classes/image-locales-extension.bbclass
+++ b/meta/classes/image-locales-extension.bbclass
@@ -60,6 +60,16 @@ __EOF__
# Install configuration into image:
sudo -E -s <<'EOSUDO'
set -e
+ localepurge_state='i'
+ if chroot '${ROOTFSDIR}' dpkg -s localepurge 2>/dev/null >&2
+ then
+ echo 'localepurge was installed (leaving it installed later)'
+ else
+ localepurge_state='p'
+ echo 'localepurge was not installed (removing it later)'
+ chroot '${ROOTFSDIR}' apt-get ${ROOTFS_APT_ARGS} localepurge
+ fi
+
cat '${WORKDIR}/locale.gen' >> '${ROOTFSDIR}/etc/locale.gen'
cat '${WORKDIR}/locale.default' > '${ROOTFSDIR}/etc/default/locale'
cat '${WORKDIR}/locale.nopurge' > '${ROOTFSDIR}/etc/locale.nopurge'
@@ -67,15 +77,6 @@ __EOF__

# Enter image and trigger locales config and localepurge:
chroot '${ROOTFSDIR}' /bin/sh <<'EOSH'
- localepurge_state='i'
- if dpkg -s localepurge 2>/dev/null >&2
- then
- echo 'localepurge was installed (leaving it installed later)'
- else
- localepurge_state='p'
- echo 'localepurge was not installed (removing it later)'
- apt-get ${ROOTFS_APT_ARGS} localepurge
- fi

echo 'running locale debconf-set-selections'
debconf-set-selections /tmp/locale.debconf
@@ -86,13 +87,13 @@ __EOF__

echo 'running localepurge'
localepurge
-
- if [ "$localepurge_state" = 'p' ]
- then
- echo removing localepurge...
- apt-get purge --yes localepurge
- apt-get autoremove --purge --yes
- fi
EOSH
+
+ if [ "$localepurge_state" = 'p' ]
+ then
+ echo removing localepurge...
+ chroot '${ROOTFSDIR}' apt-get purge --yes localepurge
+ chroot '${ROOTFSDIR}' apt-get autoremove --purge --yes
+ fi
EOSUDO
}
--
2.20.1

[Henning Schild]

Am Sun, 10 Apr 2022 22:32:28 +0530
schrieb venkat...@toshiba-tsip.com:

> From: venkata pyla <venkat...@toshiba-tsip.com>
>
> Hello all,
>
> Replying to this mail conversation,
> (https://groups.google.com/g/isar-users/c/JCgpWU_pFJU/m/IwlYTmzQBQAJ)
>
> For the problem "debconf cache non-reproducible", I figured out a
> solution in isar system itself.

Generating configuration items and placing them before a package gets
installed is very common. It is not only done by Isar but very likely
by many packages in debian that cater for other packages.

What you are suggesting is not a generic solution to "debconf cache"
being non-reproducible. It is just a quirk to hide the problem in that
one case for that one package.

Maybe the cache just needs to be excluded from the image, or cleaned or
whatever.

And i guess we should have some infrastructure in Isar to find
"non-reproducible" build issues also available for any layers. Because
layers is where the real complexity is, not those relatively boring
example images you find in isar.

regards,
Henning

[Jan Kiszka]

On 11.04.22 12:53, Henning Schild wrote:
> Am Sun, 10 Apr 2022 22:32:28 +0530
> schrieb venkat...@toshiba-tsip.com:
>
>> From: venkata pyla <venkat...@toshiba-tsip.com>
>>
>> Hello all,
>>
>> Replying to this mail conversation,
>> (https://groups.google.com/g/isar-users/c/JCgpWU_pFJU/m/IwlYTmzQBQAJ)
>>
>> For the problem "debconf cache non-reproducible", I figured out a
>> solution in isar system itself.
>
> Generating configuration items and placing them before a package gets
> installed is very common. It is not only done by Isar but very likely
> by many packages in debian that cater for other packages.
>
> What you are suggesting is not a generic solution to "debconf cache"
> being non-reproducible. It is just a quirk to hide the problem in that
> one case for that one package.
>
> Maybe the cache just needs to be excluded from the image, or cleaned or
> whatever.
>
> And i guess we should have some infrastructure in Isar to find
> "non-reproducible" build issues also available for any layers. Because
> layers is where the real complexity is, not those relatively boring
> example images you find in isar.
>

Just ran into a more complex scenario where non-reproducibility of the
image bites:

isar-cip-core [1] in secure boot mode generates dm-verity images and a
corresponding initramfs which carries the rootfs hash. As we are not
able to re-generate the very same rootfs from the same task(-hashes),
bitbake will not rebuild the initramfs, and image and initramfs will no
longer match.

Jan

[1] https://gitlab.com/cip-project/cip-core/isar-cip-core

--
Siemens AG, Technology
Competence Center Embedded Linux

[Venkat...@toshiba-tsip.com]

Hi Henning Schild,

Thank you for the comments.

Please find my comments inline.

>-----Original Message-----
>From: isar-...@googlegroups.com <isar-...@googlegroups.com> On Behalf
>Of Henning Schild
>Sent: 11 April 2022 16:24
>To: pyla venkata(ＴＳＩＰ) <Venkat...@toshiba-tsip.com>
>Cc: isar-...@googlegroups.com; jan.k...@siemens.com; dinesh kumar(ＴＳ
>ＩＰ) <dinesh...@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 □ＳＷＣ◯Ａ
>ＣＴ) <kazuhiro...@toshiba.co.jp>
>Subject: Re: [isar 0/1] Fix non-reproducible issue due to localepurge
>
>Am Sun, 10 Apr 2022 22:32:28 +0530
>schrieb venkat...@toshiba-tsip.com:
>
>> From: venkata pyla <venkat...@toshiba-tsip.com>
>>
>> Hello all,
>>
>> Replying to this mail conversation,
>> (https://groups.google.com/g/isar-users/c/JCgpWU_pFJU/m/IwlYTmzQBQAJ)
>>
>> For the problem "debconf cache non-reproducible", I figured out a
>> solution in isar system itself.
>
>Generating configuration items and placing them before a package gets
>installed is very common. It is not only done by Isar but very likely by many
>packages in debian that cater for other packages.
>
>What you are suggesting is not a generic solution to "debconf cache"
>being non-reproducible. It is just a quirk to hide the problem in that one case for
>that one package.

Yes I agree with you, Ideally this should be fixed from each package if they have such non-reproducible behaviour.

I reported a bug today in the Debian package bug-tracking system, though I don’t have solution for that,
but I want to start discussion and get some solution going forward.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1009639

Meanwhile we can use this trick to bypass non-reproducible problem, what's your opinion?

>
>Maybe the cache just needs to be excluded from the image, or cleaned or
>whatever.
>
>And i guess we should have some infrastructure in Isar to find "non-
>reproducible" build issues also available for any layers. Because layers is where
>the real complexity is, not those relatively boring example images you find in
>isar.
>
>regards,
>Henning
>
>> The problem is that, in isar system, in-order to remove unwanted
>> locales from the system image, it is installing the localepurge
>> package after its configuration is copied, due to this when the
>> package post-installation script triggered the ucf check, it creates a
>> debconf cache entry with the new configuration change (which is
>> non-reproducible).
>>
>> This would be avoided if localpurge package is installed first and
>> then copy required configuration to remove unwanted to locales from
>> the system image.
>>
>> Created below patch to fix non-reproducible problem, please review and
>> provide your feedback.
>>
>> venkata pyla (1):
>> image-locales-extension: Do localepurge configuration after
>> installation
>>
>> meta/classes/image-locales-extension.bbclass | 33
>> ++++++++++---------- 1 file changed, 17 insertions(+), 16 deletions(-)
>>
>

>--
>You received this message because you are subscribed to the Google Groups
>"isar-users" group.
>To unsubscribe from this group and stop receiving emails from it, send an email
>to isar-users+...@googlegroups.com.
>To view this discussion on the web visit https://groups.google.com/d/msgid/isar-
>users/20220411125354.7ecbe49c%40md1za8fc.ad001.siemens.net.

[Henning Schild]

Am Wed, 13 Apr 2022 12:15:47 +0000
schrieb <Venkat...@toshiba-tsip.com>:

I would want to bet that there are some packages which show similar
problems directly from upstream, but likely more in downstream isar
layers. Because a debian package on its own will simply bring its
default config, every deviation will come from something else ... like
ansible, puppet, isar ...

> I reported a bug today in the Debian package bug-tracking system,
> though I don’t have solution for that, but I want to start discussion
> and get some solution going forward.
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1009639

I am not sure that is a valid debian but either, they could just tell
you to install first and configure later. The non default config is not
coming from debian.

> Meanwhile we can use this trick to bypass non-reproducible problem,
> what's your opinion?

I would prefer a way/idea to deal with such problems in
general. Like a quality checker script that would warn about a
populated conf cache, or a postprocess rule to clear the cache.

That would be a real good thing, while your patch is only a workaround
for an exotic potential problem most people do not really care too much
about.
We know anyways that an isar build is never 100% reproducible ... just
try a wic run and binary diff the resulting images instead of diffing
the files on mounted partitions.

regards,
Henning

[Jan Kiszka]

Are wic-related differences coming from the Isar-way of using wic? Or
are we sharing issues with OE/Yocto? They claim to be reproducible. We
will have to eventually as well.

Jan

[venkat...@toshiba-tsip.com]

From: venkata pyla <venkat...@toshiba-tsip.com>

Cleaning the debconf cache files for the purpose of generating
reproducible system image.

The debconf cache database sometime generates with non-reproducible
contents because of some packages behaviour (e.g: localepurge), reported
this to package bug tracking system[1], but because this problem is not
with one particular package and can happen with other packages also,
this should be solved in common place to address reproducibility
problem, discussing with Reproducible-builds community[2] to find some
common solution for this problem.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1009639
[2] https://lists.reproducible-builds.org/pipermail/rb-general/2022-April/002546.html

Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com>
Suggested-by: Henning Schild <henning...@siemens.com>
---
meta/classes/image.bbclass | 2 +-
meta/classes/rootfs.bbclass | 6 ++++++
2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index eb879ff..a306bd2 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -71,7 +71,7 @@ image_do_mounts() {
}

ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "clean-package-cache generate-manifest export-dpkg-status clean-log-files"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest export-dpkg-status clean-log-files clean-debconf-cache"
ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index b021e72..78f4ba6 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -243,6 +243,12 @@ rootfs_postprocess_clean_log_files() {
-exec rm -f {} ';'
}

+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-debconf-cache', 'rootfs_postprocess_clean_debconf_cache', '', d)}"
+rootfs_postprocess_clean_debconf_cache() {
+ # Delete debconf cache files
+ sudo rm -rf "${ROOTFSDIR}/var/cache/debconf/"*
+}
+
ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-manifest', 'rootfs_generate_manifest', '', d)}"
rootfs_generate_manifest () {
mkdir -p ${ROOTFS_MANIFEST_DEPLOY_DIR}
--
2.20.1

[Venkat...@toshiba-tsip.com]

Hi,

Sorry, somehow my replay mail to the below patch was not went to the mailing list earlier, I am resending.

Please provide your comments for the below patch that will remove the debconf cache files in the isar image.

I am removing these files for the purpose of the solving non-reproducible problem in debconf cache file.
I am also finding solution to not remove these files, by fixing the actual problem in packages who produces the non-reproducible content (information is present in the below commit log)

Also I am checking why not remove these files in isar image.
As I understood, the debconf cache files are used to store the answers of debconf's questions asked during package installation and same cached information will be used in the next package update/re-install time.
I think in the isar build system the package installation is noninteractive and the debconf cache generates with default data, which is not necessary to store.
If my above understanding is correct, I think we can safely delete the debconf cache file in isar image.

Kindly let me know your feedback on this understanding and provide your suggestions.

Thanks,
Venkata.

>-----Original Message-----
>From: venkat...@toshiba-tsip.com <venkat...@toshiba-tsip.com>
>Sent: 06 May 2022 15:16
>To: isar-...@googlegroups.com
>Cc: pyla venkata(ＴＳＩＰ) <Venkat...@toshiba-tsip.com>;
>henning...@siemens.com; jan.k...@siemens.com; dinesh kumar(ＴＳＩ
>Ｐ) <dinesh...@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣ
>Ｔ) <kazuhiro...@toshiba.co.jp>

[Venkat...@toshiba-tsip.com]

Hi,

Any comments or suggestions on the below patch.

[Jan Kiszka]

Looks good to me.

[Anton Mikanovich]

08.06.2022 17:40, Venkat...@toshiba-tsip.com wrote:

Hi,

Any comments or suggestions on the below patch.

Thanks,
Venkata.

Hello,

Patch is ok, we will apply it after upcoming release.
Can you please resend it as a '[PATCH]'

[Venkat...@toshiba-tsip.com]

Hi,

 

Thanks for the comments, I will resend the patch again.

 

Thanks,

Venkata.

--

You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/86ba121e-b936-d57c-5fe1-c192730f4b6d%40ilbers.de.

[venkat...@toshiba-tsip.com]

From: venkata pyla <venkat...@toshiba-tsip.com>

Cleaning the debconf cache files for the purpose of generating
reproducible system image.

The debconf cache database sometime generates with non-reproducible
contents because of some packages behaviour (e.g: localepurge), reported
this to package bug tracking system[1], but because this problem is not
with one particular package and can happen with other packages also,
this should be solved in common place to address reproducibility
problem, discussing with Reproducible-builds community[2] to find some
common solution for this problem.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1009639
[2] https://lists.reproducible-builds.org/pipermail/rb-general/2022-April/002546.html

Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com>
Suggested-by: Henning Schild <henning...@siemens.com>
---
meta/classes/image.bbclass | 2 +-
meta/classes/rootfs.bbclass | 6 ++++++
2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass

index 0da56b7..dc8330a 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -69,7 +69,7 @@ image_do_mounts() {

}

ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "clean-package-cache generate-manifest export-dpkg-status clean-log-files"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest export-dpkg-status clean-log-files clean-debconf-cache"
ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass

index 7e9fb0f..3b0abc2 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -244,6 +244,12 @@ rootfs_postprocess_clean_log_files() {

-exec rm -f {} ';'
}

+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-debconf-cache', 'rootfs_postprocess_clean_debconf_cache', '', d)}"
+rootfs_postprocess_clean_debconf_cache() {
+ # Delete debconf cache files
+ sudo rm -rf "${ROOTFSDIR}/var/cache/debconf/"*
+}
+
ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-manifest', 'rootfs_generate_manifest', '', d)}"
rootfs_generate_manifest () {
mkdir -p ${ROOTFS_MANIFEST_DEPLOY_DIR}

--
2.20.1

[Anton Mikanovich]

16.06.2022 07:36, venkat...@toshiba-tsip.com wrote:
> From: venkata pyla <venkat...@toshiba-tsip.com>
>
> Cleaning the debconf cache files for the purpose of generating
> reproducible system image.
>
> The debconf cache database sometime generates with non-reproducible
> contents because of some packages behaviour (e.g: localepurge), reported
> this to package bug tracking system[1], but because this problem is not
> with one particular package and can happen with other packages also,
> this should be solved in common place to address reproducibility
> problem, discussing with Reproducible-builds community[2] to find some
> common solution for this problem.
>
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1009639
> [2] https://lists.reproducible-builds.org/pipermail/rb-general/2022-April/002546.html
>
> Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com>
> Suggested-by: Henning Schild <henning...@siemens.com>

Applied to next, thanks.
Sorry for delay, it looks like we've lost this patch after the latest
release.