Missing profile variables (Kiwi 10.1.18) )

[Massi aka Ergosum]

Hello,

it seems the .profile file in Kiwi build root does not contain all variables I'm expecting from config.xml. For example, I think the following should be populated:

kiwi_iversion=''
kiwi_keytable=''
kiwi_language=''

See attachment.

Is it a bug? Is there any other place to find those values?

Thanks

Massi

[Marcus Schäfer]

Hi,

> it seems the .profile file in Kiwi build root does not contain all
> variables I'm expecting from config.xml.

Hmm in which context of the build are you using them ?

> For example, I think the
> following should be populated:
> kiwi_iversion=''
> kiwi_keytable=''
> kiwi_language=''
> See attachment.

Yes at least iversion must always be there, keytable and language
depend on attributes in the kiwi description but without a version
you could not build. So this is really strange.

I did a test build with a config.sh script that does the following:

---
set -ex
cat /.profile
---

In the log file of the build I see the profile and it matches
my expectation:

[ INFO ]: 16:37:10 | Calling config.sh script
[ DEBUG ]: 16:37:10 | --------------out start-------------
[ DEBUG ]: 16:37:10 | kiwi_Volume_1='home|freespace:120|home'
[ DEBUG ]: 16:37:10 | kiwi_Volume_Root='|size:all|'
[ DEBUG ]: 16:37:10 | kiwi_align='1048576'
[ DEBUG ]: 16:37:10 | kiwi_boot_timeout='10'
[ DEBUG ]: 16:37:10 | kiwi_bootkernel=''
[ DEBUG ]: 16:37:10 | kiwi_bootloader='grub2'
[ DEBUG ]: 16:37:10 | kiwi_bootloader_console='serial:serial'
[ DEBUG ]: 16:37:10 | kiwi_bootprofile=''
[ DEBUG ]: 16:37:10 | kiwi_btrfs_root_is_snapshot='true'
[ DEBUG ]: 16:37:10 | kiwi_cmdline='console=ttyS0'
[ DEBUG ]: 16:37:10 | kiwi_compressed=''
[ DEBUG ]: 16:37:10 | kiwi_delete=''
[ DEBUG ]: 16:37:10 | kiwi_devicepersistency=''
[ DEBUG ]: 16:37:10 | kiwi_displayname='kiwi-test-image-disk-containers'
[ DEBUG ]: 16:37:10 | kiwi_drivers=''
[ DEBUG ]: 16:37:10 | kiwi_firmware='efi'
[ DEBUG ]: 16:37:10 | kiwi_fsmountoptions=''
[ DEBUG ]: 16:37:10 | kiwi_gpt_hybrid_mbr=''
[ DEBUG ]: 16:37:10 | kiwi_hybridpersistent=''
[ DEBUG ]: 16:37:10 | kiwi_hybridpersistent_filesystem=''
[ DEBUG ]: 16:37:10 | kiwi_iname='kiwi-test-image-disk-containers'
[ DEBUG ]: 16:37:10 | kiwi_initrd_system='dracut'
[ DEBUG ]: 16:37:10 | kiwi_install_volid='INSTALL'
[ DEBUG ]: 16:37:10 | kiwi_installboot='install'
[ DEBUG ]: 16:37:10 | kiwi_iversion='1.42.1'
[ DEBUG ]: 16:37:10 | kiwi_keytable='us'
[ DEBUG ]: 16:37:10 | kiwi_language='en_US'
[ DEBUG ]: 16:37:10 | kiwi_loader_theme='openSUSE'
[ DEBUG ]: 16:37:10 | kiwi_luks_empty_passphrase='false'
[ DEBUG ]: 16:37:10 | kiwi_oembootwait=''
[ DEBUG ]: 16:37:10 | kiwi_oemdevicefilter=''
[ DEBUG ]: 16:37:10 | kiwi_oemkboot=''
[ DEBUG ]: 16:37:10 | kiwi_oemmultipath_scan='false'
[ DEBUG ]: 16:37:10 | kiwi_oemnicfilter=''
[ DEBUG ]: 16:37:10 | kiwi_oempartition_install=''
[ DEBUG ]: 16:37:10 | kiwi_oemreboot=''
[ DEBUG ]: 16:37:10 | kiwi_oemrebootinteractive=''
[ DEBUG ]: 16:37:10 | kiwi_oemrecovery=''
[ DEBUG ]: 16:37:10 | kiwi_oemrecoveryID=''
[ DEBUG ]: 16:37:10 | kiwi_oemrecoveryInPlace=''
[ DEBUG ]: 16:37:10 | kiwi_oemrecoveryPartSize=''
[ DEBUG ]: 16:37:10 | kiwi_oemresizeonce=''
[ DEBUG ]: 16:37:10 | kiwi_oemrootMB=''
[ DEBUG ]: 16:37:10 | kiwi_oemshutdown=''
[ DEBUG ]: 16:37:10 | kiwi_oemshutdowninteractive=''
[ DEBUG ]: 16:37:10 | kiwi_oemsilentboot=''
[ DEBUG ]: 16:37:10 | kiwi_oemsilentinstall=''
[ DEBUG ]: 16:37:10 | kiwi_oemsilentverify=''
[ DEBUG ]: 16:37:10 | kiwi_oemskipverify=''
[ DEBUG ]: 16:37:10 | kiwi_oemswapMB='1024'
[ DEBUG ]: 16:37:10 | kiwi_oemtitle=''
[ DEBUG ]: 16:37:10 | kiwi_oemunattended='true'
[ DEBUG ]: 16:37:10 | kiwi_oemunattended_id=''
[ DEBUG ]: 16:37:10 | kiwi_oemvmcp_parmfile=''
[ DEBUG ]: 16:37:10 | kiwi_profiles=''
[ DEBUG ]: 16:37:10 | kiwi_ramonly=''
[ DEBUG ]: 16:37:10 | kiwi_revision='$Format:%H$'
[ DEBUG ]: 16:37:10 | kiwi_rootpartuuid=''
[ DEBUG ]: 16:37:10 | kiwi_sectorsize='512'
[ DEBUG ]: 16:37:10 | kiwi_showlicense=''
[ DEBUG ]: 16:37:10 | kiwi_splash_theme='breeze'
[ DEBUG ]: 16:37:10 | kiwi_startsector='2048'
[ DEBUG ]: 16:37:10 | kiwi_strip_delete=''
[ DEBUG ]: 16:37:10 | kiwi_strip_libs=''
[ DEBUG ]: 16:37:10 | kiwi_strip_tools=''
[ DEBUG ]: 16:37:10 | kiwi_target_blocksize=''
[ DEBUG ]: 16:37:11 | kiwi_target_removable=''
[ DEBUG ]: 16:37:11 | kiwi_timezone='Europe/Berlin'
[ DEBUG ]: 16:37:11 | kiwi_type='oem'
[ DEBUG ]: 16:37:11 | kiwi_vga=''
[ DEBUG ]: 16:37:11 | kiwi_wwid_wait_timeout=''
----

Can you give as a bit more details how to reproduce it ?

Thanks

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer

[Massi aka Ergosum]

Il giorno mercoledì 13 novembre 2024 alle 16:41:42 UTC+1 Marcus ha scritto:

Can you give as a bit more details how to reproduce it ?

Hi, I'm using these scripts [1].

If I well understand, variables should be loaded by:

test -f /.profile && . /.profile

for both config.sh and post_bootstrap.sh.

I suspect something could be wrong in my environment: I'm using podman to isolate the build process [2].
The build process is described in README file.

Note: you can see a little older version since I didn't push yet latest code.

[1] https://github.com/mbugni/debian-remix
[2] https://github.com/mbugni/debian-remix/blob/main/Containerfile

Regards

Massi
 

[Marcus Schäfer]

Hi,

> Hi, I'm using these scripts [1].
> If I well understand, variables should be loaded by:
> test -f /.profile && . /.profile

correct but not needed anymore, kiwi imports this automatically
but it also doesn't hurt if you still have this line in your
scripts

> for both config.sh and post_bootstrap.sh.

Here is the issue. The full profile is available in config.sh
and any scripts that you place after this step. But in
post_bootstrap.sh the full profile env is not yet present
because the system root tree is not yet complete. If you
look into:

kiwi/tasks/system_build.py

You see:

system.install_bootstrap(...)
...
setup.call_post_bootstrap_script()
...
system.install_system(...)
...
profile.create(...)

setup.*

==> up to hear the profile env is complete
...

In your image description I found a post_bootstrap.sh which
does a lot and yes you cannot yet rely on the profile variables
at this stage because they can still change or are not yet
known. I agree with you that this does not apply to static
information like the image version.

We could add a patch to kiwi that creates the profile prior
call_post_bootstrap_script and again after the system root
tree is complete. But this will also create an intermediate
env state which can differ between the scripts, also not
so nice

Looking at your post_bootstrap.sh it looks like most of the
code could also run in config.sh, or did you had a specific
reason to put it there ?

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer

-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------

[Massi aka Ergosum]

Il giorno gio 14 nov 2024 alle ore 14:16 Marcus Schäfer
<marcus....@gmail.com> ha scritto:

>
>
> correct but not needed anymore, kiwi imports this automatically
> but it also doesn't hurt if you still have this line in your
> scripts
>

Ok

>
> Looking at your post_bootstrap.sh it looks like most of the
> code could also run in config.sh, or did you had a specific
> reason to put it there ?
>

Following your suggestion I removed .profile import, and variables are
inherited anyway by scripts.
I'm moving most of the settings to config.sh, but I'm still missing something.
Let me show what I think to be mapped from xml to variables (in config.sh):

image.name --> kiwi_iname (ok)
image.displayname --> kiwi_displayname (ok)
image/preferences/version --> kiwi_iversion (missing)
image/preferences/locale --> kiwi_language (missing)
image/preferences/keytable --> kiwi_keytable (missing)
image/preferences/timezone --> kiwi_timezone (missing)

It seems the profile does not contain all settings from config.xml
-- or --
The above mappings are not valid

*Note*
Some settings (locale, keytable, timezone) are also in profiles, like:

<preferences profiles="Localization-it_IT">
<locale>it_IT</locale>
<keytable>it</keytable>
<timezone>Europe/Rome</timezone>
</preferences>

So basically those preferences should be inherited/overridden, could
it be an issue?

Regards
Massi

[Marcus Schäfer]

Hi,

> Following your suggestion I removed .profile import, and variables are
> inherited anyway by scripts.
> I'm moving most of the settings to config.sh, but I'm still missing something.
> Let me show what I think to be mapped from xml to variables (in config.sh):
>
> image.name --> kiwi_iname (ok)
> image.displayname --> kiwi_displayname (ok)
> image/preferences/version --> kiwi_iversion (missing)
> image/preferences/locale --> kiwi_language (missing)
> image/preferences/keytable --> kiwi_keytable (missing)
> image/preferences/timezone --> kiwi_timezone (missing)
>
> It seems the profile does not contain all settings from config.xml

Let's check that. I read a bit on your documentation. You should
be able to check the end result of the image description with
the following command:

sudo podman exec livebuild-deb12 kiwi-ng --profile=Workstation-it_IT --type=iso image info --description=/live/source/kiwi-descriptions --print-xml

This should print the final image description after all the inheritance.
Let's check what this prints and if it matches the profile.

[Massi aka Ergosum]

Il giorno gio 14 nov 2024 alle ore 20:38 Marcus Schäfer
<marcus....@gmail.com> ha scritto:
>

> Let's check that. I read a bit on your documentation. You should
> be able to check the end result of the image description with
> the following command:
>
> sudo podman exec livebuild-deb12 kiwi-ng --profile=Workstation-it_IT --type=iso image info --description=/live/source/kiwi-descriptions --print-xml
>

Ah, good point! I'm using:

root@livebuild-deb12:/# kiwi-ng --version
KIWI (next generation) version 10.1.18

I attached the result: it seems ok to me, but the schema version
doesn't match the original.

Regards
Massi

[Marcus Schäfer]

Hi,

> I attached the result: it seems ok to me, but the schema version
> doesn't match the original.

The schema version is automatically converted to the schema of
the kiwi version you used. kiwi has automatic XSL transformations
such that you can stick to an older schema but kiwi internally
converts it for you. When you use the --print-xml/--print-yaml/--print-toml
options the result image description is the end result of all
conversions.

Thanks for the information on the final image description.
It helped me to find out the issue you are running into.

In fact you were right. If there is a <preferences> section
that does not specify a version it gets overwritten as empty
even if it was set correctly in another <preferences> section.
The order of the <preferences> sections plays a role here and
that was also the reason why I did not found the problem in
the first place.

I fixed this for all env variables which are not allowed
to take an empty value and came up with the following PR:

https://github.com/OSInside/kiwi/pull/2677

I also added a helpful --print-kiwi-env option to the info
command such that we can better check how the kiwi env
will look like prior building

Best regards,

[Massi aka Ergosum]

Hi,

>
> The schema version is automatically converted to the schema of
> the kiwi version you used. kiwi has automatic XSL transformations
> such that you can stick to an older schema but kiwi internally
> converts it for you. When you use the --print-xml/--print-yaml/--print-toml
> options the result image description is the end result of all
> conversions.
>

Ok, I understand.

>
> I fixed this for all env variables which are not allowed
> to take an empty value and came up with the following PR:
>
> https://github.com/OSInside/kiwi/pull/2677
>

I subscribed the PR.

> I also added a helpful --print-kiwi-env option to the info
> command such that we can better check how the kiwi env
> will look like prior building
>

Good idea, it will be helpful for testing.

Thanks!

Massi