Strange update issue

[Andrew Coppin]

Greetings.

 

I saw an email to the effect that issue #737 is fixed now, so I updated my Kiwi-NG installation… and now I have a rather strange problem.

 

If I run our automated build system, Kiwi says:

 

[ ERROR ]: 11:50:44 | Unexpected error:

[11:50:24]Traceback (most recent call last):

[11:50:24] File "/usr/bin/kiwi-ng", line 9, in <module>

[11:50:24] load_entry_point('kiwi==9.15.3', 'console_scripts', 'kiwi-ng-3')()

[11:50:24] File "/usr/lib/python3.4/site-packages/kiwi/kiwi.py", line 64, in main

[11:50:24] App()

[11:50:24] File "/usr/lib/python3.4/site-packages/kiwi/app.py", line 30, in __init__

[11:50:24] app = CliTask(should_perform_task_setup=False)

[11:50:24] File "/usr/lib/python3.4/site-packages/kiwi/tasks/base.py", line 57, in __init__

[11:50:24] self.runtime_config = RuntimeConfig()

[11:50:24] File "/usr/lib/python3.4/site-packages/kiwi/runtime_config.py", line 44, in __init__

[11:50:24] [self._home_path(), '.config', 'kiwi', 'config.yml']

[11:50:24] File "/usr/lib/python3.4/site-packages/kiwi/runtime_config.py", line 207, in _home_path

[11:50:24] return os.environ['HOME']

[11:50:24] File "/usr/lib64/python3.4/os.py", line 631, in __getitem__

[11:50:24] raise KeyError(key) from None

[11:50:24]KeyError: 'HOME'

 

However, if I run the same command from the console, now Kiwi says:

 

[ INFO    ]: 11:54:45 | Loading XML description

[ INFO    ]: 11:54:45 | Schematron validation failed:

[ INFO    ]: 11:54:45 | --> boot attribute is only available for the following image types: oem pxe

[ ERROR   ]: 11:54:45 | KiwiDescriptionInvalid: Schema validation for Configuration/config.xml failed

 

I’m still not sure what it’s unhappy about. This configuration worked just fine before the update, but now it’s complaining.

 

For that matter, I don’t see much documentation for Kiwi-NG. In particular, I can’t seem to find any description of what the XML file is meant to look like. (Presumably not much different than legacy Kiwi.)

 

For reference, the <type> attribute is:

 

<type image=”iso” hybrid=”true” firmware=”uefi” boot=”isoboot/suse-leap42.2” flags=”overlay” checkprebuilt=”true” boottimeout=”15” kernelcmdline=”splash”/>

 

Not quite sure how to proceed here.

 

Thanks,

Andrew.

 

Andrew Coppin | Software Developer
T: 01908 597960  


Evidence Talks Ltd | www.evidencetalks.com



  

Evidence Talks Ltd, Willen House, Tongwell Street, Fox Milne, Milton Keynes, MK15 0YS
Registered in England and Wales No: 4611669 VAT No: 805591428
Registered Office: The Pinnacle, 160 Midsummer Blvd, Milton Keynes MK91FF

ISO9001:2008 Certified

This e-mail and its attachments are intended for the above named only. If they have come to you in error you must take no action based on them, nor must you copy or show them to anyone. WARNING: Computer viruses can be transmitted by Email. We take every effort to scan for and remove malicious code from both inbound and outbound email. Evidence Talks Ltd accepts no liability for any damage caused by any virus transmitted by this email.

[David Cassany]

Hi Andrew,

> I saw an email to the effect that issue #737 is fixed now, so I updated my
> Kiwi-NG installation… and now I have a rather strange problem.

Yes in latest release config-cdroot.tar can be included in the uncompressed
area of the CD.

> If I run our automated build system, Kiwi says:
>
> [ ERROR ]: 11:50:44 | Unexpected error:
> [11:50:24]Traceback (most recent call last):
> [11:50:24] File "/usr/bin/kiwi-ng", line 9, in <module>
> [11:50:24] load_entry_point('kiwi==9.15.3', 'console_scripts',
> 'kiwi-ng-3')() [11:50:24] File
> "/usr/lib/python3.4/site-packages/kiwi/kiwi.py", line 64, in main
> [11:50:24] App()
> [11:50:24] File "/usr/lib/python3.4/site-packages/kiwi/app.py", line 30, in
> __init__ [11:50:24] app = CliTask(should_perform_task_setup=False)
> [11:50:24] File "/usr/lib/python3.4/site-packages/kiwi/tasks/base.py", line
> 57, in __init__ [11:50:24] self.runtime_config = RuntimeConfig()
> [11:50:24] File "/usr/lib/python3.4/site-packages/kiwi/runtime_config.py",
> line 44, in __init__ [11:50:24] [self._home_path(), '.config', 'kiwi',
> 'config.yml']
> [11:50:24] File "/usr/lib/python3.4/site-packages/kiwi/runtime_config.py",
> line 207, in _home_path [11:50:24] return os.environ['HOME']
> [11:50:24] File "/usr/lib64/python3.4/os.py", line 631, in __getitem__
> [11:50:24] raise KeyError(key) from None
> [11:50:24]KeyError: 'HOME'

Strange, from the logs it looks like $HOME is not present in your build env...
Kiwi looks for a config file in $HOME, thus expects to $HOME variable to be
defined. Could be the case that your env does not have a $HOME?

> However, if I run the same command from the console, now Kiwi says:
>
> [ INFO ]: 11:54:45 | Loading XML description
> [ INFO ]: 11:54:45 | Schematron validation failed:
> [ INFO ]: 11:54:45 | --> boot attribute is only available for the
> following image types: oem pxe [ ERROR ]: 11:54:45 |
> KiwiDescriptionInvalid: Schema validation for Configuration/config.xml
> failed

Yes this is expected, the boot attribute for ISO images is not supported
anymore. It can only be used in OEM and PXE images with the current KIWI.
Removing it from the <type> tag should solve this issue.

> I’m still not sure what it’s unhappy about. This configuration worked just
> fine before the update, but now it’s complaining.

I guess you were testing with a way older KIWI version before. In short,
current KIWI supports dracut modules in other to create the initrd, being the
only choice for ISO and VMX. Before using dracut KIWI used to create its own
intitrd based on a KIWI description that was referenced with the `boot` <type>
attribute.

For the rest I think your <type> section is completely sane.

Regards,
David

[Andrew Coppin]

> Strange, from the logs it looks like $HOME is not present in your build env...
> Kiwi looks for a config file in $HOME, thus expects to $HOME variable to be
> defined. Could be the case that your env does not have a $HOME?

There's a good chance that quite a few environment variables aren't set. I wasn't expecting Kiwi to be looking for $HOME. If that's all the problem is, I guess I could set $HOME to... something.

> > However, if I run the same command from the console, now Kiwi says:
> >
> > [ INFO ]: 11:54:45 | Loading XML description
> > [ INFO ]: 11:54:45 | Schematron validation failed:
> > [ INFO ]: 11:54:45 | --> boot attribute is only available for the
> > following image types: oem pxe [ ERROR ]: 11:54:45 |
> > KiwiDescriptionInvalid: Schema validation for Configuration/config.xml
> > failed
>
> Yes this is expected, the boot attribute for ISO images is not supported
> anymore. It can only be used in OEM and PXE images with the current KIWI.
> Removing it from the <type> tag should solve this issue.

Without the boot attribute, how do you specify which version of OpenSUSE to build against? I thought that was the primary purpose of this field.

But let me try your advice:

* Having removed the boot attribute, Kiwi now complains that I haven't asked to install Grub-EFI.
* If I ask Kiwi to include Grub-EFI, now it complains that I have to ask for Dracut-kiwi-live.
* If I ask for Dracut-kiwi-live, then Kiwi can't find an RPM for that.

So if I download the RPM [when the OpenSUSE website stops giving me HTTP 503 and actually responds to my requests] then I get this:

...
INFO: --> Setting up shim secure boot efi image
DEBUG: EXEC: [rsync -z -a --exclude /*.module ..../Kiwi/Root/usr/lib/grub2/x86_64-efi/ ..../Kiwi/Image-1/live-media.v5kvek3w/boot/grub2/x86_64-efi]
ERROR: KiwiBootLoaderGrubSecureBootError: Microsoft signed shim loader not found

I have no idea how to fix that.

> > I’m still not sure what it’s unhappy about. This configuration worked
> > just fine before the update, but now it’s complaining.
>
> I guess you were testing with a way older KIWI version before.

Kiwi-NG 9.9.6.2, in fact.

I don't update Kiwi very often, because when I do it tends to have breaking changes. I know it probably doesn't sound like much, but right now we have lost the ability to build anything. (Certainly the ability to build any new ISO images; I'm unsure if normal disk images still work.) I'm going to have to sort out what needs to be changed to make Kiwi happy again. And then I'm going to have to commit those changes to source control for every project we build. And even then, that only allows us to build the latest versions. If we ever need to rebuild an old image for some reason, we can't do it anymore.

> In short,
> current KIWI supports dracut modules in other to create the initrd, being the
> only choice for ISO and VMX. Before using dracut KIWI used to create its own
> intitrd based on a KIWI description that was referenced with the `boot`
> <type> attribute.

Is this documented somewhere? The only documentation link I have for Kiwi is

https://suse.github.io/kiwi/index.html

which seems rather sparse compared to even the incomplete documentation that legacy Kiwi had. Most particularly, it would be useful if there was somewhere that lists breaking changes and what to do about them.

Thanks,
Andrew.



Andrew Coppin
Software Developer

Tel:
Fax:
Email: mailto:Andrew...@evidencetalks.com

The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confident and/or privileged material. Any review, retransmission, dissimination or other use of, or taking of any action in reliance upon this informationby persons or entities other than the intended recipient is prohibited. If you recieved this in error, please contact the sender and destroy any copies of this information.

[Marcus Schäfer]

Hi,

> There's a good chance that quite a few environment variables aren't set. I wasn't expecting Kiwi to be looking for $HOME. If that's all the problem is, I guess I could set $HOME to... something.

kiwi supports reading a config file. Thus the expectation that some
HOME environment is set exists without further checking. We could make
this code path more stable but for the moment I suggest to set a HOME
as I think there must be some user identity under which you drive the
build ?

> * Having removed the boot attribute, Kiwi now complains that I haven't asked to install Grub-EFI.
> * If I ask Kiwi to include Grub-EFI, now it complains that I have to ask for Dracut-kiwi-live.
> * If I ask for Dracut-kiwi-live, then Kiwi can't find an RPM for that.

What's your image target distribution

dracut-kiwi-live is a sub package of the python-kiwi package. It exists
starting with SLES12-SP3

> ...
> INFO: --> Setting up shim secure boot efi image
> DEBUG: EXEC: [rsync -z -a --exclude /*.module ..../Kiwi/Root/usr/lib/grub2/x86_64-efi/ ..../Kiwi/Image-1/live-media.v5kvek3w/boot/grub2/x86_64-efi]
> ERROR: KiwiBootLoaderGrubSecureBootError: Microsoft signed shim loader not found
>
> I have no idea how to fix that.

Your image description seems to request secure boot with firmware="uefi"
The Microsoft signed shim loader is provided with the shim package. Thus
you need

<package name="shim"/>

As the package name is not consistent accross distributions we just write
a message that we need the "Microsoft signed shim loader" and assume the
author of the image description who requested it with the firmware
attribute to know the package name for the target distro

> > intitrd based on a KIWI description that was referenced with the `boot`
> > <type> attribute.
>
> Is this documented somewhere? The only documentation link I have for Kiwi is

yes some words about that here:
http://suse.github.io/kiwi/overview/workflow.html#customizing-the-boot-process

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer (Res. & Dev.) SUSE Linux GmbH
Tel: 0911-740 53 0 Maxfeldstrasse 5
FAX: 0911-740 53 479 D-90409 Nürnberg
HRB: 21284 (AG Nürnberg) Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton
http://www.suse.de
-------------------------------------------------------

[David Cassany]

Hi,

> Without the boot attribute, how do you specify which version of OpenSUSE to
> build against? I thought that was the primary purpose of this field.

No, in fact the only thing determines the version you are building against are
the repositories you define. The boot attribute was only used to determine the
image description for the initrd. In fact, kiwi is unaware of the distribution
you aim to build.

> But let me try your advice:
>
> * Having removed the boot attribute, Kiwi now complains that I haven't asked
> to install Grub-EFI.

This is a runtime check validation to ensure grub-efi package is included when
firmware="uefi" or firmware="efi".

> * If I ask Kiwi to include Grub-EFI, now it complains
> that I have to ask for Dracut-kiwi-live.

Passed previous runtime check now reclaims for dracut-kiwi-live package, this
package is the dracut module used in ISOs. As the message states including
this package into the packages list will solve the issue.

> * If I ask for Dracut-kiwi-live,
> then Kiwi can't find an RPM for that.

This happens because the package was not there at the time of opensuse 42.2,
however you can find it here in the KIWI repo

https://download.opensuse.org/repositories/Virtualization:/Appliances:/
Builder/openSUSE_Leap_42.2/

> So if I download the RPM [when the OpenSUSE website stops giving me HTTP 503
> and actually responds to my requests] then I get this:
>
> ...
> INFO: --> Setting up shim secure boot efi image
> DEBUG: EXEC: [rsync -z -a --exclude /*.module
> ..../Kiwi/Root/usr/lib/grub2/x86_64-efi/
> ..../Kiwi/Image-1/live-media.v5kvek3w/boot/grub2/x86_64-efi] ERROR:
> KiwiBootLoaderGrubSecureBootError: Microsoft signed shim loader not found

You are missing the shim package in packages section. `shim` package is the
UEFI loader, you get this because of firmware="uefi". It is hard to make
runtime checks for this package since we aim to support several distributions
and it is called/installed differently in each distro.

> I'm going to have
> to sort out what needs to be changed to make Kiwi happy again. And then I'm
> going to have to commit those changes to source control for every project
> we build. And even then, that only allows us to build the latest versions.
> If we ever need to rebuild an old image for some reason, we can't do it
> anymore.

Well, nothing prevents you to use older KIWI for older images, in fact, for
the images that are already working, I'd say the safest is just to leave them
as they are. Note that former kiwi v7 and kiwi-ng can coexist in the same
system, the main motivation for it to not force a transition on older images.

> Is this documented somewhere? The only documentation link I have for Kiwi is
>
> https://suse.github.io/kiwi/index.html

This is the correct documentation link. The change to dracut is documented
within the schema docs (boot attribute not supported) and the customization of
the boot process documentation has been adapted to that.

Regards,
David

[Andrew Coppin]

> Hi,
>
> > There's a good chance that quite a few environment variables aren't set. I
> wasn't expecting Kiwi to be looking for $HOME. If that's all the problem is, I
> guess I could set $HOME to... something.
>
> kiwi supports reading a config file. Thus the expectation that some HOME
> environment is set exists without further checking. We could make this code
> path more stable but for the moment I suggest to set a HOME as I think there
> must be some user identity under which you drive the build ?

Setting the variable seems to fix the problem.

> > * Having removed the boot attribute, Kiwi now complains that I haven't
> asked to install Grub-EFI.
> > * If I ask Kiwi to include Grub-EFI, now it complains that I have to ask for
> Dracut-kiwi-live.
> > * If I ask for Dracut-kiwi-live, then Kiwi can't find an RPM for that.
>
> What's your image target distribution

For this particular build, OpenSUSE Leap 42.2.

> > ...
> > INFO: --> Setting up shim secure boot efi image
> > DEBUG: EXEC: [rsync -z -a --exclude /*.module
> > ..../Kiwi/Root/usr/lib/grub2/x86_64-efi/
> > ..../Kiwi/Image-1/live-media.v5kvek3w/boot/grub2/x86_64-efi]
> > ERROR: KiwiBootLoaderGrubSecureBootError: Microsoft signed shim loader
> > not found
> >
> > I have no idea how to fix that.
>
> Your image description seems to request secure boot with firmware="uefi"
> The Microsoft signed shim loader is provided with the shim package. Thus
> you need
>
> <package name="shim"/>
>
> As the package name is not consistent accross distributions we just write a
> message that we need the "Microsoft signed shim loader" and assume the
> author of the image description who requested it with the firmware attribute
> to know the package name for the target distro

Well, that fixes that particular problem... Now I have:

DEBUG: EXEC: [du -s --apparent-size --block-size 1 ..../Kiwi/Image-1/live-media.75737xzu]
INFO: Creating live ISO image
ERROR: KiwiIsoLoaderError: No isolinux loader %s found

> > > intitrd based on a KIWI description that was referenced with the
> > > `boot` <type> attribute.
> >
> > Is this documented somewhere? The only documentation link I have for
> > Kiwi is
>
> yes some words about that here:
> http://suse.github.io/kiwi/overview/workflow.html#customizing-the-boot-
> process

Thanks for the info.

[Andrew Coppin]

> > Without the boot attribute, how do you specify which version of
> > OpenSUSE to build against? I thought that was the primary purpose of this
> field.
>
> No, in fact the only thing determines the version you are building against are
> the repositories you define. The boot attribute was only used to determine
> the image description for the initrd. In fact, kiwi is unaware of the distribution
> you aim to build.

Interesting... I had no idea that was the case. I thought Kiwi would have to do a bunch of distro-specific stuff in each case.

> > * If I ask for Dracut-kiwi-live,
> > then Kiwi can't find an RPM for that.
>
> This happens because the package was not there at the time of opensuse
> 42.2, however you can find it here in the KIWI repo

Seems reasonable.

> > I'm going to have
> > to sort out what needs to be changed to make Kiwi happy again. And
> > then I'm going to have to commit those changes to source control for
> > every project we build. And even then, that only allows us to build the
> latest versions.
> > If we ever need to rebuild an old image for some reason, we can't do
> > it anymore.
>
> Well, nothing prevents you to use older KIWI for older images, in fact, for the
> images that are already working, I'd say the safest is just to leave them as
> they are. Note that former kiwi v7 and kiwi-ng can coexist in the same
> system, the main motivation for it to not force a transition on older images.

No, I mean, I have several images that build with Kiwi-NG, I updated Kiwi-NG, and now none of them build any more.

Short of running one build server for every point-release of Kiwi [we don't have enough licenses to do that], I'm not sure how to avoid this problem in future.

[Marcus Schäfer]

Hi,

> DEBUG: EXEC: [du -s --apparent-size --block-size 1 ..../Kiwi/Image-1/live-media.75737xzu]
> INFO: Creating live ISO image
> ERROR: KiwiIsoLoaderError: No isolinux loader %s found

Looks like there is no syslinux package installed

<package name="syslinux"/>

which python-kiwi package do you use ? We switched to xorriso
some time ago

[Andrew Coppin]

> > DEBUG: EXEC: [du -s --apparent-size --block-size 1
> > ..../Kiwi/Image-1/live-media.75737xzu]
> > INFO: Creating live ISO image
> > ERROR: KiwiIsoLoaderError: No isolinux loader %s found
>
> Looks like there is no syslinux package installed
>
> <package name="syslinux"/>

Ah, OK. I couldn't work out the package name.

> which python-kiwi package do you use ? We switched to xorriso some time
> ago

Python3-kiwi-9.15.3-103.1, apparently.

Thanks,