Boxbuild fails in kiwi/bootloader/config/grub2.py

[Sven Wölfel]

Was there a kiwi-ng update in the latest universal box?

 When I use the attached config.xml I get the following error during a

kiwi-ng --loglevel=10 --debug --profile kvm system boxbuild --box universal -- --description ./sles --target-dir ./img-out

run:

[ ERROR   ]: 08:48:05 | Unexpected error:
Traceback (most recent call last):
  File "/usr/bin/kiwi-ng", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/usr/lib/python3.13/site-packages/kiwi/kiwi.py", line 68, in main
    App()
    ~~~^^
  File "/usr/lib/python3.13/site-packages/kiwi/app.py", line 35, in __init__
    task_class().process()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/kiwi/tasks/system_build.py", line 355, in process
    result = image_builder.create()
  File "/usr/lib/python3.13/site-packages/kiwi/builder/disk.py", line 276, in create
    result = self.create_disk()
  File "/usr/lib/python3.13/site-packages/kiwi/builder/disk.py", line 382, in create_disk
    self._build_main_system(
    ~~~~~~~~~~~~~~~~~~~~~~~^
        stack,
        ^^^^^^
    ...<9 lines>...
        self.storage_map['integrity_root']
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/kiwi/builder/disk.py", line 810, in _build_main_system
    self._write_bootloader_meta_data_to_system_image(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        device_map, disk, system, bootloader_config
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/kiwi/builder/disk.py", line 1463, in _write_bootloader_meta_data_to_system_image
    bootloader_config.write_meta_data(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        root_device=ro_device.
        ^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        boot_options=' '.join(boot_options)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/kiwi/bootloader/config/grub2.py", line 243, in write_meta_data
    self._setup_default_grub()
    ~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/kiwi/bootloader/config/grub2.py", line 808, in _setup_default_grub
    enable_blscfg_implemented = Command.run(
        [
    ...<2 lines>...
        ], raise_on_error=False
    )
  File "/usr/lib/python3.13/site-packages/kiwi/command.py", line 136, in run
    log.debug('EXEC: [%s]', ' '.join(command))
                            ~~~~~~~~^^^^^^^^^
TypeError: sequence item 3: expected str instance, NoneType found
+ echo 0
+ finish
+ grep -q kiwi-no-halt /proc/cmdline
+ halt -p
Failed to set wall message, ignoring: Could not activate remote peer 'org.freedesktop.login1': activation request failed: unit is invalid
Call to PowerOff failed: Could not activate remote peer 'org.freedesktop.login1': activation request failed: unit is invalid

The same command with this config worked last Thursday (according to our Pipeline log).

Was there a change in the supported schema version?

Full log and config is attached.

[Marcus Schäfer]

Hi,

> Was there a kiwi-ng update in the latest universal box?

This box is based on Fedora RawHide and rebuilds according
to the development of this distribution automatically.
The error you saw is very strange and I cannot reproduce it.
A minute ago I called this:

kiwi-ng system boxbuild --box universal -- --description ~ms/Project/kiwi-descriptions/suse/x86_64/suse-leap-15.6/ --target-dir /tmp/mytest-boxbuild

and it fetched the latest universal box with this sha sum

cat .kiwi_boxes/universal/Universal-Box.x86_64-1.41.1-System-BuildBox.report.sha256

879e2072cfd56d6d7c91659da443c146fcf36c0a255dd65599589f39a6a7b5f1

Sorry I can't reproduce the error you were facing

There were some changes on the box plugin but not on the actual
code just an update of the box definition for the ubuntu box

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------

[Sven Woelfel]

Thank you for checking! I can reproduce the error with my debian config
as well.

Is there a way to "checkout" an older box version? I just wan't to make
sure it is either related to the box or my configuration (which would be
weird since it worked fine last Thursday).


Regards

Sven

[Sven Woelfel]

Ok,

I'm getting a bit confused now....

Using the same config as yesterday with the tumbleweed box worked just fine.

kiwi-ng --loglevel=10 --debug --profile kvm system boxbuild --box tumbleweed -- --description ./sles --target-dir ./img-out

I also managed to reproduce the error with the attached Debian Bookworm config. This is an enhanced version of the one in the kiwi-descriptions Github Repo and has no internal apt mirrors configured. So it should be possible to reproduce the behavior with

kiwi-ng --loglevel=10 --debug --profile kvm system boxbuild --box universal -- --description ./<folder with attached config> --target-dir ./img-out

As I write this I try to use this config with the ubuntu box in the hope of having a tumbleweed box with sles description miracle but the build is stuck at the apt-get update step for over 30 minutes.

Trying to set the kiwi version with

kiwi-ng system boxbuild ... --kiwi-version 10.2.7 ...

did not work. I'm behind a corporate proxy and it seems pip does not like that fact.

In fact pip doesn't even use the proxy env vars on my sles host and I have to explicitly tell pip to use my proxy with the --proxy option. But I have no idea how to pass that to the boxbuild environment.

@Marcus: Would you be so kind and try the attached debian config? At the moment I'm suspecting a problem with the installed python version in the universal box but that is just a wild guess.

Regards

Sven

--
You received this message because you are subscribed to a topic in the Google Groups "kiwi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kiwi-images/ZQS28FX4Q-g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kiwi-images...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kiwi-images/b0633ca1-c39e-46ec-a300-02a7b37b9633n%40googlegroups.com.

[Sven Woelfel]

Ok, another update:

Using the sles config.xml and a regular system build (kiwi-ng --profile kvm system build --description ./sles )on a sles 15 SP6 host with python 3.11.10 and kiwi-ng version 10.2.7 works as well.

The previous posted Debian config works as well when using the ubuntu box for the boxbuild. Although I had to press enter once at the apt-get update phase for it to continue the build, otherwise it is stuck at this phase.

So in summary:

- sles build on tumbleweed box works

- debian build on ubuntu box works (although weird behaviour at the apt-get update phase)

- sles build without boxbuild works as well

- debian without boxbuild can not be tested atm since I don't have a apt based host at hand that could serve as the build host and kiwi-ng moved away from debootstrap

- sles and debian build fail with a boxbuild on the universal box

Regards

Sven

On 20.01.25 10:31, Sven Wölfel wrote:

[Marcus Schäfer]

Hi Sven,

I found the issue with the unexpected error. The problem is
that there is no PATH set in the environment of the box. In
a recent change in kiwi we use os.defpath in such a case.
However, this now leads to the problem that Path.which()
searches only in

```
print(os.defpath)
:/bin:/usr/bin
```

but tools like grub-mkconfig are found in /usr/sbin. That
results in Path.which() to return a None value which is then
later used in a join which raises an exception and that's the
point where you ended up with a weird stack trace.

The reason why you only see it occasionally is because of
different PATH settings.

I'm going to fix this in the boxbuild plugin and I also add
a patch to kiwi that we don't end up in an unexpected error
followed by a stack trace

Stay tuned, I'm on it

PS: Your image builds for me if I start the box with --box-debug
then wait for the first fail, then export PATH in the box debug
shell and call 'run_kiwi' again

[Marcus Schäfer]

Hi,

> I'm going to fix this in the boxbuild plugin and I also add
> a patch to kiwi that we don't end up in an unexpected error
> followed by a stack trace

Two patches out for review:

* https://github.com/OSInside/kiwi-boxed-plugin/pull/80
* https://github.com/OSInside/kiwi/pull/2715

The first one is mandatory, the second one optional because
with a proper PATH setting you don't run into it

[Sven Woelfel]

Hi Marcus.

thank you for the fast follow up!

> I found the issue with the unexpected error. The problem is

> that there is no PATH set in the environment of the box. In

> a recent change in kiwi we use os.defpath in such a case.

> However, this now leads to the problem that Path.which()

> searches only in

> ```

> print(os.defpath)

> :/bin:/usr/bin

> ```

>

> but tools like grub-mkconfig are found in /usr/sbin. That

> results in Path.which() to return a None value which is then

> later used in a join which raises an exception and that's the

> point where you ended up with a weird stack trace.

Ok I was way of with me suspecting the python version of the universal box being the culprit...

> Stay tuned, I'm on it

Thank you!

Regards

Sven

[Marcus Schäfer]

Hi Sven,

Your image now builds for me. There are new boxes built which
fixes the PATH issue. However, they are not yet published.
You can fetch the binaries via 'osc getbinaries' from the obs
backend server but on the repos in download.opensuse.org I still
see the old binaries.

[Sven Woelfel]

Hi Marcus,

That was quick. Thank you!

I won't be able to test the new images until Monday. Bus as soon as I
did I will give you feedback.

Thanks again for your fast follow up.


Regards

Sven

[Marcus Schäfer]

Hi,

> That was quick. Thank you!

You are welcome :)

> I won't be able to test the new images until Monday. Bus as soon as I did I
> will give you feedback.

Thanks.

For all box plugin users; I had to update the plugin config and
the way how the boxes builds due to this PR:

https://github.com/OSInside/kiwi-boxed-plugin/pull/81

So please make sure to update the plugin to the latest version
v0.2.45

Thanks

Have a good weekend

[Sven Wölfel]

Hi Markus,

boxed-plugin is updated to version 0.2.45 But there seems to be an issue with the download target:

[ INFO    ]: 07:31:55 | Reading box plugin config file: /usr/local/lib/python3.11/site-packages/kiwi_boxed_plugin/config/kiwi_boxed_plugin.yml
[ DEBUG   ]: 07:31:55 | Creating directory /home/bfd.bayern.de/lff-a-woelfels/.kiwi_boxes/universal
[ INFO    ]: 07:31:55 | Validating kiwi_build_command_args:
    ['system', 'build', '--description', './sles', '--target-dir', './img-out']
[ INFO    ]: 07:31:55 | Building with:
    ['--debug', '--profile', 'kvm', 'system', 'build', '--description', './sles', '--target-dir', './img-out']
[ DEBUG   ]: 07:31:55 | Creating directory ./img-out
[ ERROR   ]: 07:31:56 | KiwiUriOpenError: HTTPError: HTTP Error 404: Not Found http://download.opensuse.org/repositories/Virtualization:/Appliances:/SelfContained:/universal/images//Universal-Box.x86_64-System.json

I tried to open it in the Browser to rule out proxy issues, but there it is 404 as well.

Regards

Sven

[Marcus Schäfer]

Hi Sven,

> boxed-plugin is updated to version 0.2.45 But there seems to be an
> issue with the download target:

grrr, yes sbom publishing is disabled by default for non suse images.
And of course I tested only with a box that has it (tumbleweed)

I'm fixing the project setup, sorry for this one

[Sven Wölfel]

No issue, it would not be a complete Monday without an error or two.

Sorry for writing your name wrong, my autocorrect seems to have an issue with 'c' in names like Eric and Marcus :-/

Regards

Sven

[Marcus Schäfer]

Hi,

> No issue, it would not be a complete Monday without an error or two.

:-)) well said

> Sorry for writing your name wrong, my autocorrect seems to have an
> issue with 'c' in names like Eric and Marcus :-/

As you said, no issue ;) and the one with the missing sbom should
be fixed now. At least my universal based build worked

[Sven Wölfel]

Hello Marcus,

sorry to reply so late. I came around to testing the changes today and I was greeted with a somewhat strange behavior.

1) The command "kiwi-ng --loglevel=10 --debug --profile kvm system boxbuild --box universal --  --description ./sles --target-dir ./img-out" exits with:

[ DEBUG   ]: 08:02:41 | Creating directory ./img-out
[ INFO    ]: 08:02:43 | Downloading Universal-Box.x86_64-Kernel.tar.xz
Loading:  25% [#########################################                                                                                                                           ] ETA:   0:00:15   2.9 MiB/s[ ERROR   ]: 08:02:55 | Unexpected error:

Traceback (most recent call last):

  File "/usr/local/lib/python3.11/site-packages/urllib3/response.py", line 754, in _error_catcher
    yield
  File "/usr/local/lib/python3.11/site-packages/urllib3/response.py", line 900, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(15251620 bytes read, 45736836 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 820, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/usr/local/lib/python3.11/site-packages/urllib3/response.py", line 1066, in stream
    data = self.read(amt=amt, decode_content=decode_content)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/response.py", line 983, in read
    data = self._raw_read(amt)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/response.py", line 878, in _raw_read
    with self._error_catcher():
  File "/usr/lib64/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/urllib3/response.py", line 778, in _error_catcher
    raise ProtocolError(arg, e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(15251620 bytes read, 45736836 more expected)', IncompleteRead(15251620 bytes read, 45736836 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/local/bin/kiwi-ng", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kiwi/kiwi.py", line 68, in main
    App()
  File "/usr/local/lib/python3.11/site-packages/kiwi/app.py", line 35, in __init__
    task_class().process()
  File "/usr/local/lib/python3.11/site-packages/kiwi_boxed_plugin/tasks/system_boxbuild.py", line 191, in process
    box_build.run(
  File "/usr/local/lib/python3.11/site-packages/kiwi_boxed_plugin/box_build.py", line 111, in run
    vm_setup = self.box.fetch(update_check)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kiwi_boxed_plugin/box_download.py", line 123, in fetch
    fetcher.wget(
  File "/usr/local/lib/python3.11/site-packages/kiwi_boxed_plugin/utils/fetch_files.py", line 57, in wget
    for chunk in response.iter_content(chunk_size=chunk_size):
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 822, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(15251620 bytes read, 45736836 more expected)', IncompleteRead(15251620 bytes read, 45736836 more expected))

When trying again the download works.

2) Kiwi always exists with the following error message(when the initial download hickup is resolved):

 ❯ kiwi-ng --loglevel=10 --debug --profile kvm system boxbuild --box universal --  --description ./sles --target-dir ./img-out
[ INFO    ]: 08:31:33 | Reading runtime config file: '/etc/kiwi.yml'
[ INFO    ]: 08:31:33 | Reading box plugin config file: /home/manji/.pyenv/versions/3.11.0/lib/python3.11/site-packages/kiwi_boxed_plugin/config/kiwi_boxed_plugin.yml
[ DEBUG   ]: 08:31:33 | Creating directory /home/manji/.kiwi_boxes/universal
[ INFO    ]: 08:31:33 | Validating kiwi_build_command_args:

    ['system', 'build', '--description', './sles', '--target-dir', './img-out']

[ INFO    ]: 08:31:33 | Building with:

    ['--debug', '--profile', 'kvm', 'system', 'build', '--description', './sles', '--target-dir', './img-out']

[ DEBUG   ]: 08:31:33 | Creating directory ./img-out
[ DEBUG   ]: 08:31:35 | Looking for tar in /home/manji/.pyenv/versions/3.11.0/bin:/usr/lib/pyenv/libexec:/home/manji/.pyenv/plugins/python-build/bin:/home/manji/.pyenv/plugins/pyenv-virtualenv/bin:/home/manji/.pyenv/plugins/pyenv-update/bin:/home/manji/.pyenv/plugins/pyenv-installer/bin:/home/manji/.pyenv/plugins/pyenv-doctor/bin:/home/manji/.pyenv/shims:/home/manji/.vscode-server/cli/servers/Stable-91fbdddc47bc9c09064bf7acf133d22631cbf083/server/bin/remote-cli:/home/manji/.local/bin:/home/manji/bin:/usr/local/bin:/usr/bin:/bin:/snap/bin
[ DEBUG   ]: 08:31:35 | EXEC: [tar -C /home/manji/.kiwi_boxes/universal --transform s/.*/kernel.x86_64/ --wildcards -xf /home/manji/.kiwi_boxes/universal/Universal-Box.x86_64-Kernel.tar.xz *.kernel]
[ DEBUG   ]: 08:31:35 | EXEC: Failed with stderr: xz: (stdin): File format not recognized
/usr/bin/tar: Child returned status 1
/usr/bin/tar: Error is not recoverable: exiting now
, stdout: (no output on stdout)
[ ERROR   ]: 08:31:35 | KiwiCommandError: tar: stderr: xz: (stdin): File format not recognized
/usr/bin/tar: Child returned status 1
/usr/bin/tar: Error is not recoverable: exiting now
, stdout: (no output on stdout)

I updated kiwi and the boxed plugin to the latest possible version(10.0.0) for my installed python version (3.11).

Same problems occurs for tumbleweed and ubuntu boxes.

The build environment is the same I used before. Any idea what could cause this?

Regards

Sven

[Sven Wölfel]

Ok, I updated kiwi-ng on all my systems to 10.2.9. The error still happens on my build systems at work but not on my private laptop.

So I would guess the company proxy intervenes.

Looking at the downloaded kernel file  "Universal-Box.x86_64-Kernel.tar.xz" I get the following output from file which makes it clear why tar complains about an unknown file type:

Universal-Box.x86_64-Kernel.tar.xz: HTML document, UTF-8 Unicode text, with very long lines

I will contact our IT if there is a change in the proxy configuration.

Where does kiwi get the information which proxy to use? Via the environment variable http_proxy oder some other mechanism?

Regards

Sven

[Marcus Schäfer]

Hi,

> requests.exceptions.ChunkedEncodingError: ('Connection broken:
> IncompleteRead(15251620 bytes read, 45736836 more expected)',
> IncompleteRead(15251620 bytes read, 45736836 more expected))

Whot, never could that type of error from requests, which is also
the reason why it's not handled and gives you that stack trace

> When trying again the download works.

great ;)

> Ok, I updated kiwi-ng on all my systems to 10.2.9. The error still
> happens on my build systems at work but not on my private laptop.
>
> So I would guess the company proxy intervenes.

Yes it would not be the first time

> Where does kiwi get the information which proxy to use? Via the
> environment variable http_proxy oder some other mechanism?

So you are building via boxbuild and the proxy setup must be
effective inside the box. To bring in that setup into the box
prior kiwi is called you can do the following:

* Add a directory named boxroot/ to your image description.
All data that is inside boxroot will be rsynced into the box
prior the kiwi call

* In addition add boxroot/etc/boxprofile.
This file gets shell sourced and allows to influence the
runtime environment. For example setting HTTP_PROXY
As it's sourced you can also invoke code in the box
if need be

With that you should have a way to setup everything to meet
your company requirements withou us having to modify the boxes.

Hope this helps

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer

[Sven Woelfel]

Ok,

I now know what happened and got it fixed. We got an exception for the boxbuild mirror in our company proxy/firewall solution. This was a bigger problem than I initially thought, and it also affected our RMT server.

Hi,

   requests.exceptions.ChunkedEncodingError: ('Connection broken:
   IncompleteRead(15251620 bytes read, 45736836 more expected)',
   IncompleteRead(15251620 bytes read, 45736836 more expected))

Whot, never could that type of error from requests, which is also
the reason why it's not handled and gives you that stack trace

   When trying again the download works.

great ;)

Yes and no. Yes the download worked, but it didn't download the Universal-Box.x86_64-Kernel.tar.xz but instead a webpage. The proxy/firewall suite our company uses deemed the file  Universal-Box.x86_64-Kernel.tar.xz a potential virus and informed me with a website that it prevented the download of it.

wget saves this page as Universal-Box.x86_64-Kernel.tar.xz and hence the tar extract command later on fails, since it is just a tar by name not by content.

I know this is a pretty exotic use case but shouldn't there be a checksum test in place like for the squashfs image of the box?

So you are building via boxbuild and the proxy setup must be
effective inside the box. To bring in that setup into the box
prior kiwi is called you can do the following:

* Add a directory named boxroot/ to your image description.
  All data that is inside boxroot will be rsynced into the box
  prior the kiwi call

* In addition add boxroot/etc/boxprofile.
  This file gets shell sourced and allows to influence the
  runtime environment. For example setting HTTP_PROXY
  As it's sourced you can also invoke code in the box
  if need be

With that you should have a way to setup everything to meet
your company requirements withou us having to modify the boxes.

The problem is not with the boxbuild but at the stage when kiwi pulls the box kernel and rootfs. But from what I can tell by the behavior kiwi showed it must get the proxy settings from the environment of the shell that executes kiwi.

Hope this helps

Yes, as always. Thank you!

Regards

Sven

[Marcus Schäfer]

Hi,

> I now know what happened and got it fixed. We got an exception for the
> boxbuild mirror in our company proxy/firewall solution. This was a
> bigger problem than I initially thought, and it also affected our RMT
> server.

OK, happy to hear you could fix it

> Yes and no. Yes the download worked, but it didn't download
> the Universal-Box.x86_64-Kernel.tar.xz but instead a webpage. The
> proxy/firewall suite our company uses deemed the file
> Universal-Box.x86_64-Kernel.tar.xz a potential virus and informed me
> with a website that it prevented the download of it.

LOL, a virus *pff*

> wget saves this page as Universal-Box.x86_64-Kernel.tar.xz and hence
> the tar extract command later on fails, since it is just a tar by name
> not by content.
>
> I know this is a pretty exotic use case but shouldn't there be a
> checksum test in place like for the squashfs image of the box?

You are absolutely right. This was just me being lazy. If the checksum
is ok for the image why should it fail for the kernel. So what
could possible go wrong... yeah everything as usual :)

I will fix this:

https://github.com/OSInside/kiwi-boxed-plugin/issues/84

At the moment I have other stuff to do that has to be done first.
But this one won't be forgotten.

Thanks

[Marcus Schäfer]

Hi,

> > I know this is a pretty exotic use case but shouldn't there be a
> > checksum test in place like for the squashfs image of the box?
>
> You are absolutely right. This was just me being lazy. If the checksum
> is ok for the image why should it fail for the kernel. So what
> could possible go wrong... yeah everything as usual :)
>
> I will fix this:

There is now a PR open:

https://github.com/OSInside/kiwi-boxed-plugin/pull/86