Boxbuild questions

[Sven Wölfel]

Hello,

since a few days ago I'm experiencing trouble with the build of ubuntu 24.04 in the universal build box.

I've got errors regarding inconsistent InRelease files and GPG check errors (despite disabling them in the config.xml). So I wanted to try using the upstream repositories without the interference of the apt-cache. But since I'm behind a corporate proxy I would need to provide the proxy information to the universal box.

I found this issue and the corresponding commit. My question now is how to use this boxprofile configuration. Is this just a simple environment file that is located at the hosts /etc folder  or do I need to prepare something special like an extra shared folder with a special configuration file (syntax wise)?

My other question is if the universal box contains the appx-util, or if there is a list of the utils that are installed in the universal box.

Also an non boxbuild question: Has someone a simple description example of a wsl image for Suse leap or Debian? I found the tumbleweed description in the kiwi repo but I wondered if there are others available to get a better Understanding what is possible (and what not).

With kind regards

Sven

[Marcus Schäfer]

Hi Sven,

> I've got errors regarding inconsistent InRelease files and GPG check
> errors (despite disabling them in the config.xml). So I wanted to try
> using the upstream repositories without the interference of the
> apt-cache. But since I'm behind a corporate proxy I would need to
> provide the proxy information to the universal box.

You can place an arbitrary data tree into the box with the boxroot/
overlay directory. So in your image description (the directory with the
kiwi file and maybe other files) create a directory named boxroot

mkdir boxroot/

You can now place data as you see fit below boxroot. When the boxplugin
starts it will copy all data as it is (rsync) into the box prior starting
the build. That way you should be able to provide all needed data

> My other question is if the universal box contains the appx-util, or if
> there is a list of the utils that are installed in the universal box.

Yes appx-util is set as a requirement to the kiwi-systemdeps-containers-wsl
package. All boxes that we build installs the kiwi-systemdeps package
which pulls in all other kiwi-systemdeps-XXX packages which then pulls
in the various tools. You can check for the included packages yourself
if you have the osc tool installed on your machine:

osc buildinfo Virtualization:Appliances:SelfContained:universal/box images x86_64 -M System | grep appx

<bdep name="appx-util" notmeta="1" version="0.5" release="5.fc41" arch="x86_64" hdrmd5="52fcbfe87c318c96678c2b6f20848f0f" project="Fedora:Rawhide" repository="standard"/>

> Also an non boxbuild question: Has someone a simple description example
> of a wsl image for Suse leap or Debian? I found the tumbleweed
> description in the kiwi repo but I wondered if there are others
> available to get a better Understanding what is possible (and what
> not).

I'm afraid we only build this one here:

https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-wsl
https://github.com/OSInside/kiwi/blob/main/build-tests/x86/tumbleweed/test-image-wsl/appliance.kiwi

as part of our integration test matrix. I'm also not too much involved
with Windows Subsystem for Linux. Maybe there are other people on the list
who can help with specific WSL questions.

Hope this helps

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer

[Sven Woelfel]

Hi Markus,

yes this helps a lot! Thank you!

Is the /etc/boxprofile file documented in some way or is it somewhat
like the /etc/sysconfig on suse systems? If there is a simple example
file somewhere in a commit or so this would be sufficient, I only need
to know how it is structured.

>I'm afraid we only build this one here:
>
> https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-wsl
> https://github.com/OSInside/kiwi/blob/main/build-tests/x86/tumbleweed/test-image-wsl/appliance.kiwi
>
>as part of our integration test matrix. I'm also not too much involved
>with Windows Subsystem for Linux. Maybe there are other people on the list
>who can help with specific WSL questions.

Ok thank you! Building WSL Images is new territory for me as well, I'm
happy about every piece of information I can get.


Thanks again!


Regards,

Sven

[Marcus Schäfer]

Hi,

> yes this helps a lot! Thank you!

You are welcome

> Is the /etc/boxprofile file documented in some way or is it somewhat like
> the /etc/sysconfig on suse systems?

It's just an environment file. If you for example write something like

export HTTP_PROXY=foo

and put that into a file

boxroot/etc/boxprofile

then it will be sourced into the runtime environment of the
kiwi call and should take effect on tools that reads it.

If you call the boxbuild with --box-debug this will keep the
VM open with a shell and you can check the environment and
all other files you expect to exist

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer

-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------

[Sven Wölfel]

Hi,

is there a way to see which environment the kiwi call used during a boxbuild?

I used --box-debug and was able to confirm that my boxconfig was placed in the etc folder of the boxbuild vm. The kiwi call still fails with network errors and the regular shell env does not have any of the proxy variables I exported via my boxconfig.

So I was wondering if kiwi-ng uses its own environment and where to see hot it is configured.

Regards,

Sven

[Marcus Schäfer]

Hi,

> I used --box-debug and was able to confirm that my boxconfig was placed
> in the etc folder of the boxbuild vm. The kiwi call still fails with
> network errors and the regular shell env does not have any of the proxy
> variables I exported via my boxconfig.

Hmm, yeah sorry my fault. The debug shell is a serial console and
the kiwi process runs inside of a new sub-shell. In that sub-shell
the environment for the call is setup. So looking at this env is
indeed not so easy.

If you are in the debug shell you can see the code that runs kiwi
in the following file:

/usr/local/bin/run_kiwi

In that file you'll find among others the following method:

function import_box_environment {
# """
# Import optional etc/boxprofile into runtime environment
# """
if [ -e /etc/boxprofile ];then
source /etc/boxprofile
fi
}

You can manually modify the code here and print the env after
sourcing it. Once you changed it just repeat the build by
calling:

run_kiwi

and you should get more information

For changing the code the I usually copy the file to the
/bundle directory:

# in-the-box-debug-shell: cp /usr/local/bin/run_kiwi /bundle

After that you can modify it on your host (because /bundle is
shared with the guest and represents the directory you provided
as --target-dir)

Once you modified it, copy it back

# in-the-box-debug-shell: cp /bundle/run_kiwi /usr/local/bin/run_kiwi

I mention that because inside of the box you have limited
tools to work with and editing tasks in a serial console can
drive you crazy :)

Let me know if you can find the issue that way

[Neal Gompa]

I'm pretty sure this has come up before, but to actually configure the
proxy, you need to configure NetworkManager. The environment variable
version does not work.

https://www.networkmanager.dev/docs/api/latest/settings-proxy.html

Unfortunately, unlike Windows and macOS, Linux doesn't have a simple
unified network proxy mechanism. NetworkManager is the closest.



--
真実はいつも一つ！/ Always, there's only one truth!

[Neal Gompa]

nmcli documentation:
https://www.networkmanager.dev/docs/api/latest/nm-settings-nmcli.html

[Sven Wölfel]

Hi,

> Let me know if you can find the issue that way

Not yet but thank you for the information.

> I'm pretty sure this has come up before, but to actually configure the
> proxy, you need to configure NetworkManager. The environment variable
> version does not work.

It is news to me, I was under the impression boxbuild would solve the issue with kiwi and building images behind a proxy.

So in order to make a proxy build work I always have to drop in to the debug terminal?

Since it is for testing now it is not a real issue but for a production environment this is not really working.

Anyway I tried to create a simple pac file that redirects all outgoing traffic over the proxy and placed it in the boxroot/etc directory, so it will be copied to the build vm.

function FindProxyForURL(url, host) {
    // Exclude localhost
    if (
        isPlainHostName(host) ||
        isInNet(dnsResolve(host), "127.0.0.0", "255.255.255.0")
    )
        return "DIRECT";
    else return "PROXY <proxyurl>:<port>";

I then start the build process and wait for the build to fail in order to get thrown into the debug terminal session.

There I get the NM connection UID with "nmcli c show" and edit it with nmcli c e <UID>. This drops me in the NM edit shell.

From here I just use " set proxy.method auto" and "set proxy.pac-script file:///etc/proxy.pac", save the connection with "save" and the connection should be using the created pac file.

I then re-initiate the build with run_kiwi but the problem remains, kiwi can not talk to the repository server over the proxy.

Did I miss something?

Regards

Sven

[Marcus Schäfer]

Hi,

> > I'm pretty sure this has come up before, but to actually configure
> the
> > proxy, you need to configure NetworkManager. The environment variable
> > version does not work.
>
> It is news to me, I was under the impression boxbuild would solve the
> issue with kiwi and building images behind a proxy.

boxbuild allows you to run the build in an isolated environment.
We provide a number of boxes which usually are good enough to
run the build. However, depending on network constraints or
other limitations not known to us, there are situations for
which our box setup is not suitable.

In such a case the current process of debugging helps to identify
what needs to change in order to successfuly build. Once this is
clear there is either a mechanic to pre-setup the box to fullfil
the requirement or in the worst case you build your own box together
with our help.

On a personal note. For a real production build I would always
clone the used box as we provide it into a private space such
that you are decoupled from changes and can decide yourself
when to add changes to this isolated environment.

Let's hope we find the connection issue soon.

[Sven Wölfel]

Hi,

thank you for the offer. In a real productive scenario I would probably prefer to use a package mirror in our local network to eliminate the proxy alltogether. But since I'm not sure if our apt-cacher has a glitch or if it is a problem with the kiwi config I tried to use upstream servers to eliminate the possibility of a faulty apt-cache.

Sorry if you misunderstood my comment, I was just referring to this issue and the discussion of it:

https://github.com/OSInside/kiwi/issues/2234 

There is was stated that the kiwi way of configuring a proxy only worked fine for Suse based systems and boxbuild (probably) solves this issue. As it stands now it seems it does not.

I'm a bit stuck now. I have no idea how to verify if the altered NM connection is used by the kiwi build or if there is an issue with it.

I know that proxy environments are a niche case but I would still prefer kiwi-ng to be able to handle a proxy connection directly instead of relying on a buildvm to solve this. Just my two cents, I know this is easier said then done. As Neal pointed out, there is no standardized way of handling proxy connections in linux.

Regards

[Marcus Schäfer]

Hi,

> thank you for the offer. In a real productive scenario I would probably
> prefer to use a package mirror in our local network to eliminate the
> proxy alltogether.

Yes sounds good

> But since I'm not sure if our apt-cacher has a
> glitch or if it is a problem with the kiwi config I tried to use
> upstream servers to eliminate the possibility of a faulty apt-cache.

ok got it

> Sorry if you misunderstood my comment, I was just referring to this
> issue and the discussion of it:
>
> https://github.com/OSInside/kiwi/issues/2234
>
> There is was stated that the kiwi way of configuring a proxy only
> worked fine for Suse based systems and boxbuild (probably) solves this
> issue. As it stands now it seems it does not.
>
> I'm a bit stuck now. I have no idea how to verify if the altered NM
> connection is used by the kiwi build or if there is an issue with it.

ok makes sense. So yes a proxy setup varies from one system to the other
and it is probably not possible to let kiwi do the job to serve them
all. I think we need to clarify first where exactly we have the connection
issue.

kiwi operates in two stages to build up the new root tree:

1. bootstrap
2. system

In the bootstrap phase all tools are called directly from the
build host. The build host in your case is the box.

My first question here is:

Q: Independent of kiwi, can the box network setup reach the
endpoints you are targeting ?

In the system phase all tools are called as chroot() operation.
This means all required steps to let the build host connect
via the proxy also needs to be done in the new root such that
a "chroot new-root apt ..." has a chance to succeed. As you
know from the issue report, at this stage kiwi is limited in
a way that we only copy "/etc/sysconfig/proxy" into the new
root and that is not enough for Debian based distros. To allow
you to run the required changes in the new root kiwi has a
script hook called:

post_bootstrap.sh

which allows you to do whatever is needed to complete the
system installation phase. Also see:

https://osinside.github.io/kiwi/concept_and_workflow/shell_scripts.html

From my experience so far users followed these steps

1. Run the box in debug mode with --box-debug
2. Come up with a setup such that the proxy based network is reachable.
This is not related to kiwi. All steps needed to access the network
they encode as additional data below the extra boxroot/ tree
and add boxroot/etc/boxprofile code/variables when needed.
If additional software was required that is not part of the box,
we can add anything you need
3. Once the box can connect, the same steps needs to be applied to
a post_bootstrap.sh such that chroot tasks will work

Brings me to my next question:

Q: Where in the above procedure are we ?


Please note I have zero knowledge on your network situation,
thus I assume I'm not of great help with a solution.
A complete build log of a failed attempt would also be helpful

Thanks

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer

[Sven Wölfel]

As always, thanks for your detailed answer.

> Q: Where in the above procedure are we ?

At the moment I'm at point 2. The box can reach our internal network, If I set the environment variables (http_proxy, https_proxy, etc...) wget and other cmd tools use the company proxy and are able to reach the outside world. So far I'm fine. But altering the network manager connection of the boxbuild vm as mentioned above was not successfull yet. I will try to do this step in the post_bootstrap.sh and see if it works. 

As an alternative I try to build the vm on my own machine at home without proxy constraints to see if there is an issue. 

> Please note I have zero knowledge on your network situation,
>thus I assume I'm not of great help with a solution.
>A complete build log of a failed attempt would also be helpful

Yes and again Im more than grateful for your provided support. As stated before I realize proxy setups are a niche case and are highly company specific. I will provide a log of the build with after configuring NM to use a proxy asap.

[Sven Woelfel]

Ok I am not getting it to work with nmcli and at this point I don't know
if it is really possible to debug the NM connection with our proxy.. BUT
the solution was there all along in one of your first replies, I was
just to focused to see it. APT uses its own proxy configuration which
can be configured independently from the system network settings. All I
had to do was creating a apt proxy configuration file and place it in
boxroot/etc/apt/apt/conf.d/ and voila.... apt can talk to the outside
world. So that issue I would call solved.

But during the build apt can not verify the gpg signatures of the
repositories, which is weird since I disabled the gpg check in the
configuration.
Attached is a log of the build and my config.
Since I can not use obs links (since they would not point to our
apt-cache server ), I had to translate them to http links.


I was able to reproduce this behavior on my private machine at home with
the following description:

https://github.com/OSInside/kiwi-descriptions/blob/main/ubuntu/x86_64/ubuntu-noble/config.xml

If i use the config as it is, it works. If I replace

<source path="obs://Virtualization:Appliances:Builder/xUbuntu_24.04"/>

with

<source
path="http://download.opensuse.org/repositories/Virtualization:Appliances:Builder/xUbuntu_24.04"/>

The gpg checks for all the declared repositories in the config fail.

The method of replacing obs links with http links in the config worked
fine for previous kiwi versions, so am I missing something again?



Regards

Sven


P.S.: My first mail seems to not be coming through so this is the second
attempt.

[Marcus Schäfer]

Hi,

> All I had to do
> was creating a apt proxy configuration file and place it in
> boxroot/etc/apt/apt/conf.d/ and voila.... apt can talk to the outside world.
> So that issue I would call solved.

Great, someone should write a short troubleshooting article here:
https://github.com/OSInside/kiwi/tree/main/doc/source/troubleshooting

Would be great I'm sure this will come up again :)

> But during the build apt can not verify the gpg signatures of the
> repositories, which is weird since I disabled the gpg check in the
> configuration.
> Attached is a log of the build and my config.

Yeah that one you can solve relatively easily. The problem is this:

---
[ ERROR ]: 06:36:23 | KiwiDebianBootstrapError: KiwiCommandError: apt-get:
stderr: E: Package 'apt:arm64' has no installation candidate
, stdout: Reading package lists...
---

and you are building for amd64 and for some reason it wants to pull
for arm64. The repos OBS creates for Debian are flat repos of multiple
architectures. I guess that's why apt gets confused. We fixed that
recently by adding an architectures attribute to the repo definition
for debian. So just setup the xUbuntu_24.04 repo as follows:

<repository type="apt-deb" alias="Virtualization" repository_gpgcheck="false" architectures="amd64">

<source path="http://download.opensuse.org/repositories/Virtualization:Appliances:Builder/xUbuntu_24.04"/>

</repository>

This will pin the repo to only one arch and should fix your issue

If you want the gpg checks to pass you also need to fetch the key from the repo
on the OBS side and import it into the box. Ubuntu has done several changes
on that gpg key setup and I don't have the details at hand

However, Neal can help with the details here

[Sven Wölfel]

> Great, someone should write a short troubleshooting article here:
> https://github.com/OSInside/kiwi/tree/main/doc/source/troubleshooting
>
> Would be great I'm sure this will come up again :)

I will do so as soon as I get the build working.

I pinned all the repositories i used to amd64 with architectures="amd64" just like this:

 <repository type="apt-deb" repository_gpgcheck="false" package_gpgcheck="false" imageinclude="true" architectures="amd64">

Nonetheless it is still the same error:

[ DEBUG   ]: 06:45:47 | Looking for apt-get in /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
[ DEBUG   ]: 06:45:47 | EXEC: [apt-get -q -c /result/build/image-root/kiwi_apt.conf -y --no-install-recommends install -oDebug::pkgDPkgPm=1 -oDPkg::Pre-Install-Pkgs::=cat >/var/tmp/kiwi_debs_28qp089g ?essential ?exact-name(usr-is-merged) base-passwd apt]
[ DEBUG   ]: 06:45:48 | EXEC: Failed with stderr: E: Package 'apt:arm64' has no installation candidate

, stdout: Reading package lists...

Building dependency tree...
Package apt:arm64 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Is this error still here because the gpg check fails and he can nor read the repositories or is there something else going on?

If the message is correct there is another package that explicitly wants the arm64 installation candidate for apt.

> If you want the gpg checks to pass you also need to fetch the key from the repo
> on the OBS side and import it into the box. Ubuntu has done several changes
> on that gpg key setup and I don't have the details at hand

To be honest I would just like to ignore them, as to why I used the repository_gpgcheck="false" in all the repository definitions.

Ubuntu not being able to verify the gpg key of a third party repository is a constant problem behind a proxy, since most key server URLs use the hkp:// prefix which is not working with the configured apt proxy entries.

And to my knowledge Apt has no config entry for a hkp proxy or a protocopl specification to contact the keyserver (not to mention a config entry to point to a specific keyserver like a local one). So behind a proxy disabling the checks is usually the only feasible option.

> However, Neal can help with the details here

This would be great.

Regards

Sven

[Marcus Schäfer]

HI,

> I pinned all the repositories i used to amd64 with
> architectures="amd64" just like this:

I believe it's only needed for the one you use from OBS
but it should also not hurt

> <repository type="apt-deb" repository_gpgcheck="false"
> package_gpgcheck="false" imageinclude="true" architectures="amd64">

yes I did the same

> Nonetheless it is still the same error:
>
> [ DEBUG ]: 06:45:47 | Looking for apt-get in
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
> [ DEBUG ]: 06:45:47 | EXEC: [apt-get -q -c
> /result/build/image-root/kiwi_apt.conf -y --no-install-recommends
> install -oDebug::pkgDPkgPm=1 -oDPkg::Pre-Install-Pkgs::=cat
> >/var/tmp/kiwi_debs_28qp089g ?essential ?exact-name(usr-is-merged)
> base-passwd apt]
> [ DEBUG ]: 06:45:48 | EXEC: Failed with stderr: E: Package
> 'apt:arm64' has no installation candidate

How can that be ??? The information from the architectures
attribute is put into the deb source file as documented here:

https://wiki.debian.org/DebianRepository/Format#Architectures

Anyway apt and multiarch is a nightmare. I have changed the structure
of the repos as we provide it on OBS for Ubuntu. See here:

https://build.opensuse.org/project/show/Virtualization:Appliances:Builder

Therefore please change your repo definition for the OBS repo as
follows:

<repository type="apt-deb" repository_gpgcheck="false" package_gpgcheck="false" alias="Virtualization" arch="x86_64">
<source path="https://download.opensuse.org/repositories/Virtualization:/Appliances:/Builder/xUbuntu_24.04_x86_64"/>
</repository>

The repo now only builds for one architecture and we offer it for
x86 and arm on different locations.

I tested an example build for Ubuntu Noble from the kiwi-descriptions
github repo as follows:

kiwi-ng system boxbuild --box ubuntu --box-debug -- --description ~ms/Project/kiwi-descriptions/ubuntu/x86_64/ubuntu-noble/ --target-dir /tmp/mytest-ubuntu

and that completed

Next try, let me know if we are making progress :)

[Sven Wölfel]

> Next try, let me know if we are making progress :)

Progress indeed! The bootstrap phase finishes now, the only error remains the gpg check. But atm. this is not a blocking issue.

After the bootstrap phase apt stops working again, since it has no proxy information. But this should not be an issue when using an apt-cache in the config.xml or fiddling in the apt proxy via the post_bootstrap script.

I will adjust the apt-cache to the new repository and will test if it works over the weekend.

Thanks again for all the help!

[Sven Wölfel]

Ok, I also tested it over the apt-cache without proxy settings during the build and everything works fine.

The only thing that isn't working is the disabling of the gpg check. 

Also due to this bug (https://bugs.launchpad.net/ubuntu/+source/dhcpcd/+bug/2076470) the build breaks since dhcpcd-base is required by cloud-init. But that is not a kiwi related matter.

Without cloud-init the build works. 

So thank you again for all your support, and I will follow through with the small addition to the documentation.

Have a nice weekend.

Regards

Sven

[Marcus Schäfer]

Hi,

> Ok, I also tested it over the apt-cache without proxy settings during
> the build and everything works fine.

Great to hear that, well done.

> The only thing that isn't working is the disabling of the gpg check.

I had hoped Neal or other people on the list who are more into
Debian distros could give a hint how to bring in the key such
that gpg checking of the repos can be enabled. I'll ping Neal on
the kiwi matrix channel

> So thank you again for all your support, and I will follow through with
> the small addition to the documentation.

That would be really good. Thanks in advance

[Sven Woelfel]

Ok I am not getting it to work with nmcli and at this point I don't know if it is really possible to debug the NM connection with our proxy.. BUT the solution was there all along in one of your first replies, I was just to focused to see it. APT uses its own proxy configuration which can be configured independently from the system network settings. All I had to do was creating a apt proxy configuration file and place it in boxroot/etc/apt/apt/conf.d/ and voila.... apt can talk to the outside world. So that issue I would call solved.

But during the build apt can not verify the gpg signatures of the repositories, which is weird since I disabled the gpg check in the configuration.

Attached is a log of the build and my config.
Since I can not use obs links (since they would not point to our apt-cache server ), I had to translate them to http links.

I was able to reproduce this behavior on my private machine at home with the following description:

https://github.com/OSInside/kiwi-descriptions/blob/main/ubuntu/x86_64/ubuntu-noble/config.xml

If i use the config as it is, it works. If I replace

<source path="obs://Virtualization:Appliances:Builder/xUbuntu_24.04"/>

with

<source path="http://download.opensuse.org/repositories/Virtualization:Appliances:Builder/xUbuntu_24.04"/>

The gpg checks for all the declared repositories in the config fail.

The method of replacing obs links with http links in the config worked fine for previous kiwi versions, so am I missing something again?

Regards

Sven

--
You received this message because you are subscribed to a topic in the Google Groups "kiwi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kiwi-images/5u7FBAnUXgA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kiwi-images...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kiwi-images/8784df6c-5672-4c3d-8513-730241063d6cn%40googlegroups.com.