How do I build my appliance with the latest package updates?

[glid...@googlemail.com]

My config.xml file has the lines listed below. My hope was that by including the update repository with a higher priority than the main repository, I'd get the latest package versions, but I've discovered that isn't working. A fully updated system has v13.2.1 of libstdc++6, whereas my appliance ends up with v12.2.1. It hasn't mattered before, but I now need v13.2.1. What am I doing wrong? I'm guessing there must be a trick for doing this.

        <repository type="rpm-md" alias="kiwi" priority="1">
                <source path="obs://Virtualization:Appliances:Builder/openSUSE_Leap_15.5"/>
        </repository>
        <repository type="yast2" alias="Leap_15.5Update" priority="2">
                <source path="obs://openSUSE:Leap:15.5:Update/standard"/>
        </repository>
        <repository type="yast2" alias="Leap_15.5" priority="3">
                <source path="obs://openSUSE:Leap:15.5/standard"/>
        </repository>
        <repository type="rpm-md" alias="perl" priority="4">
                <source path="obs://devel:languages:perl/15.5"/>
        </repository>
        <repository type="rpm-dir" alias="Local_RPMs" priority="5">
                <source path="dir:///home/public/tmp/myrepo_15.5"/>
        </repository>

[glid...@googlemail.com]

I found one way that looks to have worked. Rather than running system build, I ran system prepare and system create separately, so that I could perform steps in-between. I had to remove two packages from the package list in config.xml because they required the later version of libstdc++6. In between the two build steps, I chrooted in to the root folder and performed "zypper --non-interactive --no-gpg-checks refresh" and "zypper --non-interactive --no-gpg-checks update" (unsure whether the former was needed). I also installed the two packages that I'd initially had to leave out. I used kiwi-ng system update to do that, although I guess I could have used zypper.

I wonder if there is a better way. Maybe there's a hook somewhere for running a command just before package installation.

[Marcus Schäfer]

Hi,

> My config.xml file has the lines listed below. My hope was that by
> including the update repository with a higher priority than the main
> repository, I'd get the latest package versions, but I've discovered
> that isn't working. A fully updated system has v13.2.1 of libstdc++6,
> whereas my appliance ends up with v12.2.1. It hasn't mattered before,
> but I now need v13.2.1. What am I doing wrong? I'm guessing there must
> be a trick for doing this.

So for zypper the prio value 1 is the highest. This means your repos
has the prio order

kiwi
Leap_15.5Update
Leap_15.5
perl
Local_RPMs

I guess this is also what you want ? The reason why zypper did not
pull the updated libstdc++6, is most probably a package that has
not build against this version of the lib.

It might be possible that you force the install of the latest libstdc++6
by explicitly requesting this package via:

<package name="libstdc++13.2.1"/>

or the full name as it is provided by the rpm header. However, it's just
a guess as we only ask the package manager and it could be overruled.

Another option is to split the installation process as you already did
it with extra prepare/create. So in kiwi there is the bootstrap phase
followed by the actual install phase. You can "cheat" the installation
in a way that you put libstdc++6 into the bootstrap section:

<packages type="bootstrap">
<package name="libstdc++6"/>
...other bootstrap packages
</packages>

<packages type="image">
<package name="foo"/>
...
</packages>

This will install the latest libstdc++6 already as part of the bootstrap
phase and hopefully accepts all other packages in the image phase

No matter what you do here, it's all just workarounds to a dependency
issue caused by one or more packages of the repos. Thus a real clean
fix lives on the packaging side and not on kiwi :)

Hope this makes sense to you too

Best regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------

[glid...@googlemail.com]

Hi Marcus. Thanks for the reply.

On Saturday, February 17, 2024 at 7:42:55 PM UTC Marcus wrote:

Hi,

> My config.xml file has the lines listed below. My hope was that by
> including the update repository with a higher priority than the main
> repository, I'd get the latest package versions, but I've discovered
> that isn't working. A fully updated system has v13.2.1 of libstdc++6,
> whereas my appliance ends up with v12.2.1. It hasn't mattered before,
> but I now need v13.2.1. What am I doing wrong? I'm guessing there must
> be a trick for doing this.

So for zypper the prio value 1 is the highest. This means your repos
has the prio order

kiwi
Leap_15.5Update
Leap_15.5
perl
Local_RPMs

I guess this is also what you want ?

I'm not sure it's the ideal ordering, but that was my intention. I include perl only to pick up some packages not present in the main repos, and my local repo has packages that definitely wont be in any of the others, so I assume, gets used independent of priority. The placement of kiwi is perhaps the one I'm least sure of.

 

The reason why zypper did not
pull the updated libstdc++6, is most probably a package that has
not build against this version of the lib.

I'm skeptical of that being the cause. There were several indications that it was not.

1. Firstly, libstdc++6 looks to have been set up to be backward compatible for versions going a long way back. If I look at its dependencies (under the heading of "Provides") then there's a long list of libstdc++.so.6(GLIBCXX_a.b.c)(64bit) for many different instances of a.b.c - about 40 of them. I'd imagine it has to be made backward compatible, otherwise all projects would have to update their gcc versions in lock step.

2. If there were a conflict, shouldn't "zypper update" also refuse to install the new version?

3. It looked like it wasn't just the latest libstdc++6 package that I was failing to pick up: I seemed to be picking up nothing from the update repo, until I ran zypper update, whereupon many packages were updated. It was as one might imagine on first running update on a freshly installed system.

 

It might be possible that you force the install of the latest libstdc++6
by explicitly requesting this package via:

<package name="libstdc++13.2.1"/>

or the full name as it is provided by the rpm header. However, it's just
a guess as we only ask the package manager and it could be overruled.

Another option is to split the installation process as you already did
it with extra prepare/create. So in kiwi there is the bootstrap phase
followed by the actual install phase. You can "cheat" the installation
in a way that you put libstdc++6 into the bootstrap section:

<packages type="bootstrap">
<package name="libstdc++6"/>
...other bootstrap packages
</packages>

<packages type="image">
<package name="foo"/>
...
</packages>

This will install the latest libstdc++6 already as part of the bootstrap
phase and hopefully accepts all other packages in the image phase

No matter what you do here, it's all just workarounds to a dependency
issue caused by one or more packages of the repos. Thus a real clean
fix lives on the packaging side and not on kiwi :)

I was seeing it as a kiwi question, just wondering if I was specifying the update repo wrongly... in a way that kiwi cannot make use of it. kiwi doesn't seem to be picking up any packages from it in my case. I wondered if update repos might be entirely different from standard ones and only usable via "zypper update" and not as an initial source of packages.

[Marcus Schäfer]

Hi,

> I'm not sure it's the ideal ordering, but that was my intention. I
> include perl only to pick up some packages not present in the main
> repos, and my local repo has packages that definitely wont be in any of
> the others, so I assume, gets used independent of priority.

ok that makes sense to me too

> The placement of kiwi is perhaps the one I'm least sure of.

The order of the repos in kiwi results in a flat list of
"add-repo" instructions to the desired package manager, zypper in this case.
For the zypper case it doesn't matter in which order the repos were
added. For the later resolver task in zypper other attributes, like
the priority are taken into account.

For the repo list you specified I agree with you, no prio setting
should be needed.

> The reason why zypper did not
> pull the updated libstdc++6, is most probably a package that has
> not build against this version of the lib.
>
> I'm skeptical of that being the cause. There were several indications
> that it was not.
> 1. Firstly, libstdc++6 looks to have been set up to be backward
> compatible for versions going a long way back.

correct

> If I look at its
> dependencies (under the heading of "Provides") then there's a long list
> of libstdc++.so.6(GLIBCXX_a.b.c)(64bit) for many different instances of
> a.b.c - about 40 of them. I'd imagine it has to be made backward
> compatible, otherwise all projects would have to update their gcc
> versions in lock step.

Yes true

> 2. If there were a conflict, shouldn't "zypper update" also refuse to
> install the new version?

In case of a real conflict which requires a decision making, the
process will fail. That's because kiwi calls zypper with the --non-interactive
flag and that causes zypper to fail in case of a required user interaction

> 3. It looked like it wasn't just the latest libstdc++6 package that I
> was failing to pick up: I seemed to be picking up nothing from the
> update repo, until I ran zypper update, whereupon many packages were
> updated. It was as one might imagine on first running update on a
> freshly installed system.

Hmm, that I don't see in any of my builds. When I add the update
repo this provides a number of package updates with higher version, release
or build number and zypper pulls them in on a normal "zypper install ..."
operation, if the overall dependency resolution allows it.

> I was seeing it as a kiwi question, just wondering if I was specifying
> the update repo wrongly... in a way that kiwi cannot make use of it.

kiwi is a dumb player in this game :) Kiwi constructs a request to the
package manager. You can give some parameters to this request, like prio,
fqdn, ignores, etc etc but in the end we just give the package manager
a request and it should not be any different as if a user would use
the package manager on his system, such that also the same issues
should appear no matter if kiwi sends the request or a user.

> kiwi doesn't seem to be picking up any packages from it in my case. I
> wondered if update repos might be entirely different from standard ones
> and only usable via "zypper update" and not as an initial source of
> packages.

I can assure in this regard that an update repo or a custom repo is
not handled differently. For rpm the repo metadata is stored under
repodata/ in the respective repo. If that is an update repo or some
other repo is immaterial for the package manager reading this metadata.
Also your custom repo should have this repodata/ (produced by createrepo).
zypper creates so called solvable files from all this repodata/ information
and that is the data it use for the satsolver based dependecy resolution.

We intentionally never added any special kiwi magic to the solver
process such that building an image with kiwi does the exact same
thing as if a user would install the number of packages from the
given repos.

Debugging your issue further seems not so easy unless you can
provide access to your custom repo.

Cheers,

[Marcus Schäfer]

Hi,

I did some tests on the resovler issue you reported. I think
the problem is caused by an "incomplete" repository setup. No long
ago SUSE connected Leap with SLE. This means for a clean resolver
you also need to add the Leap/SLE repos. In my case this ended up
in the following repo setup:

<repository type="rpm-md" alias="Leap_15_5_sle" imageinclude="true">
<source path="https://download.opensuse.org/update/leap/15.5/sle"/>
</repository>
<repository type="rpm-md" alias="Leap_15_5_backports" imageinclude="true">
<source path="https://download.opensuse.org/update/leap/15.5/backports"/>
</repository>
<repository type="rpm-md" alias="Leap_15_5_update" imageinclude="true">
<source path="https://download.opensuse.org/update/leap/15.5/oss"/>
</repository>
<repository type="rpm-md" alias="Leap_15_5" imageinclude="true">
<source path="https://download.opensuse.org/distribution/leap/15.5/repo/oss"/>
</repository>

If I now resolve my image via:

kiwi-ng image info --description /path/to/description --resolve-package-list

I get the following result:

...

"libstdc++6": {
"arch": "x86_64",
"installsize_bytes": 2362632,
"source": "https://download.opensuse.org/update/leap/15.5/sle",
"status": "added_by_dependency_solver",
"version": "13.2.1+git7813-150000.1.6.1"
},

As you can see the libstdc++6 package update was taken from the update
repo connected to SLE. I think this is the real cause of the issue you
saw when you get the impression that nothing was taken from the standard
leap update repo.

Let us know if this helps on your side too.

Thanks

[glid...@googlemail.com]

Yes, that did it thanks Marcus. That makes a lot of sense too. When I ran "zypper update", I received packages from the SLE repo. I sort of noticed that and then ignored it. Doh! I'm now getting the correct version of libstdc++6 and finding some python311 packages that for which I'd previously had to add extra repos.

For some reason, I had to change all the https's to http, otherwise the build ran into an SSL problem, but that doesn't overly concern me.

Your help with these "not really kiwi" problems is very much appreciated. Thank you again.