Repo cache issue

[Roger.Ob...@gmail.com]

Repository cache in OEM build

Quite often I get the following error when building an image:

Warning: Disabling repository
'http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3' because of
the above error.
Building repository
'http:__download.opensuse.org_repositories_devel:_lang[done]
Error building the cache:
[|] Failed to cache repo (3).

It happens for all caches. I know I reported ths in July. And at that time it
seemed to work to delete /var/cache/kiwi/zypper. This no longer seems to help.

I am running oS 12.3 and kiwi-5.05.35-638.1.i586 and zypper-1.8.16-1.10.1.i586



--
Yours sincerely,

Roger Oberholtzer

Ramböll RST / Systems

Office: Int +46 10-615 60 20
Mobile: Int +46 70-815 1696
roger.ob...@ramboll.se
________________________________________

Ramböll Sverige AB
Krukmakargatan 21
P.O. Box 17009
SE-104 62 Stockholm, Sweden
www.rambollrst.se

[Marcus Schäfer]

Hi,

> Quite often I get the following error when building an image:
>
> Warning: Disabling repository
> 'http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3' because of
> the above error.

So the error for this one is not listed, but I guess it's not of interest

> Building repository
> 'http:__download.opensuse.org_repositories_devel:_lang[done]
> Error building the cache:
> [|] Failed to cache repo (3).
>
> It happens for all caches. I know I reported ths in July. And at that time it
> seemed to work to delete /var/cache/kiwi/zypper. This no longer seems to help.
>
> I am running oS 12.3 and kiwi-5.05.35-638.1.i586 and zypper-1.8.16-1.10.1.i586

I could only imagine an incompatibility between different zypper versions
and the cache handling. kiwi shares the cache from bootstrap and the
image phase. This means:

- the bootstrap is done with the host system zypper.
So what distro and zypper version is used on the machine you call kiwi on ?

- the rest of the image is installed by a chroot to the bootstraped root
This means in your case the 12.3 zypper version is used to finish the job.
the 12.3 zypper now uses the cache and all other data initially created
by the host system zypper

If the versions differ much the problems you mentioned could happen.
Problem here is imho not kiwi but zypper and its backward compatibility.
From the kiwi perspective we can only remove the code which shares
repo and cache metadata but that will be a step back with regards to
performance

What you can do is:

a) install the new root using only the host system zypper. This can
be done by putting all packages and patterns into the
<packages type="bootstrap"> section. I don't recommend this because
it's not guaranteed that an old zypper versions deals correctly
with non core packages of a new distribution

b) update your host system zypper

c) build your image in a container (e.g lxc) which is of the same
version as the target image. In your case a 12.3 container.

d) build your image in a VM (e.g KVM,VMware) which is of the same
version as the target image. In your case a 12.3 container.

With the studio project we found out that the most clean way to
build is to run kiwi in a containment system which is of the same
version as the target

Hope that helps a bit

Regards,
Marcus
--
Public Key available
gpg --keyserver pgp.mit.edu --recv-keys 0xCCE3C6A2
-------------------------------------------------------
Marcus Schäfer (Res. & Dev.) SUSE LINUX Products GmbH
Tel: 0911-740 53 0 Maxfeldstrasse 5
FAX: 0911-740 53 479 D-90409 Nürnberg
GF: Jeff Hawn,Jennifer Guild, Felix Imendörffer
HRB: 21284 (AG Nürnberg) Germany
http://www.suse.de
-------------------------------------------------------

[Roger.Ob...@gmail.com]

On Monday, October 07, 2013 09:38:04 AM Marcus Schäfer wrote:
> Hi,
>
> > Quite often I get the following error when building an image:
> >
> > Warning: Disabling repository
> > 'http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3'
> > because of the above error.
>
> So the error for this one is not listed, but I guess it's not of interest

There is no error other than what is printed. The earlier command for that
repo reported this:

Oct-07 08:28:32 <1> : Adding bootstrap zypper service:
http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3
Oct-07 08:28:32 <1> : EXEC [/usr/bin/zypper --non-interactive --no-gpg-checks
--pkg-cache-dir /var/cache/kiwi/packages --reposd-dir
/home/roger/source/RSofT_Diskless/kiwi/RSofT-preload-
openSUSE-12.3/image/build/image-root//var/cache/kiwi/zypper/repos --solv-
cache-dir /home/roger/source/RSofT_Diskless/kiwi/RSofT-preload-
openSUSE-12.3/image/build/image-root//var/cache/kiwi/zypper/solv --cache-dir
/home/roger/source/RSofT_Diskless/kiwi/RSofT-preload-
openSUSE-12.3/image/build/image-root//var/cache/kiwi/zypper --config
/var/cache/kiwi/zypper/zypper.conf.18351 --root
"/home/roger/source/RSofT_Diskless/kiwi/RSofT-preload-
openSUSE-12.3/image/build/image-root" addrepo -f --type YUM --keep-packages
'http://download.opensuse.org/repositories/X11:/XOrg/openSUSE_12.3'
http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3 2>&1]
Oct-07 08:28:32 <1> : --> Set priority to: 1
Oct-07 08:28:32 <1> : EXEC [/usr/bin/zypper --non-interactive --no-gpg-checks
--pkg-cache-dir /var/cache/kiwi/packages --reposd-dir
/home/roger/source/RSofT_Diskless/kiwi/RSofT-preload-
openSUSE-12.3/image/build/image-root//var/cache/kiwi/zypper/repos --solv-
cache-dir /home/roger/source/RSofT_Diskless/kiwi/RSofT-preload-
openSUSE-12.3/image/build/image-root//var/cache/kiwi/zypper/solv --cache-dir
/home/roger/source/RSofT_Diskless/kiwi/RSofT-preload-
openSUSE-12.3/image/build/image-root//var/cache/kiwi/zypper --config
/var/cache/kiwi/zypper/zypper.conf.18351 --root
"/home/roger/source/RSofT_Diskless/kiwi/RSofT-preload-
openSUSE-12.3/image/build/image-root" modifyrepo -p 1
http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3 2>&1]

A little later I see this:

Oct-07 08:28:40 <1> : EXEC [/dev/shm/lwp-download
http://download.opensuse.org/repositories/X11:/XOrg/openSUSE_12.3/license.tar.gz
/home/roger/source/RSofT_Diskless/kiwi/RSofT-preload-
openSUSE-12.3/image/build/image-root//license.tar.gz 2>&1]
Oct-07 08:28:40 <1> : EXEC [bash -c "PATH=$PATH:/sbin:/usr/sbin type -p lwp-
download" 2>&1]
Oct-07 08:28:40 <1> : EXEC [chmod u+x /dev/shm/lwp-download 2>&1]

And then:

Automatically trusting the following key:
Key ID: BB1AF2330F2672C8
Key Name: X11 OBS Project <X...@build.opensuse.org>
Key Fingerprint: B1DA68DE51949E7B034237C4BB1AF2330F2672C8
Key Created: Wed 16 Jan 2013 02:50:57 PM CET
Key Expires: Fri 27 Mar 2015 02:50:57 PM CET
Repository: http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3

Then, a bit later, we get the error I reported:

Repository 'http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3'
not cached. Caching...
Building repository
'http:__download.opensuse.org_repositories_X11:_XOrg_o[done]

Error building the cache:
[|] Failed to cache repo (3).

Problem loading data from
'http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3'
Resolvables from
'http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3' not loaded
because of error.

>
> > Building repository
> > 'http:__download.opensuse.org_repositories_devel:_lang[done]
> > Error building the cache:
> > [|] Failed to cache repo (3).
> >
> > It happens for all caches. I know I reported ths in July. And at that time
> > it seemed to work to delete /var/cache/kiwi/zypper. This no longer seems
> > to help.
> >
> > I am running oS 12.3 and kiwi-5.05.35-638.1.i586 and
> > zypper-1.8.16-1.10.1.i586
> I could only imagine an incompatibility between different zypper versions
> and the cache handling. kiwi shares the cache from bootstrap and the
> image phase. This means:
>
> - the bootstrap is done with the host system zypper.
> So what distro and zypper version is used on the machine you call kiwi
> on ?

It is what I reported: oS 12.3 and kiwi-5.05.35-638.1.i586 and
zypper-1.8.16-1.10.1.i586

> - the rest of the image is installed by a chroot to the bootstraped root
> This means in your case the 12.3 zypper version is used to finish the
> job. the 12.3 zypper now uses the cache and all other data initially
> created by the host system zypper
>
> If the versions differ much the problems you mentioned could happen.
> Problem here is imho not kiwi but zypper and its backward compatibility.
> From the kiwi perspective we can only remove the code which shares
> repo and cache metadata but that will be a step back with regards to
> performance
>
> What you can do is:
>
> a) install the new root using only the host system zypper. This can
> be done by putting all packages and patterns into the
> <packages type="bootstrap"> section. I don't recommend this because
> it's not guaranteed that an old zypper versions deals correctly
> with non core packages of a new distribution
>
> b) update your host system zypper

That is how it is.

If anthing, I would imagine that the problem is that my host's zypper is
updated, and, when the problem occurs, the build is using the one from the
original oS 12.3 install.

> c) build your image in a container (e.g lxc) which is of the same
> version as the target image. In your case a 12.3 container.
>
> d) build your image in a VM (e.g KVM,VMware) which is of the same
> version as the target image. In your case a 12.3 container.
>
> With the studio project we found out that the most clean way to
> build is to run kiwi in a containment system which is of the same
> version as the target

Further down in the log (after the errors), I see:

Retrieving package zypper-1.8.11-1.1.1.i586

In my image, I include the update repos (which the host system also uses).
Perhpas the use of those in the kiwi build is after the error.

What confuses me is that the host's zypper does not change, but this error
comes and goes. I would imagine that a zypper version mismatch would result
in a consistent error (as long as the versions of the mismatched zypper do not
change).

[Marcus Schäfer]

Hi,

> Automatically trusting the following key:
> Key ID: BB1AF2330F2672C8
> Key Name: X11 OBS Project <X...@build.opensuse.org>
> Key Fingerprint: B1DA68DE51949E7B034237C4BB1AF2330F2672C8
> Key Created: Wed 16 Jan 2013 02:50:57 PM CET
> Key Expires: Fri 27 Mar 2015 02:50:57 PM CET
> Repository: http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3
>
> Then, a bit later, we get the error I reported:
>
> Repository 'http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3'
> not cached. Caching...
> Building repository
> 'http:__download.opensuse.org_repositories_X11:_XOrg_o[done]
> Error building the cache:

At that point the error message is normally telling you more why it can't
build the cache right after the colon. That's empty, pretty strange

> [|] Failed to cache repo (3).
> Problem loading data from
> 'http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3'
> Resolvables from
> 'http:__download.opensuse.org_repositories_X11:_XOrg_openSUSE_12.3' not loaded
> because of error.

trouble with the network connection, any proxy or something else
network related couldn't be the culprit ?

Does it take long (several seconds, minutes) when the error happens ?
still want to understand if we are running in any network timeout here

According to the log all this happens very early, correct ? I mean
in the bootstrap phase at the very beginning of the kiwi prepare
phase (not inside the chroot). if so you can run the following command
in an extra terminal:

tail -f /var/log/zypper.log

and now the kiwi prepare again and see if you get more information.

> > - the bootstrap is done with the host system zypper.
> > So what distro and zypper version is used on the machine you call kiwi
> > on ?
>
> It is what I reported: oS 12.3 and kiwi-5.05.35-638.1.i586 and
> zypper-1.8.16-1.10.1.i586

ok, so 12.3 host and you want a 12.3 image, that's really clean
and you can forget about all what I have said concerning zypper and
version incompatibility

> In my image, I include the update repos (which the host system also uses).
> Perhpas the use of those in the kiwi build is after the error.

I don't think so because I have the same setup here and don't see
that problems

> What confuses me is that the host's zypper does not change, but this error
> comes and goes. I would imagine that a zypper version mismatch would result
> in a consistent error (as long as the versions of the mismatched zypper do not

So you are saying this happens only sometimes ?

[Roger.Ob...@gmail.com]

On Monday, October 07, 2013 11:42:21 AM Marcus Schäfer wrote:

> trouble with the network connection, any proxy or something else
> network related couldn't be the culprit ?

The network connection is very stable. No proxies. Sometimes I suspect that
the repo is in the process of being updated. But one would think that only
that repo would complain. Not all of them.

> Does it take long (several seconds, minutes) when the error happens ?
> still want to understand if we are running in any network timeout here
>
> According to the log all this happens very early, correct ? I mean
> in the bootstrap phase at the very beginning of the kiwi prepare
> phase (not inside the chroot). if so you can run the following command
> in an extra terminal:
>
> tail -f /var/log/zypper.log

Of course, I run it now and there is no problem. Sigh. I realize that it
sounds like an intermittent network problem. But I have never seen that when
running zypper for other purposes.