[PATCH] isar-bootstrap: Clear local apt state information

[vijaikumar....@gmail.com]

From: Vijai Kumar K <Vijaikumar_...@mentor.com>

Debootstrap's second stage switches to using the default debian mirror.
But the apt state information is renamed and used from the first stage.
This might cause apt-get update to fail with the below error when the
sources list contain both custom and upstream repos.

| Reading package lists...
| E: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease \
| Unable to find expected entry 'contrib/source/Sources' in Release file \
| (Wrong sources.list entry or malformed file)

Clear the apt state information before apt-get update.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index d614d9a..5a379f7 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -320,6 +320,7 @@ isar_bootstrap() {
chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
fi

+ rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*
chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
--
2.17.1

[Henning Schild]

Hi,

i am not sure i fully get the problem. With the first stage you
probably mean "debootstrap" and the second stage the isar bootstrap on
top.

The first stage should be using the primary entry from the
sources.list, so as far as i remember we are just adding more entries.
Meaning the state information should stay valid. Maybe in the offline
"base-apt" case we have a flow that actually switches the upstream
mirror.

Maybe you can go into detail how to reproduce the issue at hand,
possibly add a test-case for it.

If the files should be deleted, my guess is that the deletion is
related to the deletion of sources.list in line 282. So it might be a
good idea to add your deletion right after or before that.

Henning

Am Wed, 20 Nov 2019 16:23:40 +0530
schrieb <vijaikumar....@gmail.com>:

[Vijai Kumar K]

On Thu, Nov 21, 2019 at 10:22:18AM +0100, Henning Schild wrote:
> Hi,
>
> i am not sure i fully get the problem. With the first stage you
> probably mean "debootstrap" and the second stage the isar bootstrap on
> top.

In case of crossdebootstrap, the bootstrapping runs in two stages.
Basically qemu-debootstrap takes care of that and calls debootstrap
--second-stage when it identifies the build as crossbuild.

The apt url passed in is not used in the second stage. Instead, it uses
deb.debian.org/debian and reuses the state information like below(log
from debootstrap --second-stage)

+ mv debootstrap.invalid_dists_stretch_InRelease deb.debian.org_debian_dists_stretch_InRelease
+ mv debootstrap.invalid_dists_stretch_Release deb.debian.org_debian_dists_stretch_Release
+ mv debootstrap.invalid_dists_stretch_Release.gpg deb.debian.org_debian_dists_stretch_Release.gpg

The debootstrap.invalid_* has the InRelease contents of my custom repo(a
minimal debian repo) passed in as a argument to qemu-debootstrap. Post
second stage the apt-state information is messed up.
Has the name of deb.debian.org but contents of my local apt mirror. So,
if apt-get update is called with this state, it would
fail when I have both deb.debian.org and the custom repo mentioned in my
DEBIAN_APT_SOURCES file.

>
> The first stage should be using the primary entry from the
> sources.list, so as far as i remember we are just adding more entries.
> Meaning the state information should stay valid. Maybe in the offline
> "base-apt" case we have a flow that actually switches the upstream
> mirror.
>
> Maybe you can go into detail how to reproduce the issue at hand,
> possibly add a test-case for it.

It is one such corner case. Adding a test case for such a unique
situation is difficult since it involves custom repos.

>
> If the files should be deleted, my guess is that the deletion is
> related to the deletion of sources.list in line 282. So it might be a
> good idea to add your deletion right after or before that.

Yes. I can move it over there.


Thanks,
Vijai Kumar K

[Jan Kiszka]

On 20.11.19 11:53, vijaikumar....@gmail.com wrote:
> From: Vijai Kumar K <Vijaikumar_...@mentor.com>
>
> Debootstrap's second stage switches to using the default debian mirror.
> But the apt state information is renamed and used from the first stage.
> This might cause apt-get update to fail with the below error when the
> sources list contain both custom and upstream repos.
>
> | Reading package lists...
> | E: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease \
> | Unable to find expected entry 'contrib/source/Sources' in Release file \
> | (Wrong sources.list entry or malformed file)
>
> Clear the apt state information before apt-get update.

I had problems the other day with bootstrapping behind our proxy when
there was both a local and the upstream repo involved:

Reading package lists...



E: Repository 'http://cdn-fastly.deb.debian.org/debian buster InRelease'
changed its 'Origin' value from '' to 'Debian'


E: Repository 'http://cdn-fastly.deb.debian.org/debian buster InRelease'
changed its 'Label' value from '' to 'Debian'


E: Repository 'http://cdn-fastly.deb.debian.org/debian buster InRelease'
changed its 'Suite' value from '' to 'stable'



Would that be resolved by your patch as well? Can reproduce ATM, that's
why I'm asking.

Jan

>
> Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
> ---
> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index d614d9a..5a379f7 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -320,6 +320,7 @@ isar_bootstrap() {
> chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
> fi
>
> + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*
> chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
> chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
> chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
>


--

Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[vijaikumar....@gmail.com]

From: Vijai Kumar K <Vijaikumar_...@mentor.com>

Debootstrap's second stage switches to using the default debian mirror.
But the apt state information is renamed and used from the first stage.
This might cause apt-get update to fail with the below error when the
sources list contain both custom and upstream repos.

| Reading package lists...
| E: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease \
| Unable to find expected entry 'contrib/source/Sources' in Release file \
| (Wrong sources.list entry or malformed file)

Clear the apt state information before apt-get update.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

Changes in v2:
- Addressed review comments from Henning.

meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc

index d614d9a..9096775 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -280,6 +280,7 @@ isar_bootstrap() {
install -v -m644 "${APTSRCS_INIT}" \
"${ROOTFSDIR}/etc/apt/sources-list"
rm -f "${ROOTFSDIR}/etc/apt/sources.list"

+ rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*

mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
install -v -m644 "${WORKDIR}/isar-apt.conf" \
"${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
--
2.17.1

[Vijai Kumar K]

On Thu, Nov 21, 2019 at 11:01:31AM +0100, Jan Kiszka wrote:
> On 20.11.19 11:53, vijaikumar....@gmail.com wrote:
> > From: Vijai Kumar K <Vijaikumar_...@mentor.com>
> >
> > Debootstrap's second stage switches to using the default debian mirror.
> > But the apt state information is renamed and used from the first stage.
> > This might cause apt-get update to fail with the below error when the
> > sources list contain both custom and upstream repos.
> >
> > | Reading package lists...
> > | E: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease \
> > | Unable to find expected entry 'contrib/source/Sources' in Release file \
> > | (Wrong sources.list entry or malformed file)
> >
> > Clear the apt state information before apt-get update.
>
> I had problems the other day with bootstrapping behind our proxy when there
> was both a local and the upstream repo involved:
>
> Reading package lists...
>
>
>
> E: Repository 'http://cdn-fastly.deb.debian.org/debian buster InRelease'
> changed its 'Origin' value from '' to 'Debian'
>
>
> E: Repository 'http://cdn-fastly.deb.debian.org/debian buster InRelease'
> changed its 'Label' value from '' to 'Debian'
>
>
> E: Repository 'http://cdn-fastly.deb.debian.org/debian buster InRelease'
> changed its 'Suite' value from '' to 'stable'
>
>
>
> Would that be resolved by your patch as well? Can reproduce ATM, that's why
> I'm asking.

Possibily. But, I am not sure. You could give it a try.

Thanks,
Vijai Kumar K

>
> Jan
>
> >
> > Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
> > ---
> > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
> > 1 file changed, 1 insertion(+)
> >
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > index d614d9a..5a379f7 100644
> > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > @@ -320,6 +320,7 @@ isar_bootstrap() {
> > chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
> > fi
> > + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*
> > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
> > chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
> > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
> >
>
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux
>

> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/ed6a0a83-7189-d260-2df1-e7b0378ccd76%40siemens.com.

[Henning Schild]

Am Thu, 21 Nov 2019 15:24:01 +0530
schrieb Vijai Kumar K <vijaikumar....@gmail.com>:

I still do not fully get it.

For the test coverage of custom repos in general, and signed ones on
top, we need to find a solution anyways.

I would prefer trying to treat the real ones as "custom". i.e. by
removing the default gpg trust from debootstrap and adding it as custom
trust. For this one this trick would not work.

The second obvious way, which requires some more coding is to use Isar
itself to produce the custom repo. That is most likely the source of
your custom one anyways, it is the source of the one Jan had his custom
gpg keys for.

Henning

[Vijai Kumar K]

Ah, reading back my comment looks like I am totally against test case
for custom repo. I sounded wrong on that. What I was trying to say is we
would need to remove /var/lib/apt/lists/* since its a residue left over
by qemu-debootstrap. Not removing it has caused me an issue due to
the way how debootstrap tool handled the state information. Yes, we
should definitely do a test case for custom repo. But I dont think this
patch has any effect on that. It is just a proper cleanup before adding
our sources list.

[Jan Kiszka]

Normally, you would put the wildcard inside the quotes.

Jan

> mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
> install -v -m644 "${WORKDIR}/isar-apt.conf" \
> "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
>

--

[Vijai Kumar K]

Unfortunately file globbing doesnot work under quotes. Another alternative is to use quotes only around
the variable as "${ROOTFSDIR}"/var/lib/apt/lists/*

Thanks,
Vijai Kumar K

[Jan Kiszka]

Ah, yes, that was nonsense.

Jan

[Baurzhan Ismagulov]

On Thu, Nov 21, 2019 at 03:35:32PM +0530, vijaikumar....@gmail.com wrote:
> Debootstrap's second stage switches to using the default debian mirror.
> But the apt state information is renamed and used from the first stage.
> This might cause apt-get update to fail with the below error when the
> sources list contain both custom and upstream repos.
>
> | Reading package lists...
> | E: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease \
> | Unable to find expected entry 'contrib/source/Sources' in Release file \
> | (Wrong sources.list entry or malformed file)
>
> Clear the apt state information before apt-get update.

Applied to next, thanks.

With kind regards,
Baurzhan.