[PATCH] dpkg-base: apt-get "update" before "source"

[Henning Schild]

From: Henning Schild <henning...@siemens.com>

When rebuilding we can run into an inconsistent view where buildchroot
assumes it could download the sources of a modified upstream package.
After a "reprepro ... remove" we always need to "apt-get update" to not
operate on an old version of the metadata.

Signed-off-by: Henning Schild <henning...@siemens.com>
---
meta/classes/dpkg-base.bbclass | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 175dc80..5425df8 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -31,6 +31,10 @@ do_apt_fetch() {
fi
dpkg_do_mounts
E="${@ bb.utils.export_proxies(d)}"
+ sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
+ -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
+ -o Dir::Etc::sourceparts="-" \
+ -o APT::Get::List-Cleanup="0"
sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
sh -c 'cd ${PP} && apt-get -y source ${SRC_APT}'
dpkg_undo_mounts
--
2.19.2

[Henning Schild]

Here i see an often repeating pattern. That "apt-get update" is now
present in many entry points to the buildchroot.
I guess we should factor it out and put it into a central place. And
the rule of thumb probably is ... whenever you use anything apt,
apt-get update before you do ...

Henning

Am Mon, 11 Feb 2019 10:33:24 +0100
schrieb Henning Schild <henning...@siemens.com>:

[chomb...@gmail.com]

I have a local patch doing just that, I can post something as RFC

[Jan Kiszka]

On 11.02.19 10:35, [ext] Henning Schild wrote:
> Here i see an often repeating pattern. That "apt-get update" is now
> present in many entry points to the buildchroot.
> I guess we should factor it out and put it into a central place. And
> the rule of thumb probably is ... whenever you use anything apt,
> apt-get update before you do ...

If you are only talking about updating our locally maintained repo (like below),
that is fine to factor out and reuse. However, we must not update against public
repos after the initial pulling, in order to ensure we have a consistent package
set along the whole build.

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[chomb...@gmail.com]

Hi Jan,

Will need to recheck things here but there should be no .list files referencing public repos when building from the local apt cache (and therefore when you are wanting a reproducible build)

My local patch essentially does a full apt-get update but I now need to check if I did not alter the expected behavior :)

Cedric

[Henning Schild]

Am Mon, 11 Feb 2019 13:53:07 +0100
schrieb Jan Kiszka <jan.k...@siemens.com>:

> On 11.02.19 10:35, [ext] Henning Schild wrote:
> > Here i see an often repeating pattern. That "apt-get update" is now
> > present in many entry points to the buildchroot.
> > I guess we should factor it out and put it into a central place. And
> > the rule of thumb probably is ... whenever you use anything apt,
> > apt-get update before you do ...
>
> If you are only talking about updating our locally maintained repo
> (like below), that is fine to factor out and reuse. However, we must
> not update against public repos after the initial pulling, in order
> to ensure we have a consistent package set along the whole build.

Sure. In fact all the ones we reprepro with between partial builds. At
the moment just the one, but i guess there is more to come.

In which case it might be a good idea to switch to using
Dir::Etc::SourceParts with a value of i.e.
"sources.list.isar.d", or have all the repos with that feature on
one .list file.

Henning

[Maxim Yu. Osipov]

Hi Cedric,

Any updates on this topic?

Thanks,
Maxim.

> > schrieb Henning Schild <henning...@siemens.com <javascript:>>:

> >
> >> From: Henning Schild <henning...@siemens.com <javascript:>>
> >>
> >> When rebuilding we can run into an inconsistent view where
> buildchroot
> >> assumes it could download the sources of a modified upstream
> package.
> >> After a "reprepro ... remove" we always need to "apt-get update" to
> >> not operate on an old version of the metadata.
> >>
> >> Signed-off-by: Henning Schild <henning...@siemens.com

> <javascript:>>

> >> ---
> >>   meta/classes/dpkg-base.bbclass | 4 ++++
> >>   1 file changed, 4 insertions(+)
> >>
> >> diff --git a/meta/classes/dpkg-base.bbclass
> >> b/meta/classes/dpkg-base.bbclass index 175dc80..5425df8 100644
> >> --- a/meta/classes/dpkg-base.bbclass
> >> +++ b/meta/classes/dpkg-base.bbclass
> >> @@ -31,6 +31,10 @@ do_apt_fetch() {
> >>           fi
> >>           dpkg_do_mounts
> >>           E="${@ bb.utils.export_proxies(d)}"
> >> +        sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get
> update \
> >> +                -o
> >> Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> >> +                -o Dir::Etc::sourceparts="-" \
> >> +                -o APT::Get::List-Cleanup="0"
> >>           sudo -E chroot --userspec=$( id -u ):$( id -g )
> >> ${BUILDCHROOT_DIR} \ sh -c 'cd ${PP} && apt-get -y source
> ${SRC_APT}'
> >>           dpkg_undo_mounts
> >
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux
>

> --
> You received this message because you are subscribed to the Google
> Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to isar-users+...@googlegroups.com
> <mailto:isar-users+...@googlegroups.com>.
> To post to this group, send email to isar-...@googlegroups.com
> <mailto:isar-...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/b9ca5913-f1be-441d-9e2c-3304bc6c5868%40googlegroups.com
> <https://groups.google.com/d/msgid/isar-users/b9ca5913-f1be-441d-9e2c-3304bc6c5868%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.


--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mos...@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

[Maxim Yu. Osipov]

On 2/11/19 10:33 AM, Henning Schild wrote:
> From: Henning Schild <henning...@siemens.com>
>
> When rebuilding we can run into an inconsistent view where buildchroot
> assumes it could download the sources of a modified upstream package.
> After a "reprepro ... remove" we always need to "apt-get update" to not
> operate on an old version of the metadata.

Applied to the 'next'.

Thanks,
Maxim.

> Signed-off-by: Henning Schild <henning...@siemens.com>
> ---
> meta/classes/dpkg-base.bbclass | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 175dc80..5425df8 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -31,6 +31,10 @@ do_apt_fetch() {
> fi
> dpkg_do_mounts
> E="${@ bb.utils.export_proxies(d)}"
> + sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
> + -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> + -o Dir::Etc::sourceparts="-" \
> + -o APT::Get::List-Cleanup="0"
> sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> sh -c 'cd ${PP} && apt-get -y source ${SRC_APT}'
> dpkg_undo_mounts
>


--

[Henning Schild]

Am Mon, 11 Feb 2019 13:53:07 +0100
schrieb Jan Kiszka <jan.k...@siemens.com>:

> On 11.02.19 10:35, [ext] Henning Schild wrote:
> > Here i see an often repeating pattern. That "apt-get update" is now
> > present in many entry points to the buildchroot.
> > I guess we should factor it out and put it into a central place. And
> > the rule of thumb probably is ... whenever you use anything apt,
> > apt-get update before you do ...
>
> If you are only talking about updating our locally maintained repo
> (like below), that is fine to factor out and reuse. However, we must
> not update against public repos after the initial pulling, in order
> to ensure we have a consistent package set along the whole build.

I think that is actually not true. The whole process is an "eventual
consistency" topic. Say one of your package compiles for 2 weeks ...
you would never be able to build a buster image. And in fact, no matter
how long the time windows is. You can always hit the point where a file
that should be there according to your view of the repo, can not be
downloaded anymore .... in which case you need to "update/upgrade".

So i tend to believe that we should always "apt-get update" without
restrictions. For people that truly care about having a reproducible
build with only one consistent state, they should create images from
there partial debian mirror / cache. (where an "apt-get update" is a
noop)

Henning

[Jan Kiszka]

On 26.02.19 16:00, Henning Schild wrote:
> Am Mon, 11 Feb 2019 13:53:07 +0100
> schrieb Jan Kiszka <jan.k...@siemens.com>:
>
>> On 11.02.19 10:35, [ext] Henning Schild wrote:
>>> Here i see an often repeating pattern. That "apt-get update" is now
>>> present in many entry points to the buildchroot.
>>> I guess we should factor it out and put it into a central place. And
>>> the rule of thumb probably is ... whenever you use anything apt,
>>> apt-get update before you do ...
>>
>> If you are only talking about updating our locally maintained repo
>> (like below), that is fine to factor out and reuse. However, we must
>> not update against public repos after the initial pulling, in order
>> to ensure we have a consistent package set along the whole build.
>
> I think that is actually not true. The whole process is an "eventual
> consistency" topic. Say one of your package compiles for 2 weeks ...

Then you will not be able to work. This is no realistic scenario.

> you would never be able to build a buster image. And in fact, no matter
> how long the time windows is. You can always hit the point where a file
> that should be there according to your view of the repo, can not be
> downloaded anymore .... in which case you need to "update/upgrade".
>
> So i tend to believe that we should always "apt-get update" without
> restrictions. For people that truly care about having a reproducible

Nope, we won't do that - unless we have a stable (versioned and frozen) repo.
This does not apply to upstream Debian, so the current pattern is the way to go:
single update for a single build.

Jan

[Henning Schild]

Am Tue, 26 Feb 2019 16:03:42 +0100

schrieb Jan Kiszka <jan.k...@siemens.com>:

> On 26.02.19 16:00, Henning Schild wrote:
> > Am Mon, 11 Feb 2019 13:53:07 +0100
> > schrieb Jan Kiszka <jan.k...@siemens.com>:
> >
> >> On 11.02.19 10:35, [ext] Henning Schild wrote:
> >>> Here i see an often repeating pattern. That "apt-get update" is
> >>> now present in many entry points to the buildchroot.
> >>> I guess we should factor it out and put it into a central place.
> >>> And the rule of thumb probably is ... whenever you use anything
> >>> apt, apt-get update before you do ...
> >>
> >> If you are only talking about updating our locally maintained repo
> >> (like below), that is fine to factor out and reuse. However, we
> >> must not update against public repos after the initial pulling, in
> >> order to ensure we have a consistent package set along the whole
> >> build.
> >
> > I think that is actually not true. The whole process is an "eventual
> > consistency" topic. Say one of your package compiles for 2
> > weeks ...
>
> Then you will not be able to work. This is no realistic scenario.

Sure, just trying to stress the point that your local view of your
mirror can expire any time. ( 2 weeks or 2 minutes ... )

> > you would never be able to build a buster image. And in fact, no
> > matter how long the time windows is. You can always hit the point
> > where a file that should be there according to your view of the
> > repo, can not be downloaded anymore .... in which case you need to
> > "update/upgrade".
> >
> > So i tend to believe that we should always "apt-get update" without
> > restrictions. For people that truly care about having a
> > reproducible
>
> Nope, we won't do that - unless we have a stable (versioned and
> frozen) repo. This does not apply to upstream Debian, so the current
> pattern is the way to go: single update for a single build.

Several updates would most likely be a noop anyways but ensure that
every "apt-get install" will succeed. We do have the cache and everyone
can have a versioned and frozen repo.

I just started this topic again because Andreas introduced a full
"apt-get update" to deal with the gpg topic.

Henning

> Jan
>