[RFC][PATCH] rootfs: Retry downloads 3 times

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Avoids failing a complete rootfs installation in case of a short
hick-up.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

Does something like this make sense? Do we have more places?
We have a retry of 2 on wget bitbake fetches, e.g. (via FETCHCMD_wget).

meta/classes/rootfs.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 806e824c..afec1cbc 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -123,7 +123,7 @@ rootfs_install_pkgs_download[weight] = "600"
rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
rootfs_install_pkgs_download() {
sudo -E chroot '${ROOTFSDIR}' \
- /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
+ /usr/bin/apt-get ${ROOTFS_APT_ARGS} -o Acquire::Retries=3 --download-only ${ROOTFS_PACKAGES}
}

ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""
--
2.26.1

[Henning Schild]

The idea is good but the patch is incomplete.

To find all places where we download you might want to look at bitbake
fetchers in general - do they have a retry count?

Is there a variable controlling that?

After that look at all debian downloaders. You will find them when
looking for "download-only", "apt-get update" any maybe more. They
should use the central bitbake count or switch if available.

Your patch found one of many "download-only" and ignores bitbake. Might
be the most likely ... but we should try to be consistent.

Henning

Am Thu, 7 May 2020 16:36:18 +0200
schrieb Jan Kiszka <jan.k...@siemens.com>:

[Jan Kiszka]

On 07.05.20 20:13, Henning Schild wrote:
> The idea is good but the patch is incomplete.
>
> To find all places where we download you might want to look at bitbake
> fetchers in general - do they have a retry count?
>
> Is there a variable controlling that?

As explained in the commit log, it's not consistent in bitbake. At least
wget has such a retry counter. In other cases, bitbake seems to quickly
fall back to mirrors and fail.

>
> After that look at all debian downloaders. You will find them when
> looking for "download-only", "apt-get update" any maybe more. They
> should use the central bitbake count or switch if available.
>
> Your patch found one of many "download-only" and ignores bitbake. Might
> be the most likely ... but we should try to be consistent.

Yes, we likely also need this when fetching build deps. All images are
covered this way already, I also checked debootstrap but found no retry
switch there. So the result will remain "best effort" in general.
Therefore, I was also wondering if it's worth it.

Background was may lengthy attempt yesterday to install a complex image
from snapshot.debian.org. The "solution" (that avoided a first complete
local mirror) was to throttle the download bandwidth to some 1 MBit/s.
Maybe there was just one bad mirror on the other side, but it constantly
killed the image installed. Retrying was a futile attempt there, but
I've also seen (few) nightly builds in the past that stumbled over
individual packet downloads where this might have helped.

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Henning Schild]

Am Fri, 8 May 2020 07:57:34 +0200
schrieb Jan Kiszka <jan.k...@siemens.com>:

> On 07.05.20 20:13, Henning Schild wrote:
> > The idea is good but the patch is incomplete.
> >
> > To find all places where we download you might want to look at
> > bitbake fetchers in general - do they have a retry count?
> >
> > Is there a variable controlling that?
>
> As explained in the commit log, it's not consistent in bitbake. At
> least wget has such a retry counter. In other cases, bitbake seems to
> quickly fall back to mirrors and fail.

Ok now i know how to read that.

> >
> > After that look at all debian downloaders. You will find them when
> > looking for "download-only", "apt-get update" any maybe more. They
> > should use the central bitbake count or switch if available.
> >
> > Your patch found one of many "download-only" and ignores bitbake.
> > Might be the most likely ... but we should try to be consistent.
>
> Yes, we likely also need this when fetching build deps. All images
> are covered this way already, I also checked debootstrap but found no
> retry switch there. So the result will remain "best effort" in
> general. Therefore, I was also wondering if it's worth it.

I do not think so. We do have the cache now, maybe we can
deb_dl_dir_export after failed "download-only". That way you can at
least retry on the outside. So interactive incremental builds should
work on a shaky connection.

In a later step we could introduce the arg for apt and write a loop
around debian fetchers that can not retry ... debootstrap. That could
help reduce the number of manual increments, and maybe help
non-interactive builds with shaky networks.

Henning

[Baurzhan Ismagulov]

On Thu, May 07, 2020 at 04:36:18PM +0200, Jan Kiszka wrote:
> Avoids failing a complete rootfs installation in case of a short
> hick-up.

...

> Does something like this make sense? Do we have more places?
> We have a retry of 2 on wget bitbake fetches, e.g. (via FETCHCMD_wget).

To address the questions raised in the thread, I think a small improvement now
is a good thing. I also think it's worth to gradually fix all relevant places
and unify the retry handling.

I'd like to apply this as is. Jan, you are not planning an update for this, are
you?

With kind regards,
Baurzhan.

[Jan Kiszka]

No update planned. If you like to take it, I'm fine, if not, I'm also fine.

Jan

[Baurzhan Ismagulov]

On Mon, May 25, 2020 at 04:13:37PM +0200, Jan Kiszka wrote:
> No update planned.

Applied to next, thanks.

With kind regards,
Baurzhan.