[PATCH 1/4] rootfs: configure apt to always retry downloads

[henning...@siemens.com]

Let us stop the bit by bit micro fixes and have a central place for
everything "apt". To deal with download retries in case of network
issues.

Signed-off-by: Henning Schild <henning...@siemens.com>
---
meta/classes/image.bbclass | 1 +
meta/classes/rootfs.bbclass | 13 +++++++++++++
2 files changed, 14 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 6d77243cb1b1..9d783d80a228 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -221,6 +221,7 @@ do_rootfs_finalize() {
rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list"
rm -f "${ROOTFSDIR}/etc/apt/preferences.d/isar-apt"
rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
+ rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/80isar"

mv "${ROOTFSDIR}/etc/apt/sources-list" \
"${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 1fc487896f4d..e219be4f8529 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -98,6 +98,19 @@ EOF
EOSUDO
}

+ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_apt"
+rootfs_configure_apt[weight] = "2"
+rootfs_configure_apt() {
+ sudo -s <<'EOSUDO'
+ set -e
+
+ mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d'
+ echo 'APT::Acquire::Retries "3";' > \
+ '${ROOTFSDIR}/etc/apt/apt.conf.d/80isar'
+EOSUDO
+}
+
+
ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update"
rootfs_install_pkgs_update[weight] = "5"
rootfs_install_pkgs_update[isar-apt-lock] = "acquire-before"
--
2.34.1

[henning...@siemens.com]

This attempts to solve the apt fetch retry problem in a better way. On
the way it was found that bootstrap leaves a file around, which probably
should not remain in images.

Henning Schild (4):
rootfs: configure apt to always retry downloads
Revert "deb-dl-dir: Retry downloads 3 times for src files"
Revert "rootfs: Retry downloads 3 times"
bootstrap: do not leave 50isar.conf in final rootfss

meta/classes/deb-dl-dir.bbclass | 2 +-
meta/classes/image.bbclass | 1 +
meta/classes/rootfs.bbclass | 18 +++++++++++++++++-
.../isar-bootstrap/files/isar-apt.conf | 2 --
.../isar-bootstrap/isar-bootstrap.inc | 5 +----
5 files changed, 20 insertions(+), 8 deletions(-)
delete mode 100644 meta/recipes-core/isar-bootstrap/files/isar-apt.conf

--
2.34.1

[henning...@siemens.com]

This reverts commit 38160d28730234c482f4698dc8a63c9599914c7d.

Signed-off-by: Henning Schild <henning...@siemens.com>
---

meta/classes/rootfs.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index e219be4f8529..766a2170f43b 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -141,7 +141,7 @@ rootfs_install_pkgs_download[weight] = "600"
rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
rootfs_install_pkgs_download() {
sudo -E chroot '${ROOTFSDIR}' \
- /usr/bin/apt-get ${ROOTFS_APT_ARGS} -o Acquire::Retries=3 --download-only ${ROOTFS_PACKAGES}
+ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
}

ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""
--
2.34.1

[henning...@siemens.com]

This reverts commit d1cf0841362f5a9f8c2ab01d7cb084efca641208.

Signed-off-by: Henning Schild <henning...@siemens.com>
---

meta/classes/deb-dl-dir.bbclass | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 42542d74ff7b..ffbff0775575 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -57,7 +57,7 @@ debsrc_download() {
[ -n "$dscfile" ] && continue

sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
- sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y -o Acquire::Retries=3 --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
+ sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
done
) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"

--
2.34.1

[henning...@siemens.com]

This commits makes sure we do not leave a file around which is really
only meant for building. In case people will want to "apt-get" later
they might have their own ideas on how to deal with Recommends and
Suggests. And if so they should write a package to configure apt for
their images.

So it cleans up a file not owned by any package, and unifies the way apt
configuration for the build is done.

Signed-off-by: Henning Schild <henning...@siemens.com>
---

meta/classes/image.bbclass | 2 +-
meta/classes/rootfs.bbclass | 7 +++++--
meta/recipes-core/isar-bootstrap/files/isar-apt.conf | 2 --
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 +----
4 files changed, 7 insertions(+), 9 deletions(-)
delete mode 100644 meta/recipes-core/isar-bootstrap/files/isar-apt.conf

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 9d783d80a228..d44298bcdd7c 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -221,7 +221,7 @@ do_rootfs_finalize() {

rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list"
rm -f "${ROOTFSDIR}/etc/apt/preferences.d/isar-apt"
rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"

- rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/80isar"
+ rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar"

mv "${ROOTFSDIR}/etc/apt/sources-list" \
"${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 766a2170f43b..e8649975a333 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -105,8 +105,11 @@ rootfs_configure_apt() {
set -e

mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d'

- echo 'APT::Acquire::Retries "3";' > \
- '${ROOTFSDIR}/etc/apt/apt.conf.d/80isar'
+ {

+ echo 'APT::Acquire::Retries "3";'

+ echo 'APT::Install-Recommends "0";'
+ echo 'APT::Install-Suggests "0";'
+ } > '${ROOTFSDIR}/etc/apt/apt.conf.d/50isar'
EOSUDO
}

diff --git a/meta/recipes-core/isar-bootstrap/files/isar-apt.conf b/meta/recipes-core/isar-bootstrap/files/isar-apt.conf
deleted file mode 100644
index a62feb1722d6..000000000000
--- a/meta/recipes-core/isar-bootstrap/files/isar-apt.conf
+++ /dev/null
@@ -1,2 +0,0 @@
-APT::Install-Recommends "0";
-APT::Install-Suggests "0";
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 26205192ed9e..2f483f5a1b6a 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -9,7 +9,6 @@ LICENSE = "gpl-2.0"
LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
FILESPATH_prepend := "${THISDIR}/files:"
SRC_URI = " \
- file://isar-apt.conf \
file://isar-apt-fallback.conf \
file://locale \
file://chroot-setup.sh"
@@ -327,9 +326,6 @@ do_bootstrap() {
install -v -m644 "${APTSRCS_INIT}" "${ROOTFSDIR}/etc/apt/sources-list"
rm -f "${ROOTFSDIR}/etc/apt/sources.list"
rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*
- mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
- install -v -m644 "${WORKDIR}/isar-apt.conf" \
- "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
find ${APT_KEYS_DIR}/ -type f | while read keyfile
do
MY_GPGHOME="$(chroot "${ROOTFSDIR}" mktemp -d /tmp/gpghomeXXXXXXXXXX)"
@@ -348,6 +344,7 @@ do_bootstrap() {
done

if [ "${@get_distro_suite(d)}" = "stretch" ] && [ "${@get_host_release().split('.')[0]}" -lt "4" ]; then

+ mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"

install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \
"${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
fi
--
2.34.1

[Henning Schild]

I could not really test this because it needs a shaky network. Cutting
network will be detected as such and retries would not happen anyways.
At least the ways i tried. But i see no reason for this to not work.

regards,
Henning

Am Sat, 5 Feb 2022 16:15:24 +0100
schrieb Henning Schild <henning...@siemens.com>:

[Anton Mikanovich]

Applied to next, thanks.

[Henning Schild]

Am Mon, 21 Feb 2022 12:24:09 +0300
schrieb Anton Mikanovich <ami...@ilbers.de>:

Thanks! I see only now that the "From:" in my emails was messed with by
the SMTP server i am now forced to use. Any chance we can force-push
next to fix that?

What i did send and wanted was
"Henning Schild <henning...@siemens.com>" but what came out is
"henning...@siemens.com <henning...@siemens.com>"

Please let me know if a force-push is possible, i would prepare a
"next" on github for a hard reset.

regards,
Henning

[Baurzhan Ismagulov]

On Mon, Feb 21, 2022 at 05:12:55PM +0100, Henning Schild wrote:
> Thanks! I see only now that the "From:" in my emails was messed with by
> the SMTP server i am now forced to use. Any chance we can force-push
> next to fix that?
>
> What i did send and wanted was
> "Henning Schild <henning...@siemens.com>" but what came out is
> "henning...@siemens.com <henning...@siemens.com>"
>
> Please let me know if a force-push is possible, i would prepare a
> "next" on github for a hard reset.

Fixed, please check.

In general, I'd like to avoid rewriting history because it's, well, history :)
.

With kind regards,
Baurzhan.

[Henning Schild]

Am Mon, 21 Feb 2022 17:51:05 +0100
schrieb Baurzhan Ismagulov <i...@radix50.net>:

Thanks a million! I know that history rewriting is bad and isar usually
does not do it ... even on "next" where other projects might be more
relaxed about it.

I sure hope it does not happen again and it would be asking too much
for the maintainer to maybe double-check. But if anyone sees a "weird"
"From" just speak up and ask the contributor to check their mail setup.

I personally would do my best to deliver patches again and try to help
people to fix such issues (especially those trying to send From
@siemens.com).

regards,
Henning

> With kind regards,
> Baurzhan.
>

[Jan Kiszka]

I heard these complaints from other as well. Since our servers started
rewriting but also because of me sometime using different servers, I
*always* force git to inject a From: line into the commit messages. That
makes you independent of those manglings.

BTW, my local maintainer tool tries to detect such cases, at least when
there is also a signed-off line, and that one differs from the author's
line.

Jan

--
Siemens AG, Technology
Competence Center Embedded Linux

[Henning Schild]

Am Mon, 21 Feb 2022 20:20:21 +0100
schrieb Jan Kiszka <jan.k...@siemens.com>:

How do you force git? I am using my own force since a couple of years
... not sure why it failed this time.

https://github.com/henning-schild/git/commit/ca44f026a31537477e66559f0bf0f3acd2deecdf

I should probably go ahead and mainline that or something similar.

> BTW, my local maintainer tool tries to detect such cases, at least
> when there is also a signed-off line, and that one differs from the
> author's line.

Maybe such a tool should not remain "local", but really it should not
be required if mail servers would simply send what they are told to.

In fact i will likely switch to sending from "@hennsch.de" soon, not
caring anymore.

regards,
Henning

> Jan
>

[Bezdeka, Florian]

I'm forcing git send-email to repeat the FROM header inside the body by
setting sendemail.from in my ~/.gitconfig to a value that is different
from my "git commit identity" (user.name and user.email):

[sendemail]
from = F. Bezdeka <email>

[Jan Kiszka]

git format-patch --form="bla"
-> adds "From:" to body (unless you committed as "bla")
git send-email --from="re...@user.com"

>
>> BTW, my local maintainer tool tries to detect such cases, at least
>> when there is also a signed-off line, and that one differs from the
>> author's line.
>
> Maybe such a tool should not remain "local", but really it should not
> be required if mail servers would simply send what they are told to.

https://addons.thunderbird.net/de/thunderbird/addon/copy-patch/

Jan

>
> In fact i will likely switch to sending from "@hennsch.de" soon, not
> caring anymore.
>
> regards,
> Henning
>
>> Jan
>>
>