[PATCH 0/8] CI setup optimizations

[Daniel Braunwarth]

This series of patches optimizes the CI setup.

The installed Debian packages are optimized to only install what is really needed.

New setup.sh on Debian testing:

39 upgraded, 624 newly installed, 5 to remove and 38 not upgraded.
Need to get 185 MB of archives.
After this operation, 739 MB of additional disk space will be used.

Old setup.sh on Debian testing:

43 upgraded, 410 newly installed, 5 to remove and 34 not upgraded.
Need to get 427 MB of archives.
After this operation, 1362 MB of additional disk space will be used.


The series has been tested with the following containers:

- debian:testing-slim
- ubuntu:22.04
- ubuntu:24.04

The following commands run in each of them:

./ci/setup.sh
./ci/test-configs.sh

Daniel Braunwarth (8):
acceptance-tests: replace wget with curl
ci: drop lua pkg-config link
ci: sort packages
ci: optimize setup
ci: build dependencies out of our tree
ci: use meson to build zchunk
ci: fix zchunk installation
ci: remove apt cache during setup

ci/install-src-deps.sh | 33 ++++----
ci/setup.sh | 105 ++++++++++++-------------
scripts/acceptance-tests/CheckImage.mk | 2 +-
3 files changed, 69 insertions(+), 71 deletions(-)

--
2.45.2

[Daniel Braunwarth]

Signed-off-by: Daniel Braunwarth <o...@braunwarth.dev>
---
ci/setup.sh | 56 ++++++++++++++++++++++++++---------------------------
1 file changed, 28 insertions(+), 28 deletions(-)

diff --git a/ci/setup.sh b/ci/setup.sh
index 1c4736b..b6f3e31 100755
--- a/ci/setup.sh
+++ b/ci/setup.sh
@@ -23,48 +23,48 @@ export TZ=Europe/London

$_SUDO apt-get -qq update
$_SUDO apt-get install -y \
- build-essential \
+ autoconf-archive \
automake \
+ build-essential \
+ check \
cmake \
+ cpio \
curl \
- libzmq3-dev \
- liblua5.2-dev \
- libluajit-5.1-dev \
- libconfig-dev \
+ doxygen \
+ gawk \
+ git \
+ graphviz \
libarchive-dev \
libbtrfsutil-dev \
+ libcmocka-dev \
+ libconfig-dev \
+ libcurl4-openssl-dev \
+ libext2fs-dev \
+ libfdisk-dev \
+ libgpiod-dev \
libjson-c-dev \
- libyaml-dev \
- zlib1g-dev \
- git \
- uuid \
- uuid-dev \
+ liblua5.2-dev \
+ libluajit-5.1-dev \
liblzo2-dev \
+ libmbedtls-dev \
+ libpci-dev \
+ librsync-dev \
+ librsync2 \
libsystemd-dev \
libsystemd0 \
- check \
- librsync2 \
- librsync-dev \
- libext2fs-dev \
liburiparser-dev \
- doxygen \
- graphviz \
- autoconf-archive \
- linux-headers-generic \
- libmbedtls-dev \
- libcmocka-dev \
- libfdisk-dev \
libwebsockets-dev \
- libgpiod-dev \
- libcurl4-openssl-dev \
- libpci-dev \
- gawk \
- cpio \
+ libyaml-dev \
+ libzmq3-dev \
+ libzstd-dev \
+ linux-headers-generic \
meson \
ninja-build \
- libzstd-dev \
+ python3 \
+ uuid \
+ uuid-dev \
wget \
- python3
+ zlib1g-dev

# packages are too old in Ubuntu Jammy
if ! grep -q UBUNTU_CODENAME=jammy /etc/os-release; then
--
2.45.2

[Daniel Braunwarth]

Not all of the packages are really needed. Some of of them are only
needed on Ubuntu Jammy. Some of them are not needed at all.

Signed-off-by: Daniel Braunwarth <o...@braunwarth.dev>
---

ci/setup.sh | 41 ++++++++++++++++++++---------------------
1 file changed, 20 insertions(+), 21 deletions(-)

diff --git a/ci/setup.sh b/ci/setup.sh
index b6f3e31..2de33ed 100755
--- a/ci/setup.sh
+++ b/ci/setup.sh
@@ -21,59 +21,58 @@ fi
export DEBIAN_FRONTEND=noninteractive
export TZ=Europe/London

-$_SUDO apt-get -qq update
-$_SUDO apt-get install -y \
- autoconf-archive \
- automake \
- build-essential \
- check \
- cmake \
+$_SUDO apt-get -qq update && apt-get install --yes --no-install-recommends \
cpio \
curl \
- doxygen \
gawk \
+ gcc \
git \
- graphviz \
+ gulp \
libarchive-dev \
+ libblkid-dev \
libbtrfsutil-dev \
libcmocka-dev \
libconfig-dev \
libcurl4-openssl-dev \
+ libczmq-dev \
libext2fs-dev \
libfdisk-dev \
libgpiod-dev \
libjson-c-dev \
liblua5.2-dev \
libluajit-5.1-dev \
- liblzo2-dev \
libmbedtls-dev \
- libpci-dev \
librsync-dev \
- librsync2 \
+ libssl-dev \
libsystemd-dev \
- libsystemd0 \
+ libudev-dev \
liburiparser-dev \
libwebsockets-dev \
- libyaml-dev \
- libzmq3-dev \
libzstd-dev \
- linux-headers-generic \
- meson \
- ninja-build \
+ make \
+ npm \
python3 \
- uuid \
uuid-dev \
- wget \

zlib1g-dev

# packages are too old in Ubuntu Jammy
if ! grep -q UBUNTU_CODENAME=jammy /etc/os-release; then

- apt-get install -y \
+ $_SUDO apt-get -qq update && apt-get install --yes --no-install-recommends \
libebgenv-dev \
libmtd-dev \
libubi-dev \
libubootenv-dev \
libzck-dev
else
+ $_SUDO apt-get -qq update && apt-get install --yes --no-install-recommends \
+ autoconf \
+ autoconf-archive \
+ automake \
+ check \
+ cmake \
+ liblzo2-dev \
+ libtool \
+ libyaml-dev \
+ meson
"$SCRIPT_DIR/install-src-deps.sh"
fi
--
2.45.2

[Daniel Braunwarth]

This is, as far as I can see, the only existing usage of wget.

Dropping wget in favour of curl enables us to remove wget as a build
dependency.

Signed-off-by: Daniel Braunwarth <o...@braunwarth.dev>
---

scripts/acceptance-tests/CheckImage.mk | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/acceptance-tests/CheckImage.mk b/scripts/acceptance-tests/CheckImage.mk
index b7c8791..e29fef5 100644
--- a/scripts/acceptance-tests/CheckImage.mk
+++ b/scripts/acceptance-tests/CheckImage.mk
@@ -29,7 +29,7 @@ quiet_cmd_sign_desc = SIGN $@

URL = https://raw.githubusercontent.com/openssl/openssl/master/demos/cms
quiet_cmd_download = GET $@
- cmd_download = rm -f $@.tmp && wget -O $@.tmp $(URL)/$(notdir $@) && mv $@.tmp $@
+ cmd_download = rm -f $@.tmp && curl -s -o $@.tmp $(URL)/$(notdir $@) && mv $@.tmp $@

#
# tests to run
--
2.45.2

[Daniel Braunwarth]

This should not be needed anymore since b8f72939.

Signed-off-by: Daniel Braunwarth <o...@braunwarth.dev>
---

ci/setup.sh | 3 ---
1 file changed, 3 deletions(-)

diff --git a/ci/setup.sh b/ci/setup.sh
index 152225e..1c4736b 100755
--- a/ci/setup.sh
+++ b/ci/setup.sh
@@ -77,6 +77,3 @@ if ! grep -q UBUNTU_CODENAME=jammy /etc/os-release; then
else
"$SCRIPT_DIR/install-src-deps.sh"
fi
-
-PC_FILE_DIR=$(pkg-config --variable=pcfiledir lua5.2)
-$_SUDO ln -sf "$PC_FILE_DIR/lua5.2.pc" "$PC_FILE_DIR/lua.pc"
--
2.45.2

[Daniel Braunwarth]

This change optimizes the container layer size.

See https://docs.docker.com/build/building/best-practices.

Signed-off-by: Daniel Braunwarth <o...@braunwarth.dev>
---

ci/setup.sh | 95 +++++++++++++++++++++++++++--------------------------
1 file changed, 49 insertions(+), 46 deletions(-)

diff --git a/ci/setup.sh b/ci/setup.sh
index 2de33ed..f55aa99 100755
--- a/ci/setup.sh
+++ b/ci/setup.sh
@@ -22,57 +22,60 @@ export DEBIAN_FRONTEND=noninteractive
export TZ=Europe/London

$_SUDO apt-get -qq update && apt-get install --yes --no-install-recommends \

- cpio \
- curl \
- gawk \
- gcc \
- git \
- gulp \
- libarchive-dev \
- libblkid-dev \
- libbtrfsutil-dev \
- libcmocka-dev \
- libconfig-dev \
- libcurl4-openssl-dev \
- libczmq-dev \
- libext2fs-dev \
- libfdisk-dev \
- libgpiod-dev \
- libjson-c-dev \
- liblua5.2-dev \
- libluajit-5.1-dev \
- libmbedtls-dev \
- librsync-dev \
- libssl-dev \
- libsystemd-dev \
- libudev-dev \
- liburiparser-dev \
- libwebsockets-dev \
- libzstd-dev \
- make \
- npm \
- python3 \
- uuid-dev \
- zlib1g-dev
+ cpio \
+ curl \
+ gawk \
+ gcc \
+ git \
+ gulp \
+ libarchive-dev \
+ libblkid-dev \
+ libbtrfsutil-dev \

+ libcmocka-dev \
+ libconfig-dev \
+ libcurl4-openssl-dev \

+ libczmq-dev \

+ libext2fs-dev \
+ libfdisk-dev \
+ libgpiod-dev \

+ libjson-c-dev \
+ liblua5.2-dev \
+ libluajit-5.1-dev \
+ libmbedtls-dev \
+ librsync-dev \
+ libssl-dev \
+ libsystemd-dev \
+ libudev-dev \
+ liburiparser-dev \
+ libwebsockets-dev \
+ libzstd-dev \
+ make \
+ npm \
+ python3 \
+ uuid-dev \
+ zlib1g-dev \
+ && rm -rf /var/lib/apt/lists/*

# packages are too old in Ubuntu Jammy

if ! grep -q UBUNTU_CODENAME=jammy /etc/os-release; then

$_SUDO apt-get -qq update && apt-get install --yes --no-install-recommends \

- libebgenv-dev \
- libmtd-dev \
- libubi-dev \
- libubootenv-dev \
- libzck-dev
+ libebgenv-dev \
+ libmtd-dev \
+ libubi-dev \
+ libubootenv-dev \
+ libzck-dev \
+ && rm -rf /var/lib/apt/lists/*
else

$_SUDO apt-get -qq update && apt-get install --yes --no-install-recommends \

- autoconf \
- autoconf-archive \
- automake \
- check \
- cmake \
- liblzo2-dev \
- libtool \
- libyaml-dev \
- meson

+ autoconf \
+ autoconf-archive \
+ automake \
+ check \
+ cmake \
+ liblzo2-dev \
+ libtool \
+ libyaml-dev \

+ meson \
+ && rm -rf /var/lib/apt/lists/*
"$SCRIPT_DIR/install-src-deps.sh"
fi
--
2.45.2

[Daniel Braunwarth]

Signed-off-by: Daniel Braunwarth <o...@braunwarth.dev>
---

ci/install-src-deps.sh | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/ci/install-src-deps.sh b/ci/install-src-deps.sh
index c89293f..302d703 100755
--- a/ci/install-src-deps.sh
+++ b/ci/install-src-deps.sh
@@ -56,10 +56,9 @@ install_zchunk() {
rm -rf /tmp/zchunk
git clone https://github.com/zchunk/zchunk /tmp/zchunk
cd /tmp/zchunk
- meson build
- cd build
- ninja
- $_SUDO ninja install
+ meson setup build
+ meson compile -C build
+ $_SUDO meson install -C build
}

install_mtd_utils
--
2.45.2

[Daniel Braunwarth]

Signed-off-by: Daniel Braunwarth <o...@braunwarth.dev>
---

ci/install-src-deps.sh | 26 +++++++++++++-------------
1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/ci/install-src-deps.sh b/ci/install-src-deps.sh
index 2616679..c89293f 100755
--- a/ci/install-src-deps.sh
+++ b/ci/install-src-deps.sh
@@ -19,9 +19,10 @@ install_mtd_utils() {
$_SUDO mkdir -p /usr/local/lib
$_SUDO mkdir -p /usr/local/include
$_SUDO mkdir -p /usr/local/include/mtd
- git clone https://github.com/sigma-star/mtd-utils
- cd mtd-utils
- git checkout -b tmp v2.0.0
+ rm -rf /tmp/mtd-utils
+ git clone https://github.com/sigma-star/mtd-utils /tmp/mtd-utils
+ cd /tmp/mtd-utils
+ git checkout v2.0.0
./autogen.sh
./configure
make -j$(nproc)
@@ -29,37 +30,36 @@ install_mtd_utils() {
$_SUDO install -m 644 include/libmtd.h /usr/local/include
$_SUDO install -m 644 include/mtd/ubi-media.h /usr/local/include/mtd
$_SUDO install -m 644 *.a /usr/local/lib
- cd ..
}

install_libubootenv() {
- git clone https://github.com/sbabic/libubootenv.git
- cd libubootenv
+ rm -rf /tmp/libubootenv
+ git clone https://github.com/sbabic/libubootenv.git /tmp/libubootenv
+ cd /tmp/libubootenv
cmake .
make -j$(nproc)
$_SUDO make install
- cd ..
}

install_efibootguard() {
- git clone https://github.com/siemens/efibootguard.git
- cd efibootguard
+ rm -rf /tmp/efibootguard
+ git clone https://github.com/siemens/efibootguard.git /tmp/efibootguard
+ cd /tmp/efibootguard
git submodule update --init
autoreconf -fi
./configure --disable-bootloader
make -j$(nproc)
$_SUDO make install
- cd ..
}

install_zchunk() {
- git clone https://github.com/zchunk/zchunk
- cd zchunk
+ rm -rf /tmp/zchunk
+ git clone https://github.com/zchunk/zchunk /tmp/zchunk
+ cd /tmp/zchunk
meson build
cd build
ninja
$_SUDO ninja install
- cd ..
}

install_mtd_utils
--
2.45.2

[Daniel Braunwarth]

Without this fix zchunk is installed to /usr/local and /usr/lib64.

At least for the Ubuntu 22.04 container this prevented running
test-configs.sh.

Signed-off-by: Daniel Braunwarth <o...@braunwarth.dev>
---

ci/install-src-deps.sh | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ci/install-src-deps.sh b/ci/install-src-deps.sh
index 302d703..bfef070 100755
--- a/ci/install-src-deps.sh
+++ b/ci/install-src-deps.sh
@@ -56,7 +56,7 @@ install_zchunk() {
rm -rf /tmp/zchunk

git clone https://github.com/zchunk/zchunk /tmp/zchunk

cd /tmp/zchunk
- meson setup build
+ meson setup --prefix /usr --libdir lib build
meson compile -C build

$_SUDO meson install -C build
}

--
2.45.2

[Stefano Babic]

Reviewed-by: Stefano Babic <stefan...@swupdate.org>

Best regards,
Stefano Babic

[Stefano Babic]

On 1/4/25 22:48, Daniel Braunwarth wrote:

[Stefano Babic]

On 1/4/25 22:48, Daniel Braunwarth wrote:

[Stefano Babic]

On 1/4/25 22:48, Daniel Braunwarth wrote:

Not clear to me the purpose. Zchunk's maintainer explicitly recommends
ninja to build the package.

Best regards,
Stefano

[Daniel Braunwarth]

Hi Stefano

This is how meson itself suggests to compile:
See https://mesonbuild.com/Quick-guide.html#compiling-a-meson-project

Finally meson calls ninja so the result is the same in this case. If
ninja is not the backend used to build one don't need to distinguish
which command needs to be called to build. This will be done by meson
automatically.

I can send v2 of the series if you like to keep is as it is. Just let me
know.


Regards
Daniel