[PATCH 0/2] Support for ccache

[Uladzimir Bely]

Some custom user packages built from sources may be written in C/C++.
Using ccache will help decrease build time in case they are rebuilt.

For example, building `mc:stm32mp15x-buster:linux-mainline` during the
test took 28 minutes at second build with ccache enabled in comparison
with 115 minutes of first build.

Uladzimir Bely (2):
meta: Support for ccache for custom packages in buildchroot
doc: Add section about ccache usage

doc/user_manual.md | 16 +++++++++++++
meta-isar/conf/local.conf.sample | 5 ++++
meta/classes/buildchroot.bbclass | 6 +++++
meta/classes/ccache.bbclass | 23 +++++++++++++++++++
meta/classes/dpkg.bbclass | 12 +++++++++-
.../buildchroot/buildchroot.inc | 1 +
.../buildchroot/files/build.sh | 6 +++++
.../buildchroot/files/common.sh | 1 +
8 files changed, 69 insertions(+), 1 deletion(-)
create mode 100644 meta/classes/ccache.bbclass

--
2.20.1

[Uladzimir Bely]

Add `INHERIT += "ccache"` to local.conf or package recipe
to globally activate the ccache functionality.
Add `CCACHE_DISABLE = '1'` to the recipe to disable ccache for it.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta-isar/conf/local.conf.sample | 5 ++++
meta/classes/buildchroot.bbclass | 6 +++++
meta/classes/ccache.bbclass | 23 +++++++++++++++++++
meta/classes/dpkg.bbclass | 12 +++++++++-
.../buildchroot/buildchroot.inc | 1 +
.../buildchroot/files/build.sh | 6 +++++
.../buildchroot/files/common.sh | 1 +

7 files changed, 53 insertions(+), 1 deletion(-)
create mode 100644 meta/classes/ccache.bbclass

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 96a8beb..0f1a030 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -222,3 +222,8 @@ USER_isar[flags] += "clear-text-password"

# Uncomment the below line to debug WIC.
# WIC_CREATE_EXTRA_ARGS += "-D"
+
+# Uncomment this to use ccache for custom packages
+#INHERIT += "ccache"
+# Uncomment and set own top level ccache directory to share between builds
+#CCACHE_TOP_DIR ?= "${TMPDIR}/ccache"
diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index e9eb9af..598458d 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -32,6 +32,12 @@ buildchroot_do_mounts() {
mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${BUILDCHROOT_DIR}/isar-apt'
mountpoint -q '${BUILDCHROOT_DIR}/downloads' ||
mount --bind '${DL_DIR}' '${BUILDCHROOT_DIR}/downloads'
+ if [ "${@repr(bb.utils.to_boolean(d.getVar('CCACHE_BUILDCHROOT')))}" = 'True' ]
+ then
+ mkdir -p '${BUILDCHROOT_DIR}/ccache'
+ mountpoint -q '${BUILDCHROOT_DIR}/ccache' ||
+ mount --bind '${CCACHE_DIR}' '${BUILDCHROOT_DIR}/ccache'
+ fi
mountpoint -q '${BUILDCHROOT_DIR}/dev' ||
mount --rbind /dev '${BUILDCHROOT_DIR}/dev'
mount --make-rslave '${BUILDCHROOT_DIR}/dev'
diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
new file mode 100644
index 0000000..2259417
--- /dev/null
+++ b/meta/classes/ccache.bbclass
@@ -0,0 +1,23 @@
+#
+# Usage:
+# - Enable ccache
+# Add the following line to a conffile such as conf/local.conf:
+# INHERIT += "ccache"
+#
+# - Disable ccache for a recipe
+# Add the following line to the recipe if it can't be built with ccache:
+# CCACHE_DISABLE = '1'
+#
+# - Share ccache files between different builds
+# Set CCACHE_TOP_DIR to a shared dir
+# CCACHE_TOP_DIR = /path/to/shared_ccache/
+
+CCACHE_TOP_DIR ?= "${TMPDIR}/ccache"
+CCACHE_DIR ?= "${CCACHE_TOP_DIR}/${DISTRO}-${MACHINE}"
+
+CCACHE_BUILDCHROOT ?= "1"
+
+do_init_ccache[dirs] = "${CCACHE_DIR}"
+do_init_ccache() {
+}
+addtask do_init_ccache before do_rootfs
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 4e7c2f7..af2ba2e 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -32,6 +32,16 @@ addtask devshell after do_install_builddeps
dpkg_runbuild() {
E="${@ isar_export_proxies(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"
+ if [ "${@repr(bb.utils.to_boolean(d.getVar('CCACHE_BUILDCHROOT')))}" = 'True' ]
+ then
+ USE_CCACHE="1"
+ else
+ USE_CCACHE="0"
+ fi
+ if [ "${@repr(bb.utils.to_boolean(d.getVar('CCACHE_DISABLE')))}" = 'True' ]
+ then
+ USE_CCACHE="0"
+ fi
sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
- /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+ /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH} ${USE_CCACHE}
}
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 31524a1..726c7bb 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -32,6 +32,7 @@ BUILDCHROOT_PREINSTALL_COMMON = " \
make \
debhelper \
autotools-dev \
+ ccache \
dpkg \
locales \
docbook-to-man \
diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
index 101581d..b642f42 100644
--- a/meta/recipes-devtools/buildchroot/files/build.sh
+++ b/meta/recipes-devtools/buildchroot/files/build.sh
@@ -14,4 +14,10 @@ for i in configure aclocal.m4 Makefile.am Makefile.in; do
fi
done

+if [ "$use_ccache" == "1" ]
+then
+ export CCACHE_DIR=/ccache
+ export PATH=/usr/lib/ccache:$PATH
+fi
+
${GBP_PREFIX}dpkg-buildpackage -a$target_arch -d --source-option=-I
diff --git a/meta/recipes-devtools/buildchroot/files/common.sh b/meta/recipes-devtools/buildchroot/files/common.sh
index 0063a38..7085f44 100644
--- a/meta/recipes-devtools/buildchroot/files/common.sh
+++ b/meta/recipes-devtools/buildchroot/files/common.sh
@@ -17,6 +17,7 @@ fi

# Create human-readable names
target_arch=$2
+use_ccache=$3

set_arch="--host-arch $target_arch"

--
2.20.1

[Uladzimir Bely]

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

doc/user_manual.md | 16 ++++++++++++++++
1 file changed, 16 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 1da0e9d..903f9ae 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -18,6 +18,7 @@ Copyright (C) 2016-2019, ilbers GmbH
- [Add a New Image Type](#add-a-new-image-type)
- [Add a Custom Application](#add-a-custom-application)
- [Enabling Cross-compilation](#isar-cross-compilation)
+ - [Using ccache for custom packages](#using-ccache-for-custom-packages)
- [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)
- [Create a containerized Isar SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem)
- [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages)
@@ -896,6 +897,21 @@ package build. To invoke it, just call
`bitbake mc:${MACHINE}-${DISTRO}:<package_name> -c devshell`.


+## Using ccache for custom packages
+
+While base system is created from binary debian repositories, some user
+packages are built from sources. It's possible to reduce build time
+for such packages by enabling ccache.
+
+To enable global ccache functionality, `INHERIT += ccache` can be added
+to `local.conf`. If some package requires ccache to be always disabled,
+`CCACHE_DISABLE = '1'` can be used in the recipe.
+
+By default, ccache directory is created inside `TMPDIR`, but it can be
+adjusted by `CCACHE_TOP_DIR` variable in `local.conf`. Ccache directory
+`CCACHE_DIR` default value is `"${CCACHE_TOP_DIR}/${DISTRO}-${MACHINE}"`,
+that means caches for different distros and machines are not overlapped.
+
## Create an ISAR SDK root filesystem

### Motivation
--
2.20.1

[Jan Kiszka]

Why so complex? ISAR_CROSS_COMPILE is managed with a single variable,
including per-recipe out-out/in. Would be better to follow that pattern
consistently.

> +
> +By default, ccache directory is created inside `TMPDIR`, but it can be
> +adjusted by `CCACHE_TOP_DIR` variable in `local.conf`. Ccache directory
> +`CCACHE_DIR` default value is `"${CCACHE_TOP_DIR}/${DISTRO}-${MACHINE}"`,

DISTRO-DISTRO_ARCH, just like for other cachings. There is nothing
machine-specific in the cache.

Jan

> +that means caches for different distros and machines are not overlapped.
> +
> ## Create an ISAR SDK root filesystem
>
> ### Motivation
>

--

Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

[Jan Kiszka]

On 13.09.21 17:10, Uladzimir Bely wrote:

Why so complex? Elsewhere, we just do

if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then

and are done.

Way to convoluted as I already commented on patch 2. Just use a single
ISAR_CCACHE_ENABLE test here. That var can be set globally (local.conf &
Co) and overwritten locally (recipes).

You can do that unconditionally. That will also save you from passing
use_ccache down.

> +
> ${GBP_PREFIX}dpkg-buildpackage -a$target_arch -d --source-option=-I
> diff --git a/meta/recipes-devtools/buildchroot/files/common.sh b/meta/recipes-devtools/buildchroot/files/common.sh
> index 0063a38..7085f44 100644
> --- a/meta/recipes-devtools/buildchroot/files/common.sh
> +++ b/meta/recipes-devtools/buildchroot/files/common.sh
> @@ -17,6 +17,7 @@ fi
>
> # Create human-readable names
> target_arch=$2
> +use_ccache=$3
>
> set_arch="--host-arch $target_arch"
>
>

In general, a very nice feature! I'm looking forward to much faster
kernel builds when only adding a patch on top of flipping a simple
feature. We just need to tune the interface, and the implementation can
become even more compact.

Jan

[Henning Schild]

This looks like a very good addition to the sstate feature that has
been brewing in our basement. The combination would be really cool.

With sstate we are already seeing drastic speedups because tasks get
skipped over. But if a long compile task can not be taken from the
cache, this guy could come in as just the helper we needed here.

We will look into combining the two and see when our in-house stuff is
ready for sharing.

Our biggest issue still is "eviction". We decided to solve that with
cron and wipe the whole cache every night. For sstate that is important
because if debootstrap rootfss (something we cache) get too old, they
would need "apt-get update/upgrade" ... so we only cache for so long.

Another reason for eviction might not just be age but space. How to
make sure that ccache does not grow forever? Instructions on how to
evict are missing in the documentation. Or is it somehow self
regulating?

The next question is whether it is thread-safe, meaning many instances
of isar/bitbake can use it at the same time. Possibly from multiple
machines sharing that cache folder ...

Henning

Am Mon, 13 Sep 2021 17:10:07 +0200
schrieb Uladzimir Bely <ub...@ilbers.de>:

[Jan Kiszka]

On 15.09.21 18:43, Henning Schild wrote:
> This looks like a very good addition to the sstate feature that has
> been brewing in our basement. The combination would be really cool.
>
> With sstate we are already seeing drastic speedups because tasks get
> skipped over. But if a long compile task can not be taken from the
> cache, this guy could come in as just the helper we needed here.
>
> We will look into combining the two and see when our in-house stuff is
> ready for sharing.
>
> Our biggest issue still is "eviction". We decided to solve that with
> cron and wipe the whole cache every night. For sstate that is important
> because if debootstrap rootfss (something we cache) get too old, they
> would need "apt-get update/upgrade" ... so we only cache for so long.
>
> Another reason for eviction might not just be age but space. How to
> make sure that ccache does not grow forever? Instructions on how to
> evict are missing in the documentation. Or is it somehow self
> regulating?

It's self-regulating, and you can set an upper size limit in
~/.ccache/ccache.conf. I'm running it locally for ages without observing
issues.

>
> The next question is whether it is thread-safe, meaning many instances
> of isar/bitbake can use it at the same time. Possibly from multiple
> machines sharing that cache folder ...

It is by nature as it has to support make -j as well.

ccache is a rather solid, actually old technology. When we use it in
lock-step with the compiler shipped by the same debian release, we are
on a safe ground. If that qualifies for default-on, I don't know, but at
least "recommend-on".

[Baurzhan Ismagulov]

On Mon, Sep 13, 2021 at 08:16:53PM +0200, Jan Kiszka wrote:
> > +To enable global ccache functionality, `INHERIT += ccache` can be added
> > +to `local.conf`. If some package requires ccache to be always disabled,
> > +`CCACHE_DISABLE = '1'` can be used in the recipe.
>
> Why so complex? ISAR_CROSS_COMPILE is managed with a single variable,
> including per-recipe out-out/in. Would be better to follow that pattern
> consistently.

I'm all in. The only reason for this was following Yocto user interfaces 1:1.
We'll rework to a single var with positive logic.

With kind regards,
Baurzhan.