Enable ccache for CI builds?

[Vadim Zeitlin]

Hello,

I've enabled ccache[0] in https://github.com/wxWidgets/wxWidgets/pull/2571,
however this wasn't such an obvious gain as I hoped it would be.

Here are the selected build times for a CI run just before this commit[1],
the first CI run after adding ccache[2], i.e. without any cache yet, and
another CI run[3] immediately afterwards, i.e. without any changes to the
sources, resulting in 99% cache hit rate:

CI run | wxX11 | wxGTK2 | wxGTK3 STL | Total |
----------|-------|--------|------------|-------|
No ccache | 7.5 | 14 | 21 | 21.5 |
100% miss | 14 | 19 | 25.5 | 26 |
99% hit | 6.5 | 8 | 8.5 | 11 |

(times are in minutes, rounded).

As you can see, using ccache does result in noticeable gains when we don't
actually need to recompile anything, but it also has huge negative effect
when the cache is being populated, which is much bigger than what I would
have naively expected. I don't know if it's possible to reduce it by
tweaking ccache options (using "direct mode" only?) or even if we care
about it, i.e. whether the extra overhead will ever be so big again.

Does anybody here have more experience of using ccache with wxWidgets?
I've stopped using it myself many years ago because it didn't work well
with the precompiled headers we still used back then and didn't start using
it again even when we disabled the use of PCH by default with gcc, so I'm
not sure if it's worth it, but it would seem that it could still be useful
for the CI builds, which often run even if nothing much has been changed.

Thanks,
VZ

[0]: https://github.com/wxWidgets/wxWidgets/pull/2571/commits/9320aeb28bb639b99d3bd3545568334237d76fee
[1]: https://github.com/wxWidgets/wxWidgets/actions/runs/1391857016
[2]: https://github.com/wxWidgets/wxWidgets/actions/runs/1391955314
[3]: https://github.com/wxWidgets/wxWidgets/actions/runs/1392059407

[Robin Dunn]

I used ccache a lot, up until several (7-10?) years ago. Around that
time I found that the real-world build times (i.e. when at least some
non-trivial number of files needed rebuilt) that the build times with
ccache and the build times without were approaching the same average value.

My theory was that CPU efficiency improvements have outpaced disk IO
improvements enough that avoiding CPU-bound tasks, like compiling code,
and replacing it with more disk IO, was simply not as big of an
improvement as it used to be.

Robin


Vadim Zeitlin wrote on 10/27/21 3:08 PM:

[Vadim Zeitlin]

On Thu, 25 Nov 2021 14:17:48 -0800 Robin Dunn wrote:

RD> I used ccache a lot, up until several (7-10?) years ago. Around that
RD> time I found that the real-world build times (i.e. when at least some
RD> non-trivial number of files needed rebuilt) that the build times with
RD> ccache and the build times without were approaching the same average value.
RD>
RD> My theory was that CPU efficiency improvements have outpaced disk IO
RD> improvements enough that avoiding CPU-bound tasks, like compiling code,
RD> and replacing it with more disk IO, was simply not as big of an
RD> improvement as it used to be.

This makes sense in theory, but OTOH CI builds are not typical real-world
builds as they often rebuild everything (i.e. start from the clean slate),
so I hoped that ccache might still help for them.

I've ended up merging that PR, but I admit I haven't really checked the
build times much since then. I wonder if there is already a web app
somewhere allowing to plot the CI build times? It looks like this could be
pretty useful for many various reasons and I think that I did see something
like this somewhere, but can't find it right now.

Regards,
VZ

[wsu]

For what it's worth, last week we were doing some experiments with ccache at my job (which has nothing to do with wxWidgets), and found that one package we tried to build reported errors at build time, and another did not report any errors at build time, but failed to actually run.  I therefore don't trust ccache, and am lobbying not to use it at work.

[Vadim Zeitlin]

On Fri, 26 Nov 2021 16:25:15 -0800 (PST) wsu wrote:

w> For what it's worth, last week we were doing some experiments with ccache
w> at my job (which has nothing to do with wxWidgets), and found that one
w> package we tried to build reported errors at build time, and another did
w> not report any errors at build time, but failed to actually run. I
w> therefore don't trust ccache, and am lobbying not to use it at work.

This is indeed worrisome, but perhaps it's something that happens with
only some very specific build systems? In any case, if you can reproduce
it, it's probably worth reporting it to ccache developers, it definitely
wouldn't be nice if the build behaviour actually changed due to ccache use.
Luckily, I haven't seen this happen with wx (nor another project for which
I also enabled use of ccache for the CI builds).

Regards,
VZ

[wsu]

The build environment in question is quite complex; it's a Yocto image build.  (Yocto is a way to create one's own specialized Linux distribution, typically for embedded hardware systems, and involves cross-compiling everything intended to run on the destination hardware.)  In some previous situations, trying to coach consultants through dealing with Yocto has required substantial effort, and I doubt it's worth the effort just to get ccache working.  We'll see how other people in my company feel.