Build cache for offline builds

[Sidhin Thomas]

Hey all,

Had a question for folks who use offline build in their workflow. I wanted to check what kind of build configs are useful to facilitate faster dev builds. 

Mostly around build cache. With RBE that's implicit. But for offline builds, is there a way to cache builds outputs between different users in same machine or chromium checkouts? 

For example I built a branch based on main on my checkout no. 1 - > ~/cr1/ and want to build another branch based on main in a second checkout ~/cr2/ . Is there a reliable way to share the build outputs between these to avoid a complete rebuild for ~/cr2/ ?

Thanks,

Sidhin

[Nico Weber]

I haven't tried it myself, but using ccache might do what you want: https://chromium.googlesource.com/chromium/src.git/+/master/docs/ccache_mac.md

The build attempts to be independent of the current directory (https://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html), so it's at least conceivable that two different checkouts could share ccache cache hits.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/f077bff4-8a1f-47c4-8304-cfe975311eb8n%40chromium.org.

[Sidhin Thomas]

I have the ccache set up. But from the stats, the cacheable calls are far too less.  I am using "cc_wrapper="env CCACHE_SLOPPINESS=time_macros ccache" in my args.gn. Not sure if it's due to the cache size being just 5G. But it's unlikely since it's not filled yet.

~/cr/src> ccache -s

Cacheable calls:     64547 / 859950 ( 7.51%)

  Hits:              11750 /  64547 (18.20%)

    Direct:            192 /  11750 ( 1.63%)

    Preprocessed:    11558 /  11750 (98.37%)

  Misses:            52797 /  64547 (81.80%)

Uncacheable calls:  795387 / 859950 (92.49%)

Errors:                 16 / 859950 ( 0.00%)

Local storage:

  Cache size (GiB):    3.9 /    5.0 (78.14%)

  Hits:              11750 /  64547 (18.20%)

  Misses:            52797 /  64547 (81.80%)

[Nico Weber]

5G feels very low, but as I said, I don't use ccache myself. I'd look at articles for ccache hit rate debugging (https://stackoverflow.com/questions/55890684/check-if-a-ccache-call-was-a-cache-hit, https://interrupt.memfault.com/blog/ccache-debugging, etc). I'd try building in one build dir first and then the same target in a different build dir (at same rev, same args.gn) and check the hit rate of just the second build dir. If it's less than 100%, compare compile invocations, etc.

In theory, build commands should all be relative and not contain absolute paths, but maybe something in your specific setup gets in the way. Build determinism is generally a project goal though :)

Good luck, and let us know if you find anything interesting :)

[Yngve N. Pettersen]

Hello all,

At Vivaldi we have been running ccache for our Mac autobuild builds for a long time, since we only had one build worker for each CPU target on Mac.

However, after the update to 142 ccache _stopped_ working as a cache, even after updating the installations! I have not been able to figure out why (140 worked fine). (Non-official) Builds became essentially a full rebuild every time, taking 90+ minutes (on top of the line machines). On one builder (arm64) we have changed to Siso (in LERC mode), even if that take at least twice as long as ccache did _before_ 142 (issue with Mac File flush FAICT, requires fixing the golang source to mitigate). We haven't changed the intel builder yet; need to confirm that Siso works properly there first, and we aren't building the intel version that frequently.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CADZ1XiYr47Die28ith2jjeRNQtzY5q%3DKuLRwRpLc7Uhg7LrPKQ%40mail.gmail.com.

-- 
Sincerely,
Yngve N. Pettersen
Vivaldi Technologies AS

[Lei Zhang]

This is just speculation on my part, but https://crrev.com/1515831
landed in M142. Maybe that has something to do with the ccache issue?

> To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/1762979585380.3673374890.3916213158%40vivaldi.com.

[Yngve N. Pettersen]

Hi,

Looks like it is indeed clang modules that is causing the problem. The tests I ran during the night seeded the cache, then used as much time as the pre-142 builds did.

On Thursday, 13 November, 2025 09:11:18 (+01:00), Yuta Sekiguchi wrote:

Hi.

It looks ccache does not support C++20 modules yet.

https://github.com/ccache/ccache/issues/1252

———————————————————————————————————

Yuta Sekiguchi (関口雄太) | ACCESS CO., LTD.

Email: yuta.se...@access-company.com

http://www.access-company.com/

2025年11月13日木曜日 6:34:38 UTC+9 Lei Zhang:

.

[Sidhin Thomas]

Lei is spot on. Found this in the ccache log 

"[2025-11-13T08:32:02.150733 2512461] Compiler option -fmodules is unsupported without direct depend mode"

After setting CCACHE to depend mode and cache sloppiness to module. Got 100% hits - 

~/cr/src/out/Debug> gn clean .

~/cr/src/out/Debug> autoninja base_unittests

offline mode

exec_root=/home/sidhin/cr/src dir=out/Debug

 5.08s Regenerating ninja files

build finished

local:5622 remote:0 cache:0 cache-write:0(err:0) fallback:0 retry:0 skip:1592

fs: ops: 208856(err:49886) / r:40296(err:0) 4.34GiB / w:1419(err:0) 178.64MiB

41.52s Build Succeeded: 5622 steps - 135.42/s

~/cr/src/out/Debug> ccache -s

Cacheable calls:   3797 /  3797 (100.0%)

  Hits:            3797 /  3797 (100.0%)

    Direct:        3797 /  3797 (100.0%)

    Preprocessed:     0 /  3797 ( 0.00%)

  Misses:             0 /  3797 ( 0.00%)

Local storage:

  Cache size (GB):  0.6 / 200.0 ( 0.30%)

  Hits:            3797 /  3797 (100.0%)

  Misses:             0 /  3797 ( 0.00%)

This is the env variables configuration that worked - 

CCACHE_DEPEND  = 1 

CCACHE_SLOPPINESS = 'time_macros,modules'
CCACHE_BASEDIR = '<point_to_cr_src'>

This is all linux btw.

Thanks a lot Nico, Lei and Yngve.

[Junji Watanabe]

The CL we landed enables Clang modules instead of C++20 modules.

https://chromium.googlesource.com/chromium/src/+/main/docs/modules.md

There seems to be "limited support" in ccache for Clang modules with some requirements.

I'm not familiar with ccache/clang modules. But, I'd recommend setting `use_clang_modules=false` explicitly to disable clang modules for now. 

We will follow up automatically disabling clang modules when ccache is used. e.g. check cc_wrapper is set here?

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/1763025017200.974081193.2307776807%40vivaldi.com.

[Liviu Laslau]

After setting ccache to sloppyness modules and depend mode true, it does work.

However, hit rates when using a http server cache is very low if you use 3-4 computers (even macs).

Reclient also disables modules so I think any server side solution of caching also doesn't work properly for now.

[Yuta Sekiguchi]

Hi.

It looks ccache does not support C++20 modules yet.

https://github.com/ccache/ccache/issues/1252

———————————————————————————————————

Yuta Sekiguchi (関口雄太) | ACCESS CO., LTD.

Email: yuta.se...@access-company.com

http://www.access-company.com/

2025年11月13日木曜日 6:34:38 UTC+9 Lei Zhang:

This is just speculation on my part, but https://crrev.com/1515831

.

[Takuto Ikuta]

If clang modules is culprit, you can disable that by `use_clang_modules=false` in your args.gn. If that makes ccache work again, I'll disable clang modules when ccache is used by default.