Yocto Clean Download Cache ##BEST##
Sharmaine Kachmar
Start with a clean cache by "rm tmp/cache -rf". You can then trigger bitbake's profiling mode with the "-P" option, so "bitbake -p -P". Bitbake will parse the recipes, then print a message saying "Processed parsing statistics saved to profile-parse.log.processed".
A powerful feature of the Yocto build system is the shared state cache, usually called sstate cache for brevity. After having built a recipe successfully, bitbake stores the output results (without all the intermediate files) into the sstate cache. When the exact same recipe needs to be built again, instead of running through the expensive tasks of the actual build, bitbake will simply extract the resulting binaries it had previously stored.
The sstate cache saves a huge amount of time when building big projects. It also saves disk space, as it does not need to extract all the source code and to produce all the intermediate artifacts: only the final package is extracted.
However, just like the download directory, also the directory storing the sstate cache is by default relative to the build directory, being in $TOPDIR/sstate-cache. No surprise that it can be modified, just like the downloads directory, by modifying the SSTATE_DIR variable, for example:
If you have multiple directories building a similar distribution (e.g. two different branches of the same projects), this can reuse the output products for most of your recipes. But it is useful even if you have projects involving totally different target CPU architectures, such as an ARM project and a RISC-V project, because native packages are also saved into the sstate cache. And even in case nothing can be shared, it will not give any noticeable performance penalty to your builds.
Bitbake implements the mechanisms needed to make this safe. For the sstate especially, each cache entry is saved along with a hash, which is computed from all the variables and functions that are in the recipe. If you have ever run bitbake -e , you have an idea of what those variables and functions are. This means that if you rebuild the same recipe with even slightly different settings (source code version, applied patches, compiler optimization flags, permissions of installed files, whatever) the hash value will change, and a different sstate cache entry will be created.
Curious about the time saving you can get? You can easily discover that building from scratch an entire project that takes an hour at the first build, it will easily take less than a minute when rebuilding from scratch using a pre-populated sstate cache!
The yocto autobuilders share their sstate and do a huge amount of builds, running on many builders having several different distributions, and building dunfell, kirkstone, mickledore, nanbield and master, all in parallel. This is definitely largely tested!
Note that the stamps and sstate cache are still enabled. There are some subtle details about making sure BitBake is aware that BUILD_ID is indirectly referenced by the do_compile task so that it gets included in the task hash (see how OS_RELEASE_FIELDS is used in os-release.bb). That ensures that the task hash changes whenever the SHA1 of the OEROOT git repo HEAD changes which means the caches will be invalidated then as well.
There are no shortcuts with Yocto. You need a massive machine or a server farm to get clean build times down to a few hours. If you are using a quad-core i-7, do not be surprised if your build takes over 24 hours.
Please note that I am not saying that I never got frustrated by Yocto having to get back the ashes and perform a clean build. This would not be true at all. I have already made mistakes that were faster to solve by deleting all the cache and starting from zero. What I am saying is that this event is not the rule for me. Actually, it is far away from being the rule.
SSD has a much slower write speed than a good (WD Black) spinning disk. Yocto from scratch and other large build which have north of 10K temporary object files and include the same standard headers (think stdlib.h and such) thousands of times pop past the end of that cache almost instantly.
The OpenEmbedded(OE) build system produces a lot of intermediate output when processing the many tasks entailed in building the packages comprising the Linux OS stack. SSTATE(Shared State) CACHE provides a cache mechanism which drastically reduces build time, especially when you create a new PetaLinux project and run "petalinux-build --sdk". The OE build system detects changes in the "inputs" to a given task by creating a checksum (or signature) of the task's inputs. If the checksum changes, the system assumes the inputs have changed and the task needs to be rerun. The intermediate output can be reused for future builds and shared between multiple build environments to speed up the build process.
Two key variables are used with the shared state cache feature. The SSTATE_DIR variable contains the path to the shared state cache. The variable DL_DIR tells BitBake where to place the source downloads. The default setting places the files in the directory downloads beneath the top directory(TOPDIR) of your build environment. The variable TOPDIR contains the full (absolute) path to the build environment. Source downloads can be shared among multiple build environments.
Hi! Our company has an extra large C++ embedded project built with Bitbake on Jenkins. Also, we are using ccache mounted from NFS to speed up build. When nothing changes in source code and all entries are found in ccache, both build-wrapper dump and CMake compile commands are empty and no build commands can be captured. Clean build without ccache takes up to 12 hours! We want to run Sonarqube scanner on each periodic build and for PR. Waiting 12 hours for PR is too long)) What can you suggest to generate build wrapper output or compile commands each time?
Thank you!
Hi, Colin.
I saw this answer. It says about support of CCache during scan. Unfortunately, there is no described case when CCache is fully cover all files (no changes in source code, all entries are cached). In this case, compiler (may be) not executes any commands, so build wrapper and cmake compile_commands does not capture any build commands and they both are empty.
Full clean build will take up to 12 hours and there is no bitbake/cmake flag to execute compile command anyway if cache exists.
Could you, please improve build_wrapper functionality to capture compilation commands anyway if all cache is valid?
For build-wrapper, also it should work even if Ccache is 100% hit. What version of SonarQube are you using? what version of build-wrapper? Did you have any run with build-wrapper not empty? the empty build-wrapper might not be related to Ccache.
Hi, Abbas,
Could you, please, provide command or tell me how to trigger generation step in CMake without build?
About build wrapper, issue was on sonarqube 9.4 and build-wrapper downloaded from sonarqube api. We have non empty build-wrapper output when cache hit is not 100%. Unfortunately, we can not reproduce this issue, because our sonarqube developer trial license expired and this issue blocks us from buying new one.
Thank you!
Once you get your first "long" i.e. from scratch Yocto build to complete, you can reuse the downloads and sstate-cache folders that it generated to speed up future builds (sometimes called "incremental" or "short" builds). You can copy downloads and sstate-cache to another location (e.g. or /mnt/resource), delete the var-fslc-yocto folder, and start over but append e.g. the following lines to build/conf/local.conf:
rm_work drastically reduces the storage space used while downloads and sstate-cache drastically reduce the build time, usually to under 30 minutes i.e. something that is acceptable for continuous integration. More CPU cores reduce the build time further.
The downloads and especially sstate-cache folders tend to slowly grow over time and will eventually exhaust the space on the virtual machine. In this case, builds will fail and human intervention is required to take the machine down for maintenance, delete the downloads and/or sstate-cache folder(s), and run a yocto build from scratch to recreate downloads and/or sstate-cache.
continuous integration: the process of automatically running compilers, static code analysis, style checkers, and other tools on a build server each time a developer pushes changes to ensure code quality.
downloads: Yocto's folder for storing source code downloaded from remote repositories. Reusing downloads speeds up the build and prevents the build server from having to make network requests to fetch source code.
manifest: a small repository containing a single file called default.xml, which is the input to Google's repo tool.
meta: a Yocto "layer" containing recipes (.bb files).
repo: Google's command line tool for managing multiple invocations of "git clone", using default.xml as input. Most Yocto builds use repo to download the meta repositories necessary to bootstrap the build.
rm_work: a space-saving Yocto directive in local.conf that instructs Yocto to delete intermediate files in the build/tmp/work folder after they are no longer needed.
sed: a standard Unix/Linux command line tool that has been around for decades and is extremely useful for putting Yocto builds under continuous integration by performing simple text replacements.
sstate-cache: Yocto's folder for storing a cache to speed up the build process. Reusing sstate-cache drastically speeds up the build.
Yocto: The Yocto Project is a system for building a custom Linux-based operating system from thousands of source code repositories.
After I have removed an application that installed some shared libraries, does it know to also remove the information from the ld.so.cache? If I use ldconfig will it remove the information? How does it actually work?
8d45195817