Unnecessary restriction on config file paths

[Paul Barker]

Hi folks,

I've again been bitten by an unnecessary restriction on how kas can be used. I appreciate that the intention here is consistency between builds but at the end of the day a tool needs to trust the developer using it. Preventing actions because they may in some use cases be harmful will just result in people bypassing kas when they need to, which actually works against the intention of consistency.

In this case I have a config file which sets up a local sstate & downloads cache, stored in /mnt/cache/yocto/cache.yml so that it can be shared between projects. However, when I try to use this I get an error:

pbarker@owl:~/Projects/Sancloud/meta-sancloud> kas build kas/bbe-poky.yml:kas/inc/ci.yml:kas/inc/debug.yml:/mnt/cache/yocto/cache.yml
2021-09-23 10:48:36 - INFO - kas 2.5 started
2021-09-23 10:48:36 - INFO - /home/pbarker/Projects/Sancloud/meta-sancloud/kas$ git rev-parse --show-toplevel
2021-09-23 10:48:36 - INFO - /home/pbarker/Projects/Sancloud/meta-sancloud/kas/inc$ git rev-parse --show-toplevel
2021-09-23 10:48:36 - INFO - /home/pbarker/Projects/Sancloud/meta-sancloud/kas/inc$ git rev-parse --show-toplevel
2021-09-23 10:48:36 - INFO - /mnt/cache/yocto$ git rev-parse --show-toplevel
2021-09-23 10:48:36 - INFO - /mnt/cache/yocto$ hg root
2021-09-23 10:48:37 - ERROR - All concatenated config files must belong to the same repository or all must be outside of versioning control
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/kas/kas.py", line 180, in main
sys.exit(kas(sys.argv[1:]))
File "/usr/lib/python3.6/site-packages/kas/kas.py", line 169, in kas
plugin.run(args)
File "/usr/lib/python3.6/site-packages/kas/plugins/build.py", line 79, in run
ctx.config = Config(args.config, args.target, args.task)
File "/usr/lib/python3.6/site-packages/kas/config.py", line 49, in __init__
raise IncludeException('All concatenated config files must '
kas.includehandler.IncludeException: All concatenated config files must belong to the same repository or all must be outside of versioning control.

Removing the check and exception results in the local.conf file being written correctly and the build proceeds as expected. So this check is not preventing any actual error, it's just enforcing the intended use case.

If I submit a patch to remove this check, or turn it into a warning which can be disabled, would that be likely to be accepted?

Thanks,

--
Paul Barker
https://pbarker.dev/

[Henning Schild]

Am Thu, 23 Sep 2021 11:06:18 +0100
schrieb "Paul Barker" <pa...@pbarker.dev>:

In general kas aims to produce reproducible builds, and that is
probably one fact why people like that tool. Hence also such seemingly
weird checks.

Maybe we can have a key in such xmls "extern = true".

But do you really need to have that external? We are using shared
sstate together with kas on isar based projects. If SSTATE_DIR is in
env it will win, and kas plus kas-container are aware of that variable.

Meaning on that machine of yours you have SSTATE_DIR=/mnt/cache/yocto/
in your global env and kas will respect that, native or in container.
If it is more than just setting SSTATE_DIR and if it is project
specific, maybe drop that yml into the repo after all.

Henning

> Thanks,
>

[Paul Barker]

Hi Henning,

The config file in question sets SSTATE_DIR based on the distro and version, and allows overriding to put the cache for particular projects in to a "silo" if needed (used mainly for testing/debugging any sstate issues):

include silo.conf

SSTATE_SILO ?= "${DISTRO}"
SSTATE_SILO_VERSION ?= "${DISTRO_CODENAME}"
DISTRO_CODENAME ??= "unknown"

CACHE_ROOT ?= "/mnt/cache/yocto"
DL_DIR = "${CACHE_ROOT}/downloads"
SSTATE_DIR = "${CACHE_ROOT}/sstate-${SSTATE_SILO}/${SSTATE_SILO_VERSION}"

So this isn't something that can just be set in the environment, it needs to be set within bitbake so it has access to DISTRO and DISTRO_CODENAME.

The config file is project independent. Copying or linking it into each project will just result in needing to mess with .gitignore as well, potentially in repositories where we can't push that change upstream.

[Henning Schild]

Am Thu, 23 Sep 2021 11:50:30 +0100

This indeed looks more involved but also like a valid use-case that kas
should support.

As i proposed you can maybe send a patch that allows marking yml bits
as external. I guess for most people the pedantic checking will still
have some value to prevent mistakes.

But i think you can also get away without changing kas. Just place
those files into a "repo" that will be hosted locally. That would also
allow you to have configuration management based on branches or shas
... say some day you will want to version that cache config where some
projects will need other values than others.

Or maybe something like this will work:

repo: cache
path: /mnt/cache/

Henning

[Jan Kiszka]

Nope, this would break kas-container support.

When building inside a container, kas has to know which directories need
to be mounted into that world and which paths need to be adjusted to
make things work there. If you want to model something new, you need to
check against that scenario as well and propose a solution for it,
ideally something not increasing the complexity of kas-container even
further.

Jan