[PATCH 0/6] add plugin to dump flattened config

[Felix Moessbauer]

Dear KAS users,

this series adds a new plugin 'dump' to create a flattened version of
all referenced kas files. In addition, relative refspecs like tags or
branches can be resolved and replaced by the actual revision at time
of the checkout. The output can be generated as kas-schema compatible
yaml file, or as json.

This output can also be used as a manifest, as requested in [1] or
as a basis to add support for kas in renovate [2].

Please note, that this only has been tested against git repositories
yet (but it should work with any as we don't use git specifics).
I don't have access to any mercurial repo. If you have, it would be
great to give it a try and report back.

Best regards,
Felix

[1] https://github.com/siemens/kas/issues/53
[2] https://github.com/renovatebot/renovate/issues/7877

Felix Moessbauer (6):
add plugin to dump flattened config
add test of dump plugin
add support for the dump plugin in kas-container
add revision attribute to repo class
extend dump plugin to support resolving revisions
test dump plugin support for resolved refs

docs/userguide.rst | 5 ++
kas-container | 14 +++-
kas/plugins/__init__.py | 2 +
kas/plugins/dump.py | 138 ++++++++++++++++++++++++++++++++++++++++
kas/repos.py | 9 +++
tests/test_commands.py | 30 +++++++++
6 files changed, 197 insertions(+), 1 deletion(-)
create mode 100644 kas/plugins/dump.py

--
2.30.2

[Felix Moessbauer]

This patch adds a new plugin 'dump' which resolves all kas config
includes and combines them in a single file. The format of the output
file can either be JSON or YAML. In case of YAML, the generated config
fulfills the kas-config schema and can be used as input to kas again.
In addition, the generated files can be used by external tools to
analyse the dependencies and versions of the project.

While the generated configuration is semantically identical to the
input config, we do not guarantee binary compatibility as especially
YAML provides multiple ways to serialize strings and null values.
Not giving this guarantee makes it easier to evolve and maintain the plugin.

The plugin itself extends the checkout plugin, as referenced repos have
to be checked-out first to resolve cross-repo references. This also
requires, that the declared refspec of all referenced repos can actually
be found and checked out.

Signed-off-by: Felix Jonathan Moessbauer <felix.mo...@siemens.com>
---
docs/userguide.rst | 5 ++
kas/plugins/__init__.py | 2 +
kas/plugins/dump.py | 128 ++++++++++++++++++++++++++++++++++++++++
3 files changed, 135 insertions(+)
create mode 100644 kas/plugins/dump.py

diff --git a/docs/userguide.rst b/docs/userguide.rst
index 6c058f1..81c0f2a 100644
--- a/docs/userguide.rst
+++ b/docs/userguide.rst
@@ -89,6 +89,11 @@ typically provides a single command.

.. automodule:: kas.plugins.checkout

+``dump`` plugin
+~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: kas.plugins.dump
+
``for-all-repos`` plugin
~~~~~~~~~~~~~~~~~~~~~~~~

diff --git a/kas/plugins/__init__.py b/kas/plugins/__init__.py
index 4346d05..f2b9c3f 100644
--- a/kas/plugins/__init__.py
+++ b/kas/plugins/__init__.py
@@ -43,9 +43,11 @@ def load():
from . import checkout
from . import shell
from . import menu
+ from . import dump

register_plugins(build)
register_plugins(checkout)
+ register_plugins(dump)
register_plugins(for_all_repos)
register_plugins(shell)
register_plugins(menu)
diff --git a/kas/plugins/dump.py b/kas/plugins/dump.py
new file mode 100644
index 0000000..be7beda
--- /dev/null
+++ b/kas/plugins/dump.py
@@ -0,0 +1,128 @@
+# kas - setup tool for bitbake based projects
+#
+# Copyright (c) Siemens AG, 2017-2022
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+"""
+ This plugin implements the ``kas dump`` command.
+
+ When this command is executed, kas will parse all referenced config
+ files, expand includes and print a flattened yaml version of the
+ configuration to the outfile. This config is semantically identical to the
+ input, but does not include any references to other configuration files.
+ The output of this command can be used to further analyse the build
+ configuration. As the dump command extends the checkot command, all
+ checkout options can be used.
+
+ Please note:
+
+ - the dumped config is semantically identical but not bit-by-bit identical.
+ - referenced repositories are checked out to resolve cross-repo configs.
+ - writing to stdout is not supported, as it can interfer with other output
+
+ For example, to get a single config representing the final build config of
+ ``kas-project.yml:target-override.yml`` you could run::
+
+ kas dump kas-project.yml:target-override.yml kas-project-expanded.yml
+
+ The generated config can be used as input for kas::
+
+ kas build kas-project-expanded.yml
+"""
+
+import logging
+import sys
+import json
+import yaml
+from collections import OrderedDict
+from kas.context import get_context
+from kas.plugins.checkout import Checkout
+
+__license__ = 'MIT'
+__copyright__ = 'Copyright (c) Siemens AG, 2022'
+
+
+class Dump(Checkout):
+ """
+ Implements a kas plugin that combines multiple kas configs and dumps it.
+ """
+
+ name = 'dump'
+ helpmsg = 'Expand and dump the final config.'
+
+ class KasYamlDumper(yaml.Dumper):
+ """
+ Yaml formatter (dumper) that generates output in a formatting which
+ is similar to kas example input files.
+ """
+
+ def __init__(self, stream, **kwargs):
+ super().__init__(stream, **kwargs)
+
+ def represent_data(self, data):
+ if isinstance(data, str):
+ if data.count('\n') > 0:
+ return self.represent_scalar(
+ 'tag:yaml.org,2002:str',
+ data,
+ style='|')
+ return self.represent_scalar('tag:yaml.org,2002:str', data)
+ if isinstance(data, OrderedDict):
+ return self.represent_mapping(
+ 'tag:yaml.org,2002:map',
+ data.items())
+ return super().represent_data(data)
+
+ @classmethod
+ def setup_parser(cls, parser):
+ super().setup_parser(parser)
+ parser.add_argument('--format',
+ choices=['yaml', 'json'],
+ default='yaml',
+ help='Output format')
+ parser.add_argument('--indent',
+ type=int,
+ default=4,
+ help='Line indent (number of spaces)')
+ parser.add_argument('outfile',
+ help='Output filename')
+
+ def run(self, args):
+ super().run(args)
+ ctx = get_context()
+ config_expanded = ctx.config.get_config()
+
+ # includes are already expanded, delete the key
+ if 'includes' in config_expanded['header']:
+ del config_expanded['header']['includes']
+
+ with open(args.outfile, 'w') as f:
+ if args.format == 'json':
+ json.dump(config_expanded, f, indent=args.indent)
+ elif args.format == 'yaml':
+ yaml.dump(
+ config_expanded, f,
+ indent=args.indent,
+ Dumper=self.KasYamlDumper)
+ else:
+ logging.error('invalid format %s', args.format)
+ sys.exit(1)
+
+
+__KAS_PLUGINS__ = [Dump]
--
2.30.2

[Felix Moessbauer]

This patch adds a basic test of the dump plugin. We check if the configuration
is flattened and external references are included.
In addition, we check if no other files are referenced anymore.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
tests/test_commands.py | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)

diff --git a/tests/test_commands.py b/tests/test_commands.py
index d045e3f..a45bb5c 100644
--- a/tests/test_commands.py
+++ b/tests/test_commands.py
@@ -23,6 +23,8 @@
import glob
import os
import shutil
+import json
+import yaml
from kas import kas


@@ -66,3 +68,20 @@ def test_repo_includes(changedir, tmpdir):
shutil.copytree('tests/test_repo_includes', tdir)
os.chdir(tdir)
kas.kas(['checkout', 'test.yml'])
+
+
+def test_dump(changedir, tmpdir):
+ tdir = str(tmpdir.mkdir('test_commands'))
+ shutil.rmtree(tdir, ignore_errors=True)
+ shutil.copytree('tests/test_repo_includes', tdir)
+ os.chdir(tdir)
+
+ formats = ['json', 'yaml']
+ for f in formats:
+ outfile = 'test_flat.%s' % f
+ kas.kas(('dump --format %s test.yml %s' % (f, outfile)).split())
+ assert os.path.exists('test_flat.%s' % f)
+ with open(outfile) as conff:
+ flatconf = json.load(conff) if f == 'json' else yaml.safe_load(conff)
+ assert flatconf['repos']['kas3']['refspec'] == 'master'
+ assert 'includes' not in flatconf['header']
--
2.30.2

[Felix Moessbauer]

This patch adds support to use the dump plugin in kas-container.
It requires special handling of the arguments, as the last positional
argument is the output file and not an input file to kas.
This is implemented in a way, that running without a kas input file
still works.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

kas-container | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/kas-container b/kas-container
index bc52863..62e8b3a 100755
--- a/kas-container
+++ b/kas-container
@@ -30,12 +30,14 @@ set -e
usage()
{
printf "%b" "Usage: $0 [OPTIONS] { build | checkout | shell } [KASOPTIONS] [KASFILE]\n"
+ printf "%b" " $0 [OPTIONS] dump [KASOPTIONS] [KASFILE] [OUTFILE]\n"
printf "%b" " $0 [OPTIONS] for-all-repos [KASOPTIONS] [KASFILE] COMMAND\n"
printf "%b" " $0 [OPTIONS] { clean | cleansstate | cleanall}\n"
printf "%b" " $0 [OPTIONS] menu [KCONFIG]\n"
printf "%b" "\nPositional arguments:\n"
printf "%b" "build\t\t\tCheck out repositories and build target.\n"
printf "%b" "checkout\t\tCheck out repositories but do not build.\n"
+ printf "%b" "dump\t\t\tCheck out repositories and write flat version of config to OUTFILE.\n"
printf "%b" "shell\t\t\tRun a shell in the build environment.\n"
printf "%b" "for-all-repos\t\tRun specified command in each repository.\n"
printf "%b" "clean\t\t\tClean build artifacts, keep sstate cache and " \
@@ -265,7 +267,7 @@ while [ $# -gt 0 ]; do
shift 1
break
;;
- build|checkout|for-all-repos|menu)
+ build|checkout|dump|for-all-repos|menu)
KAS_REPO_MOUNT_OPT_DEFAULT="ro"
KAS_CMD=$1
shift 1
@@ -311,6 +313,12 @@ while [ $# -gt 0 ] && [ $KAS_EXTRA_BITBAKE_ARGS -eq 0 ]; do
shift 1
;;
*)
+ # the last argument of dump is the output file
+ if [ "$KAS_CMD" = "dump" ] && [ "$#" -eq "1" ]; then
+ KAS_DUMP_OUTFILE="$1"
+ shift 1
+ continue
+ fi
KAS_FILES=
# SC2086: Double quote to prevent globbing and word splitting.
# shellcheck disable=2086
@@ -504,6 +512,10 @@ if [ "$KAS_CMD" = "for-all-repos" ]; then
set -- "$@" "${KAS_REPO_CMD}"
fi

+if [ "$KAS_CMD" = "dump" ]; then
+ set -- "$@" "${KAS_DUMP_OUTFILE}"
+fi
+
# rotate any extra bitbake args from the front to the end of the argument list
while [ $KAS_EXTRA_BITBAKE_ARGS -gt 0 ]; do
arg="$1"
--
2.30.2

[Felix Moessbauer]

This patch adds a programatical attribute 'revision' to the repo class.
This attribute contains the exact revision at the time of the checkout
of a repository. By that, we can avoid the ambiguity of refspecs
containing tags or branch names. Internally, the revision is not yet
used but just made available for future downstream users (e.g. plugins).

Note, that the revision has to be re-queried on each access, as the
Config class re-instantiates the repos for each consumer.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

kas/repos.py | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/kas/repos.py b/kas/repos.py
index 8921bb4..af0dcae 100644
--- a/kas/repos.py
+++ b/kas/repos.py
@@ -72,6 +72,15 @@ class Repo:
except ValueError:
continue
return self.url
+ elif item == 'revision':
+ if not self.refspec:
+ return None
+ (_, output) = run_cmd(self.resolve_branch_cmd(),
+ cwd=self.path, fail=False)
+ if output:
+ return output.strip()
+ return self.refspec
+
# Default behaviour
raise AttributeError

--
2.30.2

[Felix Moessbauer]

This patch adds a flag --resolve-refs to the dump plugin.
Once enabled, all relative refspecs (e.g. branch or tag names) are
resolved and replaced by their exact revision.
When re-running kas on the flattened and expanded config file, the build
is executed against exactly the versions of the dependencies that where
used when dumping the configuration. This helps to keep track of build
versions for future use (e.g. reproducible builds) or for
version-tracking tools like renovate.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

kas/plugins/dump.py | 10 ++++++++++
1 file changed, 10 insertions(+)

diff --git a/kas/plugins/dump.py b/kas/plugins/dump.py
index be7beda..d443fa7 100644
--- a/kas/plugins/dump.py
+++ b/kas/plugins/dump.py
@@ -100,6 +100,9 @@ class Dump(Checkout):
type=int,
default=4,

help='Line indent (number of spaces)')

+ parser.add_argument('--resolve-refs',
+ action='store_true',
+ help='Replace floating refs with exact SHAs')
parser.add_argument('outfile',
help='Output filename')

@@ -112,6 +115,13 @@ class Dump(Checkout):

if 'includes' in config_expanded['header']:

del config_expanded['header']['includes']

+ if args.resolve_refs:
+ repos = ctx.config.get_repos()
+ for r in repos:
+ if r.refspec:
+ config_expanded['repos'][r.name]['refspec'] = r.revision
+ pass

+
with open(args.outfile, 'w') as f:

if args.format == 'json':
json.dump(config_expanded, f, indent=args.indent)
--
2.30.2

[Felix Moessbauer]

This patch extends the test of the dump plugin.
In addition, we test if relative refspecs are expanded and if the
generated output can be used as input to kas again.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

tests/test_commands.py | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/tests/test_commands.py b/tests/test_commands.py
index a45bb5c..47c6315 100644
--- a/tests/test_commands.py
+++ b/tests/test_commands.py
@@ -77,11 +77,22 @@ def test_dump(changedir, tmpdir):
os.chdir(tdir)

formats = ['json', 'yaml']

- for f in formats:
+ resolve = ['', '--resolve-refs']
+ # test cross-product of these options (formats x resolve)
+ for f, r in ((f, r) for f in formats for r in resolve):

outfile = 'test_flat.%s' % f

- kas.kas(('dump --format %s test.yml %s' % (f, outfile)).split())
+ kas.kas(('dump --format %s %s test.yml %s' % (f, r, outfile)).split())

assert os.path.exists('test_flat.%s' % f)

- with open(outfile) as conff:
- flatconf = json.load(conff) if f == 'json' else yaml.safe_load(conff)
- assert flatconf['repos']['kas3']['refspec'] == 'master'
+ with open(outfile) as cf:
+ flatconf = json.load(cf) if f == 'json' else yaml.safe_load(cf)
+ refspec = flatconf['repos']['kas3']['refspec']
+ if r == '':
+ assert refspec == 'master'
+ else:
+ assert refspec != 'master'

assert 'includes' not in flatconf['header']

+ # check if kas can read the generated file
+ if f == 'yaml':
+ shutil.rmtree('%s/build' % tdir, ignore_errors=True)
+ kas.kas(('checkout %s' % outfile).split())
+ assert os.path.exists('build/conf/local.conf')
--
2.30.2

[Jan Kiszka]

Looks good to me so far, but I would love to see try-out reports from
folks who thought about that, asked for it already.

Regarding the patching topic: We should clarify, maybe just in the
documentation, that layer patches modifying refspecs are not compatible
with this approach - at least this is my current understanding. If
layer-repo A carries a patch to layer-repo B which modifies in a kas
file in B which revision of layer-repo C should be checked out, this
cannot be flattened by kas dump, can it?

Jan

--
Siemens AG, Technology
Competence Center Embedded Linux