[PATCH] kas: new plugin fetch-all-repos

[Frank Bielig]

A new plugin has been added for easier maintenance of the repository
mirrors. With

$ kas fetch-all-repositories kas-project.yml

all repositories used by kas-project.yml are cloned or updated in the
repository reference directory. The definition of the environmental variable
`KAS_REPO_REF_DIR` is mandatory.

Signed-off-by: Frank Bielig <frank....@mbition.io>
Signed-off-by: Tobias Hunger <tobias...@mbition.io>
---
CHANGELOG.md | 5 ++
docs/userguide.rst | 5 ++
kas-container | 3 +-
kas/plugins/__init__.py | 2 +
kas/plugins/fetch_all_repos.py | 91 ++++++++++++++++++++++++++++++++++
kas/repos.py | 3 ++
tests/test_commands.py | 50 ++++++++++++++-----
7 files changed, 145 insertions(+), 14 deletions(-)
create mode 100644 kas/plugins/fetch_all_repos.py

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 637c2fb..8b4c918 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,9 @@
+2.5-mbition
+
+- kas: add support for fetch-all-repos
+
2.5
+
- kas: Apply patches before doing an environment setup
- kas: repos: strip dot from layer name
- kas: Introduce KAS_BUILD_DIR environment variable
diff --git a/docs/userguide.rst b/docs/userguide.rst
index e996245..9ddf3ca 100644
--- a/docs/userguide.rst
+++ b/docs/userguide.rst
@@ -86,6 +86,11 @@ typically provides a single command.

.. automodule:: kas.plugins.for_all_repos

+``fetch-all-repos`` plugin
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: kas.plugins.fetch_all_repos
+
``shell`` plugin
~~~~~~~~~~~~~~~~

diff --git a/kas-container b/kas-container
index 787f65b..6d0a62a 100755
--- a/kas-container
+++ b/kas-container
@@ -31,6 +31,7 @@ usage()
{
printf "%b" "Usage: $0 [OPTIONS] { build | checkout | shell } [KASOPTIONS] KASFILE\n"
printf "%b" " $0 [OPTIONS] for-all-repos KASFILE COMMAND\n"
+ printf "%b" " $0 [OPTIONS] fetch-all-repos KASFILE\n"
printf "%b" " $0 [OPTIONS] clean\n"
printf "%b" "\nPositional arguments:\n"
printf "%b" "build\t\t\tCheck out repositories and build target.\n"
@@ -220,7 +221,7 @@ while [ $# -gt 0 ]; do
shift 1
break
;;
- build|checkout|for-all-repos)
+ build|checkout|for-all-repos|fetch-all-repos)
KAS_REPO_MOUNT_OPT_DEFAULT="ro"
KAS_CMD=$1
shift 1
diff --git a/kas/plugins/__init__.py b/kas/plugins/__init__.py
index bbdb072..936131b 100644
--- a/kas/plugins/__init__.py
+++ b/kas/plugins/__init__.py
@@ -40,12 +40,14 @@ def load():
"""
from . import build
from . import for_all_repos
+ from . import fetch_all_repos
from . import checkout
from . import shell

register_plugins(build)
register_plugins(checkout)
register_plugins(for_all_repos)
+ register_plugins(fetch_all_repos)
register_plugins(shell)


diff --git a/kas/plugins/fetch_all_repos.py b/kas/plugins/fetch_all_repos.py
new file mode 100644
index 0000000..84545da
--- /dev/null
+++ b/kas/plugins/fetch_all_repos.py
@@ -0,0 +1,91 @@
+# kas - setup tool for bitbake based projects
+#
+# Copyright (c) Frank Bielig, 2021
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+"""
+ This plugin implements the ``kas fetch-all-repos`` command.
+
+ When this command is executed, kas will fetch all repositories listed
+ in the chosen config file in the repository reference directory
+ defined by environment variable ``KAS_REPO_REF_DIR``.
+"""
+
+import os
+import pathlib
+import subprocess
+
+from kas.config import Config
+from kas.context import create_global_context
+from kas.libcmds import Macro
+
+__license__ = 'MIT'
+__copyright__ = 'Copyright (c) Frank Bielig, 2021'
+
+
+class FetchAllRepos:
+ name = 'fetch-all-repos'
+ helpmsg = (
+ 'Clone or update all referenced repositories.'
+ )
+
+ @classmethod
+ def setup_parser(cls, parser):
+ pass
+
+ def run(self, args):
+ ctx = create_global_context(args)
+ ctx.config = Config(args.config)
+
+ macro = Macro()
+ macro.add(FetchAllReposCommand())
+ macro.run(ctx, args.skip)
+
+
+class FetchAllReposCommand:
+ def __init__(self):
+ super().__init__()
+
+ def __str__(self):
+ return 'fetch-all-repos'
+
+ def execute(self, ctx):
+ if not ctx.kas_repo_ref_dir:
+ raise AssertionError("KAS_REPO_REF_DIR needs to be set")
+
+ ref_dir = pathlib.Path(ctx.kas_repo_ref_dir)
+
+ checked = set()
+ for repo in ctx.config.get_repos():
+ if repo.qualified_name in checked:
+ continue
+ if repo.is_local:
+ continue
+ repo_dir = os.path.join(ref_dir, repo.qualified_name)
+ if os.path.exists(repo_dir):
+ print("Updating %s ..." % repo.url)
+ subprocess.run(["git", "-C", repo_dir, "fetch"], check=True)
+ else:
+ print("Cloning %s ..." % repo.url)
+ subprocess.run(["git", "clone", "--bare", repo.url, repo_dir],
+ check=True)
+ checked.update({repo.qualified_name})
+
+
+__KAS_PLUGINS__ = [FetchAllRepos]
diff --git a/kas/repos.py b/kas/repos.py
index edd46ee..3e5f697 100644
--- a/kas/repos.py
+++ b/kas/repos.py
@@ -54,6 +54,9 @@ class Repo:
if item == 'layers':
return [os.path.join(self.path, layer).rstrip(os.sep + '.')
for layer in self._layers]
+ elif item == 'is_local':
+ url = urlparse(self.url)
+ return (url.netloc or 'file') == 'file'
elif item == 'qualified_name':
url = urlparse(self.url)
return ('{url.netloc}{url.path}'
diff --git a/tests/test_commands.py b/tests/test_commands.py
index 9762025..7b11e91 100644
--- a/tests/test_commands.py
+++ b/tests/test_commands.py
@@ -23,30 +23,54 @@
import glob
import os
import shutil
+
+import pytest
+
from kas import kas


def test_for_all_repos(changedir, tmpdir):
- tdir = str(tmpdir.mkdir('test_commands'))
- shutil.rmtree(tdir, ignore_errors=True)
- shutil.copytree('tests/test_commands', tdir)
- os.chdir(tdir)
+ test_dir = str(tmpdir.mkdir('test_commands'))
+ shutil.rmtree(test_dir, ignore_errors=True)
+ shutil.copytree('tests/test_commands', test_dir)
+ os.chdir(test_dir)
kas.kas(['for-all-repos', 'test.yml',
- 'git rev-parse HEAD >> %s/ref_${KAS_REPO_NAME}' % (tdir)])
+ 'git rev-parse HEAD >> %s/ref_${KAS_REPO_NAME}' % test_dir])

with open('ref_kas_1.0', 'r') as f:
- assert(f.readline().strip()
- == '907816a5c4094b59a36aec12226e71c461c05b77')
+ assert (f.readline().strip()
+ == '907816a5c4094b59a36aec12226e71c461c05b77')
with open('ref_kas_1.1', 'r') as f:
- assert(f.readline().strip()
- == 'e9ca55a239caa1a2098e1d48773a29ea53c6cab2')
+ assert (f.readline().strip()
+ == 'e9ca55a239caa1a2098e1d48773a29ea53c6cab2')
+
+
+def test_fetch_all_repos(capsys, tmpdir):
+ test_dir = str(tmpdir.mkdir('test_commands'))
+ shutil.rmtree(test_dir, ignore_errors=True)
+ shutil.copytree('tests/test_commands', test_dir)
+
+ os.chdir(test_dir)
+ with pytest.raises(AssertionError):
+ kas.kas(['fetch-all-repos', 'test.yml'])
+
+ ref_dir = str(tmpdir.mkdir('repo_ref'))
+ os.environ['KAS_REPO_REF_DIR'] = ref_dir
+ kas.kas(['fetch-all-repos', 'test.yml'])
+ captured = capsys.readouterr()
+ assert(os.path.exists(os.path.join(ref_dir, 'github.com.siemens.kas.git')))
+ assert(captured.out.startswith("Cloning"))
+
+ kas.kas(['fetch-all-repos', 'test.yml'])
+ captured = capsys.readouterr()
+ assert(captured.out.startswith("Updating"))


def test_checkout(changedir, tmpdir):
- tdir = str(tmpdir.mkdir('test_commands'))
- shutil.rmtree(tdir, ignore_errors=True)
- shutil.copytree('tests/test_commands', tdir)
- os.chdir(tdir)
+ test_dir = str(tmpdir.mkdir('test_commands'))
+ shutil.rmtree(test_dir, ignore_errors=True)
+ shutil.copytree('tests/test_commands', test_dir)
+ os.chdir(test_dir)
kas.kas(['checkout', 'test.yml'])

# Ensure that local.conf and bblayers.conf are populated, check that no
--
2.31.1

[Jan Kiszka]

On 02.06.21 21:50, 'Frank Bielig' via kas-devel wrote:
> A new plugin has been added for easier maintenance of the repository
> mirrors. With
>
> $ kas fetch-all-repositories kas-project.yml
>
> all repositories used by kas-project.yml are cloned or updated in the
> repository reference directory. The definition of the environmental variable
> `KAS_REPO_REF_DIR` is mandatory.
>

First of all, thanks for your contribution! Just please provide the use
case in more details. Specifically explain why "kas checkout" is not
what you need and can't be enhanced to achieve that. From the distance,
a user will very likely wonder which of the two to use when.

> Signed-off-by: Frank Bielig <frank....@mbition.io>
> Signed-off-by: Tobias Hunger <tobias...@mbition.io>

Co-developed? Or signed for formal OSS release?

> ---
> CHANGELOG.md | 5 ++
> docs/userguide.rst | 5 ++
> kas-container | 3 +-
> kas/plugins/__init__.py | 2 +
> kas/plugins/fetch_all_repos.py | 91 ++++++++++++++++++++++++++++++++++
> kas/repos.py | 3 ++
> tests/test_commands.py | 50 ++++++++++++++-----
> 7 files changed, 145 insertions(+), 14 deletions(-)
> create mode 100644 kas/plugins/fetch_all_repos.py
>
> diff --git a/CHANGELOG.md b/CHANGELOG.md
> index 637c2fb..8b4c918 100644
> --- a/CHANGELOG.md
> +++ b/CHANGELOG.md
> @@ -1,4 +1,9 @@
> +2.5-mbition

That's likely from our own fork. ;)

> +
> +- kas: add support for fetch-all-repos
> +
> 2.5
> +

No need to touch CHANGELOG. I'll do that on the next release.

Jan

Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

[Henning Schild]

Am Wed, 2 Jun 2021 21:50:56 +0200
schrieb "'Frank Bielig' via kas-devel" <kas-...@googlegroups.com>:

What about "disable_operations" ? That is set when adding a repo
without an url. Maybe the same thing, if not the plugin should consider
"disable_operations".

> + repo_dir = os.path.join(ref_dir, repo.qualified_name)
> + if os.path.exists(repo_dir):
> + print("Updating %s ..." % repo.url)

This is missing hg support, not even checking the "type" to skip hg in
case it should be ignored.

> + subprocess.run(["git", "-C", repo_dir, "fetch"],
> check=True)

repo.fetch_cmd()

> + else:
> + print("Cloning %s ..." % repo.url)
> + subprocess.run(["git", "clone", "--bare", repo.url,
> repo_dir],
> + check=True)

repo.clone_cmd()

And that should be done async like repos.py does things in fetch_async.

Seems like all this should be based on repos.py instead of reinventing
things and forgetting cases on the way.

Henning

[Frank Bielig]

Am 6/3/21 um 8:56 AM schrieb Jan Kiszka:

> On 02.06.21 21:50, 'Frank Bielig' via kas-devel wrote:
>> A new plugin has been added for easier maintenance of the repository
>> mirrors. With
>>
>> $ kas fetch-all-repositories kas-project.yml
>>
>> all repositories used by kas-project.yml are cloned or updated in the
>> repository reference directory. The definition of the environmental variable
>> `KAS_REPO_REF_DIR` is mandatory.
>>
>
> First of all, thanks for your contribution! Just please provide the use
> case in more details. Specifically explain why "kas checkout" is not
> what you need and can't be enhanced to achieve that. From the distance,
> a user will very likely wonder which of the two to use when.
>

The use-case is in the context of CIs, when several developers use the
same or similar base. In this case, bootstrapping should become faster
by providing the repositories locally on the one hand, and on the other
hand also relieve the servers.

In the Yocto project for Arm you can already find such a solution
integrated into the GitLab CI:

https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/tree/ci/update-repos

I would like to avoid such additional scripts. Kas has from my point of
view exactly the goal of summarizing all relevant information in one
place. This script contradicts this approach. In addition, there is
duplication in the code.

>> Signed-off-by: Frank Bielig <frank....@mbition.io>
>> Signed-off-by: Tobias Hunger <tobias...@mbition.io>
>
> Co-developed? Or signed for formal OSS release?
>

Tobias signed it internally.

>> ---
>> CHANGELOG.md | 5 ++
>> docs/userguide.rst | 5 ++
>> kas-container | 3 +-
>> kas/plugins/__init__.py | 2 +
>> kas/plugins/fetch_all_repos.py | 91 ++++++++++++++++++++++++++++++++++
>> kas/repos.py | 3 ++
>> tests/test_commands.py | 50 ++++++++++++++-----
>> 7 files changed, 145 insertions(+), 14 deletions(-)
>> create mode 100644 kas/plugins/fetch_all_repos.py
>>
>> diff --git a/CHANGELOG.md b/CHANGELOG.md
>> index 637c2fb..8b4c918 100644
>> --- a/CHANGELOG.md
>> +++ b/CHANGELOG.md
>> @@ -1,4 +1,9 @@
>> +2.5-mbition
>
> That's likely from our own fork. ;)
>
>> +
>> +- kas: add support for fetch-all-repos
>> +
>> 2.5
>> +
>
> No need to touch CHANGELOG. I'll do that on the next release.
>

Got it. I haven't seen the release script.

Frank Bielig | frank....@daimler.com | +49 (176) 30969212 | System
Architect | MBition GmbH | Dovestraße 1, 10587 Berlin, Germany
Geschäftsführer/Board of Directors: Gregor Zetsche,
Registergericht/Court of Registry: Amtsgericht Charlottenburg
HRB-Nr./Commercial Register No.: 188241 B

[Jan Kiszka]

OK, so the use case is filling the REPO_REF_DIR without building -
understood. But, again, when prevents using "kas checkout" for this purpose?

Jan

[Frank Bielig]

Am 6/3/21 um 2:36 PM schrieb Jan Kiszka:

The call of

$ KAS_WORK_DIR=$(pwd)/work KAS_REPO_REF_DIR=$(pwd)/ref kas checkout
kas-config.yml

clones the repos in working directory but repository reference directory
is still empty. Obviously I have to debug or do you see already any issue?

Frank

[Frank Bielig]

Am 6/3/21 um 1:22 PM schrieb Henning Schild:

Cloning of local repositories into reference directory isn't really
needed. With the new attribute 'is_local' I want to handle the missing
url configuration (leads to 'disable_operations') and URLs of with
scheme 'file' at the same time.

>> + repo_dir = os.path.join(ref_dir, repo.qualified_name)
>> + if os.path.exists(repo_dir):
>> + print("Updating %s ..." % repo.url)
>
> This is missing hg support, not even checking the "type" to skip hg in
> case it should be ignored.
>

Agree, needs to be adjusted.

>> + subprocess.run(["git", "-C", repo_dir, "fetch"],
>> check=True)
>
> repo.fetch_cmd()
>
>> + else:
>> + print("Cloning %s ..." % repo.url)
>> + subprocess.run(["git", "clone", "--bare", repo.url,
>> repo_dir],
>> + check=True)
>
> repo.clone_cmd()
>
> And that should be done async like repos.py does things in fetch_async.

Agree. Could be one optimization.

> Seems like all this should be based on repos.py instead of reinventing
> things and forgetting cases on the way.
>
> Henning

Probably I found the real issue and might need your support here. If
this works as expected, then `kas checkout kas.yml` could fulfill the
use case I want to support.

repos.py:174
if not os.path.exists(self.path):
os.makedirs(os.path.dirname(self.path), exist_ok=True)
sdir = os.path.join(get_context().kas_repo_ref_dir or '',
self.qualified_name)
logging.debug('Looking for repo ref dir in %s', sdir)

(retc, _) = await run_cmd_async(
---> self.clone_cmd(sdir),
cwd=get_context().kas_work_dir)
if retc == 0:
logging.info('Repository %s cloned', self.name)
return retc

The clone command always checks out the given repository into
'self.path'. But it should happen in 'sdir' or am I wrong?

Additionally I'd like to have a mirror of remote repositories in
reference directory to support multiple branches out of the box
('git clone --bare' or even 'git clone --mirror').

If you give me some advices for your desired solution, I can take care
of it and provide a new patch.

[Jan Kiszka]

OK, the problem is that KAS_REPO_REF_DIR is currently supposed to be
filled manually upfront:

| ``KAS_REPO_REF_DIR`` | The path to the repository reference directory. |
| | Repositories in this directory are used as |
| | references when cloning. In order for kas to find |
| | those repositories, they have to be named in a |
| | specific way. The repo URLs are translated like |
| | this: "https://github.com/siemens/meta-iot2000.git" |
| | resolves to the name |
| | "github.com.siemens.meta-iot2000.git". |

You are looking for a way to do that with the help of kas ("fill-repo-
ref-dir").

I suppose the current mode was implemented as a simple solution, and
also to avoid having to deal with concurrency etc. Still, adding such a
creation feature sounds reasonable to me.

Interface-wise, this could either be a separate plugin, like you
suggested, just with better naming. That would allow to offload the
synchronization task to the user. Or we actually implement auto-filling/
updating on repo checkout (like I thought we have), though that will be
a bit more complicated but possibly also more user-friendly.

[Frank Bielig]

Am 6/4/21 um 9:00 AM schrieb Jan Kiszka:

I'm fine with both solutions. In the end I want to avoid several
solutions on user side for the same topic. Especially for using kas in
CIs this topic is quite important.

I decided for the most transparent solution in the beginning. Surely the
name can be improved and the suggestions from Henning should be
incorporated.

On the other hand I found a possible issue in the current core. It looks
like everything was prepared for filling the repo ref dir, but it
doesn't work correcly. Please have a look into the other mail thread.
Fixing it could be the solution for the second option you proposed.

[Jan Kiszka]

I don't think it was designed to be self-filling. Here is the commit log
from Claudius when the feature was added in 2017 (prio to the OSS release):

repos: Add option to use repositories as reference and global kas state
config

With this commits its possible to declare a directory as containing
reference repositories via the "KAS_REPO_REF_DIR" environment variable.

In order for kas to find those repositories, they have to be named

correctly. Those names are derived from the repo url in the kas config.
(E.g. url: "https://github.com/siemens/meta-iot2000.git" resolves to the
name "github.com.siemens.meta-iot2000.git")


And since then, nothing was done to change that as far as I can see.
Again: we would need concurrency management in case of writing to a
potentially shared reference dir, just like bitbake implements for its
download cache dir.

[Claudius Heine]

IIRC, when I implemented this, I thought of the repo ref dir as a path,
that is shared between different CI runs.

If one CI run has write access to the repo ref dir, it could manipulate
it and possible affect future CI runs, so I assumed that they would be
read-only and opted for letting the CI server operator do the updating
of those.

Maybe my concerns are baseless, but I rather wanted to err on the side
of caution and conservatism and this decision can of course be revisited.

regards,
Claudius