[PATCH 1/1] repos: Add KAS_GIT_SHALLOW to implement git shallow clone/fetch

[Felix Moessbauer]

From: Marek Vasut <ma...@denx.de>

Add new environment variable KAS_GIT_SHALLOW which adds '--depth=N'
to the 'git clone' and 'git fetch' commands. This forces git to
perform shallow clone, which saves bandwidth and CI runner disk
space. The depth 'N' is derived from KAS_GIT_SHALLOW value.

This is useful in case CI always starts with empty work directory
and this directory is always discarded after the CI run. In that
case, it makes no sense to clone the entire repository, instead
clone just enough to reproduce the desired state of the repository
and assemble the checkout of it.

This is also useful when cloning massive repositories which would
otherwise take long time to clone.

[Felix: rebased, sanitize input values, adapted doc entry to new
format, port test over to monkeykas infrastructure, forward from
kas-container]

Signed-off-by: Marek Vasut <ma...@denx.de>
Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
docs/command-line/environment-variables.inc | 6 ++++++
kas-container | 2 +-
kas/repos.py | 12 ++++++++++++
tests/conftest.py | 1 +
tests/test_commands.py | 21 ++++++++++++++++++++-
tests/test_commands/test-shallow.yml | 9 +++++++++
6 files changed, 49 insertions(+), 2 deletions(-)
create mode 100644 tests/test_commands/test-shallow.yml

diff --git a/docs/command-line/environment-variables.inc b/docs/command-line/environment-variables.inc
index db1ac928a..6bd163fbf 100644
--- a/docs/command-line/environment-variables.inc
+++ b/docs/command-line/environment-variables.inc
@@ -72,6 +72,12 @@ Variables Glossary
| ``DISTRO_APT_PREMIRRORS``| Specifies alternatives for apt URLs. Just like |
| (C) | ``KAS_PREMIRRORS``. |
+--------------------------+--------------------------------------------------+
+| ``KAS_GIT_SHALLOW`` | Perform shallow git clone/fetch using --depth=N |
+| (C, K) | specified by this variable. This is useful in |
+| | case CI always starts with empty work directory |
+| | and this directory is always discarded after the |
+| | CI run. |
++--------------------------+--------------------------------------------------+
| ``SSH_PRIVATE_KEY`` | Variable containing the private key that should |
| (K) | be added to an internal ssh-agent. This key |
| | cannot be password protected. This setting is |
diff --git a/kas-container b/kas-container
index 39c984a29..5eca76c8b 100755
--- a/kas-container
+++ b/kas-container
@@ -546,7 +546,7 @@ if [ -n "${KAS_REPO_REF_DIR}" ]; then
-e KAS_REPO_REF_DIR=/repo-ref
fi

-for var in TERM KAS_DISTRO KAS_MACHINE KAS_TARGET KAS_TASK \
+for var in TERM KAS_DISTRO KAS_MACHINE KAS_TARGET KAS_TASK KAS_GIT_SHALLOW \
KAS_PREMIRRORS DISTRO_APT_PREMIRRORS BB_NUMBER_THREADS PARALLEL_MAKE \
GIT_CREDENTIAL_USEHTTPPATH; do
if [ -n "$(eval echo \$${var})" ]; then
diff --git a/kas/repos.py b/kas/repos.py
index 62b2bbd75..fd1ef3638 100644
--- a/kas/repos.py
+++ b/kas/repos.py
@@ -523,6 +523,13 @@ class GitRepo(RepoImpl):

def clone_cmd(self, srcdir, createref):
cmd = ['git', 'clone', '-q']
+
+ depth = os.environ.get('KAS_GIT_SHALLOW')
+ if depth:
+ if not depth.isdigit():
+ raise KasUserError('KAS_GIT_SHALLOW must be a number')
+ cmd.extend(['--depth', depth])
+
if createref:
cmd.extend([self.effective_url, '--bare', srcdir])
elif srcdir:
@@ -543,6 +550,11 @@ class GitRepo(RepoImpl):

def fetch_cmd(self):
cmd = ['git', 'fetch', '-q']
+
+ depth = os.environ.get('KAS_GIT_SHALLOW')
+ if depth:
+ cmd.extend(['--depth', depth])
+
if self.tag:
cmd.append('--tags')

diff --git a/tests/conftest.py b/tests/conftest.py
index ef0dc9d10..f0f134c82 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -32,6 +32,7 @@ ENVVARS_KAS = [
'KAS_TARGET',
'KAS_TASK',
'KAS_PREMIRRORS',
+ 'KAS_GIT_SHALLOW',
'SSH_PRIVATE_KEY',
'SSH_PRIVATE_KEY_FILE',
'SSH_AUTH_SOCK',
diff --git a/tests/test_commands.py b/tests/test_commands.py
index 849882e6a..4d178b800 100644
--- a/tests/test_commands.py
+++ b/tests/test_commands.py
@@ -29,7 +29,8 @@ import yaml
import subprocess
import pytest
from kas import kas
-from kas.libkas import TaskExecError
+from kas.libkas import run_cmd
+from kas.libkas import TaskExecError, KasUserError


def test_for_all_repos(monkeykas, tmpdir):
@@ -107,6 +108,24 @@ def test_checkout_create_refs(monkeykas, tmpdir):
assert os.path.exists('kas/.git/objects/info/alternates')


+def test_checkout_shallow(monkeykas, tmpdir):
+ tdir = str(tmpdir / 'test_commands')
+ shutil.copytree('tests/test_commands', tdir)
+ monkeykas.chdir(tdir)
+ with monkeykas.context() as mp:
+ mp.setenv('KAS_GIT_SHALLOW', 'invalid')
+ with pytest.raises(KasUserError):
+ kas.kas(['checkout', 'test-shallow.yml'])
+
+ with monkeykas.context() as mp:
+ mp.setenv('KAS_GIT_SHALLOW', '1')
+ kas.kas(['checkout', 'test-shallow.yml'])
+ (rc, output) = run_cmd(['git', 'rev-list', '--count', 'HEAD'], cwd='kas',
+ fail=False, liveupdate=False)
+ assert rc == 0
+ assert output.strip() == '1'
+
+
def test_repo_includes(monkeykas, tmpdir):
tdir = str(tmpdir / 'test_commands')
shutil.copytree('tests/test_repo_includes', tdir)
diff --git a/tests/test_commands/test-shallow.yml b/tests/test_commands/test-shallow.yml
new file mode 100644
index 000000000..a3128d3bc
--- /dev/null
+++ b/tests/test_commands/test-shallow.yml
@@ -0,0 +1,9 @@
+header:
+ version: 14
+
+repos:
+ this:
+
+ kas:
+ url: https://github.com/siemens/kas.git
+ commit: master
--
2.39.2

[Marek Vasut]

On 5/16/24 10:15 AM, Felix Moessbauer wrote:
> From: Marek Vasut <ma...@denx.de>
>
> Add new environment variable KAS_GIT_SHALLOW which adds '--depth=N'
> to the 'git clone' and 'git fetch' commands. This forces git to
> perform shallow clone, which saves bandwidth and CI runner disk
> space. The depth 'N' is derived from KAS_GIT_SHALLOW value.
>
> This is useful in case CI always starts with empty work directory
> and this directory is always discarded after the CI run. In that
> case, it makes no sense to clone the entire repository, instead
> clone just enough to reproduce the desired state of the repository
> and assemble the checkout of it.
>
> This is also useful when cloning massive repositories which would
> otherwise take long time to clone.
>
> [Felix: rebased, sanitize input values, adapted doc entry to new
> format, port test over to monkeykas infrastructure, forward from
> kas-container]
>
> Signed-off-by: Marek Vasut <ma...@denx.de>
> Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>

Thanks for revisiting this, I have to admit, I did put this on a back
burner and didn't get to it since.

[Jan Kiszka]

I'm not directly seeing our discussion in
https://groups.google.com/d/msgid/kas-devel/20231031190555.145934-1-marex%40denx.de
being addressed in this version, but I may also still lack enough coffee.

Jan

--
Siemens AG, Technology
Linux Expert Center

[MOESSBAUER, Felix]

Maybe it was my lack of coffee. I totally overlooked this, despite
being involved in the discussion back then.

Unfortunately this topic is more complicated than just limiting the
clone depth. While I'm working at it, you can use the KAS_REPO_REF_DIR
together with a CI cache to speedup slow upstream clones.

Felix