[ANNOUNCE] Release 4.3

[Jan Kiszka]

Hi all,

A new release 4.3 is available. A big thanks to all contributors:
Felix Moessbauer, Jan Kiszka

Highlights in 4.3
- kas: fix including from transitively referenced repos
- kas: Add support for .gitconfig pass-through
- kas: Optimize checkout of repos in larger configurations
- kas: Reduce verbosity of kas startup output
- kas: check if branch contains commit if both are set
- kas: Improve error reporting in several places
- kas-container: Bit-identically reproducible images
- kas-container: Enrich manifests with provenance information
- kas-container: Add bash completion for kas
- docs: Separate man pages per subcommand
- docs: Various smaller improvements

Fun fact: A glitch popped up in the release pipeline itself, causing the
'latest' tag to be pushed by master, rather than the release workflow.
But as we are now fully reproducible, I just triggered the release
pipeline again, letting it align 'latest' with the 4.3 release. Guess
what? The container images didn't change!

Thanks,
Jan

https://github.com/siemens/kas/releases/tag/4.3
https://github.com/orgs/siemens/packages/container/package/kas%2Fkas
https://github.com/orgs/siemens/packages/container/package/kas%2Fkas-isar


Felix Moessbauer (43):
docs: improve documentation w.r.t empty env vars
fix including from transitively referenced repos
setup repos: print which repos are missing
add test for transitive repo includes
check if branch contains commit if both are set
raise error when tag and branch are defined
document how to pass multiple config files in help
log shell command inputs as debug instead of info
patch-apply: do not log output on success
fix(ssh config): do not emit messages on stdout
refactor(ssh config): use run_cmd infra for agent setup
document commit, branch and tag relations more precisely
cache repo instances to reduce nb of external cmds
test: add missing test for branch and tag on HG
schema: limit repo type to git and hg
tests: use monkeypatch to change cwd
tests: isolate test-env from global env
make PyYAML mandatory dependency
run commands without live-output as default
improved logging around repo checkouts
refactor: move git config part to dedicated function
add support to make .gitconfig available to kas
kas-container: add support for GITCONFIG_FILE
always make .gitconfig available in github ci
add test for git config rewrite
raise error on invalid SSH arg combinations
add test for setup and forward of SSH agent
check if required tools are installed for repo type
stop SSH setup if ssh tools are not found
docs: add section about credential handling
issue warning on requested update of pinned repo
docs: add meaningful title to manpage
docs: set language to en
docs: bump copyright version to 2024
make license detectable by licensee
update copyright year in license file
docs: move plugins to separate files
docs: move project-configuration to separate file
docs: move credentials to separate file
docs: create individual manpages
add bash completion for kas in docker
chore: on commit not in branch error show repo name
docs: document multiconfig semantics in config

Jan Kiszka (23):
ci: Update actions revisions
tests: Cleanup after run_cmd liveupdate change
scripts: checkcode: Guess the source directory if no argument is provided
plugins: menu: Adjust global variable name for newt availability
Lift GitPython into 3.x version series
Dockerfile: Remove /kas folder after installation
Dockerfile: Make Debian base image tag configurable
Dockerfile: Build against snapshot.debian.org
Dockerfile: Delete non-reproducible logs and caches
Dockerfile: Address non-reproducible installation of git-lfs
Dockerfile: Cache the apt cache locally
ci: Move common container build setup steps into a composite action
ci: Skip build steps completely on releases
ci: Use path-based git context
ci: Enable reproducible container builds
ci: Parallelize container build via matrix
ci: Switch to automatic token
ci: Enable apt caching for container builds
scripts: Add scripts for building and reproducing kas containers
ci: Enable build provenance attestation
ci: Add image descriptions and source
docs: Update devguide to latest changes on container images
Release 4.3


--
Siemens AG, Technology
Linux Expert Center

[Jörg Sommer]

On 11.03.24 21:32, 'Jan Kiszka' via kas-devel wrote:
> Hi all,
>
> A new release 4.3 is available. A big thanks to all contributors:

I'm getting an error:

% docker pull ghcr.io/siemens/kas/kas:4.3
Error response from daemon: Head "https://ghcr.io/v2/siemens/kas/kas/manifests/4.3": denied: denied


Kind regards

Jörg Sommer
--
Navimatix GmbH
Tatzendpromenade 2
D-07745 Jena
Geschäftsführer: Steffen Späthe, Jan Rommeley
Registergericht: Amtsgericht Jena, HRB 501480

[Jan Kiszka]

On 12.03.24 10:13, 'Jörg Sommer' via kas-devel wrote:
> On 11.03.24 21:32, 'Jan Kiszka' via kas-devel wrote:
>> Hi all,
>>
>> A new release 4.3 is available. A big thanks to all contributors:
>
> I'm getting an error:
>
> % docker pull ghcr.io/siemens/kas/kas:4.3
> Error response from daemon: Head "https://ghcr.io/v2/siemens/kas/kas/manifests/4.3": denied: denied
>

No idea yet what could cause this. Maybe there have been changes on
ghcr.io that make unauthenticated pulling incompatible with some docker
or podman versions. What exactly are you using?

FWIW, I've just updated one of our CI systems to 4.3, and that one
definitely pulls from ghcr.io without any login - and it works. Also,
that manifest access gave the same errors for older releases, thus is
nothing specific to how we produce now with 4.3.

Jan

[Jan Kiszka]

Suggestion of a colleague of mine: try "docker logout ghcr.io", then
retry pulling. Maybe there are credential remainders in play on your end.

[Jörg Sommer]

This helped. After the logout the pull works. Thanks for your help.

[Jasper Orschulko]

4.3 seems to break SSH authentication, at least when using
SSH_PRIVATE_KEY variable.

---

In 4.2:

docker run -it --rm -v /home/jasper/playground/kas-4.3-test/:/mnt
ghcr.io/siemens/kas/kas:4.2
builder@2762bbe85bde:~$ export SSH_PRIVATE_KEY="-----BEGIN OPENSSH
PRIVATE KEY-----
<REDACTED>
-----END OPENSSH PRIVATE KEY-----"
builder@2762bbe85bde:~$ cd /mnt/
builder@2762bbe85bde:/mnt$ kas checkout kas-irma6-pa.yml
2024-03-12 14:01:38 - INFO - kas 4.2 started
2024-03-12 14:01:38 - INFO - /mnt$ git rev-parse --show-toplevel
2024-03-12 14:01:38 - INFO - /mnt$ hg root
2024-03-12 14:01:38 - INFO - /mnt$ git rev-parse --show-toplevel
2024-03-12 14:01:38 - INFO - /mnt$ hg root
2024-03-12 14:01:38 - INFO - adding SSH key
2024-03-12 14:01:38 - INFO - /mnt$ git clone -q
g...@gitlab.devops.defra01.iris-sensing.net:rd-s/generation6/meta-
iris.git /mnt/meta-iris
2024-03-12 14:01:39 - ERROR - Warning: Permanently added
'gitlab.devops.defra01.iris-sensing.net' (ED25519) to the list of known
hosts.
2024-03-12 14:01:58 - INFO - Repository meta-iris cloned
2024-03-12 14:01:58 - INFO - /mnt/meta-iris$ git remote set-url
origin g...@gitlab.devops.defra01.iris-sensing.net:rd-
s/generation6/meta-iris.git
2024-03-12 14:01:58 - INFO - /mnt/meta-iris$ git cat-file -t
develop
2024-03-12 14:01:58 - INFO - Repository meta-iris already contains
develop as commit
2024-03-12 14:01:58 - INFO - /mnt/meta-iris$ git status -s
2024-03-12 14:01:58 - INFO - /mnt/meta-iris$ git rev-parse --verify
-q origin/develop
2024-03-12 14:01:58 - INFO -
05cb687b45c987279dc786dcebcb1916194ca841
2024-03-12 14:01:58 - INFO - /mnt/meta-iris$ git checkout -q
05cb687b45c987279dc786dcebcb1916194ca841 -B develop


---

In 4.3:

docker run -it --rm -v /home/jasper/playground/kas-4.3-test/:/mnt
ghcr.io/siemens/kas/kas:4.3
builder@8ced903ee704:~$ cd /mnt/
builder@8ced903ee704:/mnt$ export SSH_PRIVATE_KEY="-----BEGIN OPENSSH
PRIVATE KEY-----
<REDACTED>
-----END OPENSSH PRIVATE KEY-----"
builder@8ced903ee704:/mnt$ kas checkout kas-irma6-pa.yml
2024-03-12 13:58:49 - INFO - kas 4.3 started
2024-03-12 13:58:49 - INFO - adding SSH key
2024-03-12 13:58:49 - ERROR - failed to add ssh key: b'Could not
open a connection to your authentication agent.\n'
2024-03-12 13:58:49 - INFO - Cloning repository meta-iris
2024-03-12 13:58:49 - ERROR - Command "/mnt$ git clone -q
g...@gitlab.devops.defra01.iris-sensing.net:rd-s/generation6/meta-
iris.git /mnt/meta-iris" failed
--- Error summary ---
Warning: Permanently added 'gitlab.devops.defra01.iris-sensing.net'
(ED25519) to the list of known hosts.
g...@gitlab.devops.defra01.iris-sensing.net: Permission denied
(publickey,keyboard-interactive).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

2024-03-12 13:58:49 - ERROR - fetch repos failed: error code 128


Best regards,
Jasper

[Jasper Orschulko]

Looking at this closer, the issue is that the ssh-agent is not started
in 4.3, which seems to be due to this change:

@@ -352,11 +352,9 @@ def ssh_setup_agent(envkeys=None):
"""
Starts the ssh-agent
"""
- ctx = get_context()
- env = ctx.environ
+ env = get_context().environ
envkeys = envkeys or ['SSH_PRIVATE_KEY', 'SSH_PRIVATE_KEY_FILE']
- (_, output) = run_cmd(['ssh-agent', '-s'], env=env,
- cwd=ctx.kas_work_dir)
+ output = os.popen('ssh-agent -s').readlines()
for line in output:
matches = re.search(r"(\S+)\=(\S+)\;", line)
if matches:

It seems the env is missing from the os.popen, which should probably be
replaced with subprocess.Popen?

On Tue, 2024-03-12 at 14:07 +0000, 'Jasper Orschulko' via kas-devel
wrote:

[Jasper Orschulko]

Whoops, the diff is actually the other way around:

@@ -352,9 +352,11 @@ def ssh_setup_agent(envkeys=None):
"""
Starts the ssh-agent
"""
- env = get_context().environ
+ ctx = get_context()
+ env = ctx.environ

envkeys = envkeys or ['SSH_PRIVATE_KEY', 'SSH_PRIVATE_KEY_FILE']

- output = os.popen('ssh-agent -s').readlines()
+ (_, output) = run_cmd(['ssh-agent', '-s'], env=env,
+ cwd=ctx.kas_work_dir)

for line in output:
matches = re.search(r"(\S+)\=(\S+)\;", line)
if matches:

On Tue, 2024-03-12 at 14:27 +0000, 'Jasper Orschulko' via kas-devel

[MOESSBAUER, Felix]

On Tue, 2024-03-12 at 14:39 +0000, 'Jasper Orschulko' via kas-devel
wrote:

> Whoops, the diff is actually the other way around:

Yep! That's my cleanup change. I'll have a look.
Thanks for reporting.

Felix