[PATCH v2 1/1] kas-container: support git worktrees

[Felix Moessbauer]

From: Jörg Sommer <joerg....@navimatix.de>

Git support sharing the .git directory across multiple worktrees each having
its own HEAD, index, rebase-todo and so on. The worktree directory (created
with `git worktree`) contains no directory .git, but a file .git with a
reference to the main git directory. Therefore, this directory must also be
mounted in the container to make git operable.

[Felix]
Git uses an absolute path to anchor the worktrees to the common
repository. As we do not want to recreate the full host path inside the
container, we mount the common repo as /repo-common and patch the .git
file of the worktree to point to that instead of the path on the host.
This is implemented using a temporary .git file with the patched path.
By that, the worktree on the host is kept unchanged.

Closes: #107
Signed-off-by: Jörg Sommer <joerg....@navimatix.de>
Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
@Jörg: It would be great if you could test this patch and confirm that it works
as expected. Thanks!

Best regards,
Felix Moessbauer
Siemens AG

kas-container | 16 ++++++++++++++++
1 file changed, 16 insertions(+)

diff --git a/kas-container b/kas-container
index 0437381ab..2b239a916 100755
--- a/kas-container
+++ b/kas-container
@@ -454,6 +454,17 @@ if [ "$(id -u)" -eq 0 ] && [ "${KAS_ALLOW_ROOT}" != "yes" ] ; then
"KAS_ALLOW_ROOT=yes to override."
fi

+if git_com_dir=$(git -C "${KAS_REPO_DIR}" rev-parse --git-common-dir 2>/dev/null) \
+ && [ "$git_com_dir" != "$(git -C "${KAS_REPO_DIR}" rev-parse --git-dir 2>/dev/null)" ]; then
+ # If (it's a git repo) and the common dir isn't the git-dir, it is shared worktree and
+ # we have to mount the common dir in the container to make git work
+ # The mount path inside the container is different from the host path. Hence, we over-mount
+ # the .git file to point to the correct path.
+ git_overlay_file=$(mktemp)
+ sed "s|gitdir: ${git_com_dir}/|gitdir: /repo-common/|" "${KAS_REPO_DIR}/.git" > "${git_overlay_file}"
+ set -- "$@" -v "${git_com_dir}:/repo-common:${KAS_REPO_MOUNT_OPT}" -v "${git_overlay_file}:/repo/.git:ro"
+fi
+
set -- "$@" -v "${KAS_REPO_DIR}:/repo:${KAS_REPO_MOUNT_OPT}" \
-v "${KAS_WORK_DIR}":/work:rw -e KAS_WORK_DIR=/work \
-v "${KAS_BUILD_DIR}":/build:rw \
@@ -609,3 +620,8 @@ done

# shellcheck disable=SC2086
trace ${KAS_CONTAINER_COMMAND} run "$@"
+
+# cleanup
+if [ -f "${git_overlay_file}" ]; then
+ trace rm -f "${git_overlay_file}"
+fi
--
2.39.5

[Quirin Gylstorff]

On 12/17/24 13:32, 'Felix Moessbauer' via kas-devel wrote:
> From: Jörg Sommer <joerg....@navimatix.de>
>
> Git support sharing the .git directory across multiple worktrees each having
> its own HEAD, index, rebase-todo and so on. The worktree directory (created
> with `git worktree`) contains no directory .git, but a file .git with a
> reference to the main git directory. Therefore, this directory must also be
> mounted in the container to make git operable.
>
> [Felix]
> Git uses an absolute path to anchor the worktrees to the common
> repository. As we do not want to recreate the full host path inside the
> container, we mount the common repo as /repo-common and patch the .git
> file of the worktree to point to that instead of the path on the host.
> This is implemented using a temporary .git file with the patched path.
> By that, the worktree on the host is kept unchanged.
>
> Closes: #107
> Signed-off-by: Jörg Sommer <joerg....@navimatix.de>
> Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>

I tested it with one off my projects - works.
Tested-by: Quirin Gylstorff <quirin.g...@siemens.com>

[Jan Kiszka]

KAS_GIT_OVERLAY_FILE - we generally capitalize vars and use a prefix.

In addition, KAS_GIT_OVERLAY_FILE should be initialized as empty outside
of this if block, just to be safe.

> + sed "s|gitdir: ${git_com_dir}/|gitdir: /repo-common/|" "${KAS_REPO_DIR}/.git" > "${git_overlay_file}"
> + set -- "$@" -v "${git_com_dir}:/repo-common:${KAS_REPO_MOUNT_OPT}" -v "${git_overlay_file}:/repo/.git:ro"
> +fi
> +
> set -- "$@" -v "${KAS_REPO_DIR}:/repo:${KAS_REPO_MOUNT_OPT}" \
> -v "${KAS_WORK_DIR}":/work:rw -e KAS_WORK_DIR=/work \
> -v "${KAS_BUILD_DIR}":/build:rw \
> @@ -609,3 +620,8 @@ done
>
> # shellcheck disable=SC2086
> trace ${KAS_CONTAINER_COMMAND} run "$@"
> +
> +# cleanup
> +if [ -f "${git_overlay_file}" ]; then
> + trace rm -f "${git_overlay_file}"
> +fi

Jan

--
Siemens AG, Foundational Technologies
Linux Expert Center

[Jörg Sommer]

Felix Moessbauer schrieb am Di 17. Dez, 13:32 (+0100):

> diff --git a/kas-container b/kas-container
> index 0437381ab..2b239a916 100755
> --- a/kas-container
> +++ b/kas-container
> @@ -454,6 +454,17 @@ if [ "$(id -u)" -eq 0 ] && [ "${KAS_ALLOW_ROOT}" != "yes" ] ; then
> "KAS_ALLOW_ROOT=yes to override."
> fi
>
> +if git_com_dir=$(git -C "${KAS_REPO_DIR}" rev-parse --git-common-dir 2>/dev/null) \
> + && [ "$git_com_dir" != "$(git -C "${KAS_REPO_DIR}" rev-parse --git-dir 2>/dev/null)" ]; then

I found that you can remove the second 2>/…, because at this point it's a
git dir and rev-parse should give real errors.

> + # If (it's a git repo) and the common dir isn't the git-dir, it is shared worktree and
> + # we have to mount the common dir in the container to make git work
> + # The mount path inside the container is different from the host path. Hence, we over-mount
> + # the .git file to point to the correct path.
> + git_overlay_file=$(mktemp)

How about using trap? Now, in case of SIGINT (or SIGTERM) the clean-up code
is never reached.

> + sed "s|gitdir: ${git_com_dir}/|gitdir: /repo-common/|" "${KAS_REPO_DIR}/.git" > "${git_overlay_file}"

I really would not edit git's internal files. I found no public interface
like git-config to change this value, and I think they intentionally provide
only GIT_COMMON_DIR.

Inside the worktrees are also files they contain paths. I don't know when
they are used, but with uncommon values they might need to be updated, too.

% gcat .git/worktrees/tmp/*dir
.git/worktrees/tmp/commondir:1:../..
.git/worktrees/tmp/gitdir:1:/home/joerg_sommer/kein_backup/tmp/.git

But as I said, I would not edit git's internal state, because this interface
can become different with the next version.

> + set -- "$@" -v "${git_com_dir}:/repo-common:${KAS_REPO_MOUNT_OPT}" -v "${git_overlay_file}:/repo/.git:ro"

I think you need to add `-v "${git_overlay_file}:/work/.git:ro"`

> +fi
> +

At me, I did

% git worktree add -b wt-tmp ~/kein_backup/tmp origin/main

% sha256sum kas-container
0099fb0c93e3ca849def15c6512e6e4a28df870c3a76681af1f730ee463e9fde kas-container

% git am /tmp/pat
Applying: Git support sharing the .git directory across multiple worktrees each having

% ./kas-container --log-level warning shell -c 'echo works'
fatal: not a git repository: /home/joerg_sommer/Projekte/prj/.git/worktrees/tmp
abort: no repository found in '/work' (.hg not found)
fatal: not a git repository: /home/joerg_sommer/Projekte/prj/.git/worktrees/tmp
abort: no repository found in '/work' (.hg not found)
fatal: not a git repository: /home/joerg_sommer/Projekte/prj/.git/worktrees/tmp
abort: no repository found in '/work' (.hg not found)
fatal: not a git repository: /home/joerg_sommer/Projekte/prj/.git/worktrees/tmp
fatal: not a git repository: /home/joerg_sommer/Projekte/prj/.git/worktrees/tmp
works

% ./kas-container --log-level debug shell -c 'echo works'
+ mkdir -p /home/joerg_sommer/.cache/yocto-dl-cache
+ mkdir -p /home/joerg_sommer/.cache/yocto-sstate
+ docker run -v /home/joerg_sommer/Projekte/prj/.git:/repo-common:rw -v /tmp/tmp.6jSUsXT9bf:/repo/.git:ro -v /home/joerg_sommer/kein_backup/tmp:/repo:rw -v /home/joerg_sommer/kein_backup/tmp:/work:rw -e KAS_WORK_DIR=/work -v /home/joerg_sommer/kein_backup/tmp/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=1001 -e GROUP_ID=1001 --rm --init -t -i -v /home/joerg_sommer/.cache/yocto-dl-cache:/downloads:rw -e DL_DIR=/downloads -v /home/joerg_sommer/.cache/yocto-sstate:/sstate:rw -e SSTATE_DIR=/sstate -e TERM=screen -e SHELL=/bin/bash --log-driver=none --user=root ghcr.io/siemens/kas/kas:4.6 -l debug shell -c echo works
2024-12-20 06:58:29 - INFO - kas 4.6 started
2024-12-20 06:58:29 - DEBUG - Using selector: EpollSelector
2024-12-20 06:58:29 - DEBUG - /work$ git rev-parse --show-toplevel --show-superproject-working-tree
fatal: not a git repository: /home/joerg_sommer/Projekte/prj/.git/worktrees/tmp
2024-12-20 06:58:29 - DEBUG - /work$ hg root
abort: no repository found in '/work' (.hg not found)
2024-12-20 06:58:29 - DEBUG - /work$ git rev-parse --show-toplevel --show-superproject-working-tree
fatal: not a git repository: /home/joerg_sommer/Projekte/prj/.git/worktrees/tmp
2024-12-20 06:58:29 - DEBUG - /work$ hg root
abort: no repository found in '/work' (.hg not found)
2024-12-20 06:58:29 - DEBUG - execute setup_dir
2024-12-20 06:58:29 - DEBUG - execute setup_home
2024-12-20 06:58:29 - DEBUG - execute init_setup_repos
2024-12-20 06:58:29 - DEBUG - append lockfile /work/.kas/upstream.lock.yml
2024-12-20 06:58:29 - DEBUG - append lockfile /work/.kas/upstream.lock.yml
2024-12-20 06:58:29 - DEBUG - execute repo_setup_loop
2024-12-20 06:58:29 - DEBUG - Loop repo_setup_loop: execute setup_repos_step
2024-12-20 06:58:29 - DEBUG - execute finish_setup_repos
2024-12-20 06:58:29 - DEBUG - /work$ git rev-parse --show-toplevel --show-superproject-working-tree
fatal: not a git repository: /home/joerg_sommer/Projekte/prj/.git/worktrees/tmp
2024-12-20 06:58:29 - DEBUG - /work$ hg root
abort: no repository found in '/work' (.hg not found)
2024-12-20 06:58:29 - INFO - Using /work as root for repository meta-prj
2024-12-20 06:58:29 - DEBUG - /work$ git remote get-url origin
fatal: not a git repository: /home/joerg_sommer/Projekte/prj/.git/worktrees/tmp
2024-12-20 06:58:29 - DEBUG - /work$ git rev-parse --verify HEAD
fatal: not a git repository: /home/joerg_sommer/Projekte/prj/.git/worktrees/tmp


Regards, Jörg

--
http://www.gnu.org/fun/jokes/ed.msg.html
»Note the consistent user interface and error reportage. Ed is generous
enough to flag errors, yet prudent enough not to overwhelm the novice with
verbosity.«

[MOESSBAUER, Felix]

We did not have one (yet), that's why I didn't add it. But you're
right. I'll change that in a v3.

>
> > +       sed "s|gitdir: ${git_com_dir}/|gitdir: /repo-common/|"
> > "${KAS_REPO_DIR}/.git" > "${git_overlay_file}"
>
> I really would not edit git's internal files. I found no public
> interface
> like git-config to change this value, and I think they intentionally
> provide
> only GIT_COMMON_DIR.

The problem is that that the interface we need simply does not exist.
We also can't simply bind-mount the host-path into the container under
the same path as this might interfere with kas-container internals. By
that, we have to cut corners or can't support git worktrees in the
first place.

I'm actually thinking about adding a bold statement to the docs that
git worktrees are only supported on a best-effort base. But this is
something Jan as maintainer should decide.

>
> Inside the worktrees are also files they contain paths. I don't know
> when
> they are used, but with uncommon values they might need to be
> updated, too.
>
> % gcat .git/worktrees/tmp/*dir
> .git/worktrees/tmp/commondir:1:../..
> .git/worktrees/tmp/gitdir:1:/home/joerg_sommer/kein_backup/tmp/.git

This is the view from the host side. As long as you don't operate on
the worktree within the container, I guess it is safe. From host-
perspective we don't modify the worktree common or the view in any
kind. But if the view is mounted rw inside the container we actually
might get trouble. Probably we better forbid that and always mount the
repos RO if it is a worktree.

>
> But as I said, I would not edit git's internal state, because this
> interface
> can become different with the next version.

Well... yes, but what is the alternative? A bind-mount of the host path
under the same path inside the container is not acceptable either.

>
> > +       set -- "$@" -v "${git_com_dir}:/repo-
> > common:${KAS_REPO_MOUNT_OPT}" -v
> > "${git_overlay_file}:/repo/.git:ro"
>
> I think you need to add `-v "${git_overlay_file}:/work/.git:ro"`

Hmm... why did it work for me? I'll check again.

I'll try to reproduce. All these false errors are an artifact from
repo-probing which is fixed in latest next. It is not critical, but
annoying and probably will be fixed in a 4.6.1 release.

Best regards,
Felix

>
>
> Regards, Jörg
>

--
Siemens AG, Technology
Linux Expert Center

[Felix Moessbauer]

From: Jörg Sommer <joerg....@navimatix.de>

Git support sharing the .git directory across multiple worktrees each having

its own HEAD, index, rebase-todo and so on. The worktree directory (created
with `git worktree`) contains no directory .git, but a file .git with a
reference to the main git directory. Therefore, this directory must also be
mounted in the container to make git operable.

[Felix]
Git uses an absolute path to anchor the worktrees to the common
repository. As we do not want to recreate the full host path inside the
container, we mount the common repo as /repo-common and patch the .git
file of the worktree to point to that instead of the path on the host.
This is implemented using a temporary .git file with the patched path.
By that, the worktree on the host is kept unchanged.

Closes: #107
Signed-off-by: Jörg Sommer <joerg....@navimatix.de>
Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

Changes since v2:

- use trap to cleanup temporary file
- make KAS_GIT_OVERLAY_FILE uppercase and move to global scope
- removed the second 2>/dev/null check as proposed by Jörg
- also over-mount the /work/.git file
- move git worktree detection after /repo and /work are mounted
(on podman that does not seem to matter, but on docker it does)
- add warning about experimental support (@Jan feel free to drop this)

I tested this with kas menu, kas build (no-config), kas build (with config)
and kas shell.

Best regards,
Felix Moessbauer

kas-container | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)

diff --git a/kas-container b/kas-container
index 0437381ab..02b350cd6 100755
--- a/kas-container
+++ b/kas-container
@@ -153,6 +153,15 @@ run_clean() {
fi
}

+KAS_GIT_OVERLAY_FILE=""
+kas_container_cleanup()
+{
+ if [ -f "${KAS_GIT_OVERLAY_FILE}" ]; then
+ trace rm -f "${KAS_GIT_OVERLAY_FILE}"
+ fi
+}
+trap kas_container_cleanup EXIT
+
set_container_image_var() {
KAS_IMAGE_VERSION="${KAS_IMAGE_VERSION:-${KAS_IMAGE_VERSION_DEFAULT}}"
KAS_CONTAINER_IMAGE_NAME="${KAS_CONTAINER_IMAGE_NAME:-${KAS_CONTAINER_IMAGE_NAME_DEFAULT}}"
@@ -461,6 +470,20 @@ set -- "$@" -v "${KAS_REPO_DIR}:/repo:${KAS_REPO_MOUNT_OPT}" \
-e KAS_BUILD_DIR=/build \
-e USER_ID="$(id -u)" -e GROUP_ID="$(id -g)" --rm --init

+if git_com_dir=$(git -C "${KAS_REPO_DIR}" rev-parse --git-common-dir 2>/dev/null) \
+ && [ "$git_com_dir" != "$(git -C "${KAS_REPO_DIR}" rev-parse --git-dir)" ]; then
+ # If (it's a git repo) and the common dir isn't the git-dir, it is shared worktree and

+ # we have to mount the common dir in the container to make git work
+ # The mount path inside the container is different from the host path. Hence, we over-mount
+ # the .git file to point to the correct path.

+ warning "Shared git worktree detected. Support is experimental."
+ KAS_GIT_OVERLAY_FILE=$(mktemp)
+ sed "s|gitdir: ${git_com_dir}/|gitdir: /repo-common/|" "${KAS_REPO_DIR}/.git" > "${KAS_GIT_OVERLAY_FILE}"
+ set -- "$@" -v "${git_com_dir}:/repo-common:${KAS_REPO_MOUNT_OPT}" \
+ -v "${KAS_GIT_OVERLAY_FILE}:/repo/.git:ro" \
+ -v "${KAS_GIT_OVERLAY_FILE}:/work/.git:ro"
+fi
+
if [ -n "${KAS_SSH_DIR}" ] ; then
if [ ! -d "${KAS_SSH_DIR}" ]; then
fatal_error "passed KAS_SSH_DIR '${KAS_SSH_DIR}' is not a directory"
--
2.39.5

[Quirin Gylstorff]

On 12/20/24 12:41, 'Felix Moessbauer' via kas-devel wrote:
> From: Jörg Sommer <joerg....@navimatix.de>
>
> Git support sharing the .git directory across multiple worktrees each having
> its own HEAD, index, rebase-todo and so on. The worktree directory (created
> with `git worktree`) contains no directory .git, but a file .git with a
> reference to the main git directory. Therefore, this directory must also be
> mounted in the container to make git operable.
>
> [Felix]
> Git uses an absolute path to anchor the worktrees to the common
> repository. As we do not want to recreate the full host path inside the
> container, we mount the common repo as /repo-common and patch the .git
> file of the worktree to point to that instead of the path on the host.
> This is implemented using a temporary .git file with the patched path.
> By that, the worktree on the host is kept unchanged.
>
> Closes: #107
> Signed-off-by: Jörg Sommer <joerg....@navimatix.de>
> Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>

Tested-by: Quirin Gylstorff <quirin.g...@siemens.com>

I tested it with kas menu, kas build on isar-cip-core and some internal
projects with config and shell.

Best regards,
Quirin

[joerg....@navimatix.de]

MOESSBAUER, Felix schrieb am Fr 20. Dez, 10:14 (+0000):

I don't know if it's worth to add a generic mechanism, but this is a helper
function I use in scripts:

on_exit() {
on_exit="${on_exit:+$on_exit; }$*"
trap '(set +e; eval "$on_exit")' EXIT INT TERM
}

on_exit "rm '$file'"

> > > +       sed "s|gitdir: ${git_com_dir}/|gitdir: /repo-common/|"
> > > "${KAS_REPO_DIR}/.git" > "${git_overlay_file}"
> >
> > I really would not edit git's internal files. I found no public
> > interface
> > like git-config to change this value, and I think they intentionally
> > provide
> > only GIT_COMMON_DIR.
>
> The problem is that that the interface we need simply does not exist.

I think this is intentionally. If git people would have wanted to allow
changing this value, they would have opened the API. In programming
languages this can be enforces by private/public. And even if you know the
layout of the struct you should not access members with pointer arithmetic.
Or doing "class_eval" in Ruby to access the private members.

> We also can't simply bind-mount the host-path into the container under the
> same path as this might interfere with kas-container internals. By that,
> we have to cut corners or can't support git worktrees in the first place.

Is it possible to detect these cases they “might” interface with
kas-container and warn/deny in this case?

Or add code like

if test -z "NO_GIT_WT_CHECK" && test is git worktree
then
echo "Git worktrees are not supported, because they might interfere" \
"with kas-container internals. Set the environment variable" \
"NO_GIT_WT_CHECK=1 and add call kas-container with --runtime-args"
"'-v' and try if it works. Good luck."
exit 1
fi

> > Inside the worktrees are also files they contain paths. I don't know
> > when
> > they are used, but with uncommon values they might need to be
> > updated, too.
> >
> > % gcat .git/worktrees/tmp/*dir
> > .git/worktrees/tmp/commondir:1:../..
> > .git/worktrees/tmp/gitdir:1:/home/joerg_sommer/kein_backup/tmp/.git
>
> This is the view from the host side. As long as you don't operate on
> the worktree within the container, I guess it is safe.

We do every time. We inject the current git commit ID in /etc/os-release.
And sometimes I use `git -C /work` in the devshell of recipes.

> From host- perspective we don't modify the worktree common or the view in
> any kind. But if the view is mounted rw inside the container we actually
> might get trouble. Probably we better forbid that and always mount the
> repos RO if it is a worktree.

But it's handy to tweak some code in devshell and then do

git diff > /work/…/pkg/tweak.patch

> > But as I said, I would not edit git's internal state, because this
> > interface
> > can become different with the next version.
>
> Well... yes, but what is the alternative? A bind-mount of the host path
> under the same path inside the container is not acceptable either.

What would be such an example that breaks? How has the setup to look like
that it conflicts with kas-container internals?


Regards, Jörg

--
Du kannst einem Schwein einen goldenen Ring durch die Nase ziehen,
deswegen bleibt es trozdem ein Schwein!

[Jan Kiszka]

Let's not overengineer here, also because eval should be avoided
whenever possible.

> }
>
> on_exit "rm '$file'"
>
>>>> +       sed "s|gitdir: ${git_com_dir}/|gitdir: /repo-common/|"
>>>> "${KAS_REPO_DIR}/.git" > "${git_overlay_file}"
>>>
>>> I really would not edit git's internal files. I found no public
>>> interface
>>> like git-config to change this value, and I think they intentionally
>>> provide
>>> only GIT_COMMON_DIR.
>>
>> The problem is that that the interface we need simply does not exist.
>

It's not really like it is undocumented:

https://git-scm.com/docs/gitrepository-layout

I don't see a practical issue in using this knowledge and gluing the
worktree differently to the main repo inside the container.

> I think this is intentionally. If git people would have wanted to allow
> changing this value, they would have opened the API. In programming
> languages this can be enforces by private/public. And even if you know the
> layout of the struct you should not access members with pointer arithmetic.
> Or doing "class_eval" in Ruby to access the private members.
>
>> We also can't simply bind-mount the host-path into the container under the
>> same path as this might interfere with kas-container internals. By that,
>> we have to cut corners or can't support git worktrees in the first place.
>
> Is it possible to detect these cases they “might” interface with
> kas-container and warn/deny in this case?

I clearly prefer the relocation pattern which we are already using for
KAS_*_DIR. Even if we detect potential conflicts upfront (repo gitdir
already exists on startup), we still can't tell what will happen during
the execution of the container.

We will have to find a way to enable --repo-rw, right. Possibly, we also
need to adjust the back-link from the repo to the worktree inside the
container (.git/worktree/*/gitdir).

>>> But as I said, I would not edit git's internal state, because this
>>> interface
>>> can become different with the next version.
>>
>> Well... yes, but what is the alternative? A bind-mount of the host path
>> under the same path inside the container is not acceptable either.
>
> What would be such an example that breaks? How has the setup to look like
> that it conflicts with kas-container internals?
>

See above.

[Jan Kiszka]

We also need support for --repo-rw.

> +fi
> +
> if [ -n "${KAS_SSH_DIR}" ] ; then
> if [ ! -d "${KAS_SSH_DIR}" ]; then
> fatal_error "passed KAS_SSH_DIR '${KAS_SSH_DIR}' is not a directory"

[joerg....@navimatix.de]

Jan Kiszka schrieb am Sa 21. Dez, 09:34 (+0100):

I thought more of a sequence of mkdir, git, touch commands to produce a
crash. But above I can't see it anywhere. Could you show the three or four
steps I have to do to see the “conflicts with kas-container internals”? Is
it something like `sudo git init /work`?


Regards, Jörg

--
»Um ihre Haltung zu trainieren, brauchen sie keine Turnschuhe, keine
Gewichte, sondern nur ihren Kopf. Üben sie zum Beispiel jeden Tag, über
den Tellerrand zu schau'n. Das geht sogar im Sitzen, am besten mit dem
Standard.« https://derstandard.at/

[Jan Kiszka]

What if

- some of the build scripts that the container executes operate on some
path element that happens to be part of the common repo path?
- we detect a conflict with the common repo (start with /build,
/builder, /work, /repo, /tmp, ...) path and therefore need to reject
the execution of the container?

I dislike both cases, thus I dislike exposing the host path layout to
the container. I would only make all this fragile.

Jan

[Felix Moessbauer]

Hi,

that's already supported. The worktree root is mounted with the same rw
/ro mode as all other repos. Only the .git file is mounted RO, but that
does not matter as this file is not written to (in fact it must not be
written to as this matches the kas-container internal repo relocation).

Just to be clear: Using "git worktree ..." (for write operations)
inside kas *will break things*, while using git {commit|add|...} works
flawlessly. Using "git worktree ..." outside of kas also keeps working.

IMHO this patch is ready to go.

Felix

[Jan Kiszka]

Right, this is already correct.

>
> Just to be clear: Using "git worktree ..." (for write operations)
> inside kas *will break things*, while using git {commit|add|...} works
> flawlessly. Using "git worktree ..." outside of kas also keeps working.

That's a fair limitation.

>
> IMHO this patch is ready to go.
>

There is a bug remaining: You must not create a .git overlay if
KAS_WORK_DIR != repo. It makes /work a git repo which isn't.

Jan

[Felix Moessbauer]

From: Jörg Sommer <joerg....@navimatix.de>

Git support sharing the .git directory across multiple worktrees each having
its own HEAD, index, rebase-todo and so on. The worktree directory (created
with `git worktree`) contains no directory .git, but a file .git with a
reference to the main git directory. Therefore, this directory must also be
mounted in the container to make git operable.

[Felix]
Git uses an absolute path to anchor the worktrees to the common
repository. As we do not want to recreate the full host path inside the
container, we mount the common repo as /repo-common and patch the .git
file of the worktree to point to that instead of the path on the host.
This is implemented using a temporary .git file with the patched path.
By that, the worktree on the host is kept unchanged.

Closes: #107
Signed-off-by: Jörg Sommer <joerg....@navimatix.de>
Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>

Tested-by: Quirin Gylstorff <quirin.g...@siemens.com>
---

Changes since v3:

- only mount .git in /work if /work and /repo are the same directory

Changes since v2:

- use trap to cleanup temporary file
- make KAS_GIT_OVERLAY_FILE uppercase and move to global scope
- removed the second 2>/dev/null check as proposed by Jörg
- also over-mount the /work/.git file
- move git worktree detection after /repo and /work are mounted
(on podman that does not seem to matter, but on docker it does)
- add warning about experimental support (@Jan feel free to drop this)

I tested this with kas menu, kas build (no-config), kas build (with config)
and kas shell.

Best regards,
Felix Moessbauer

kas-container | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)

diff --git a/kas-container b/kas-container
index 0437381ab..c826059c2 100755

--- a/kas-container
+++ b/kas-container
@@ -153,6 +153,15 @@ run_clean() {
fi
}

+KAS_GIT_OVERLAY_FILE=""
+kas_container_cleanup()
+{
+ if [ -f "${KAS_GIT_OVERLAY_FILE}" ]; then
+ trace rm -f "${KAS_GIT_OVERLAY_FILE}"
+ fi
+}
+trap kas_container_cleanup EXIT
+
set_container_image_var() {

KAS_IMAGE_VERSION="${KAS_IMAGE_VERSION:-${KAS_IMAGE_VERSION_DEFAULT}}"
KAS_CONTAINER_IMAGE_NAME="${KAS_CONTAINER_IMAGE_NAME:-${KAS_CONTAINER_IMAGE_NAME_DEFAULT}}"

@@ -461,6 +470,23 @@ set -- "$@" -v "${KAS_REPO_DIR}:/repo:${KAS_REPO_MOUNT_OPT}" \

-e KAS_BUILD_DIR=/build \
-e USER_ID="$(id -u)" -e GROUP_ID="$(id -g)" --rm --init

+if git_com_dir=$(git -C "${KAS_REPO_DIR}" rev-parse --git-common-dir 2>/dev/null) \
+ && [ "$git_com_dir" != "$(git -C "${KAS_REPO_DIR}" rev-parse --git-dir)" ]; then
+ # If (it's a git repo) and the common dir isn't the git-dir, it is shared worktree and

+ # we have to mount the common dir in the container to make git work
+ # The mount path inside the container is different from the host path. Hence, we over-mount
+ # the .git file to point to the correct path.
+ warning "Shared git worktree detected. Support is experimental."
+ KAS_GIT_OVERLAY_FILE=$(mktemp)
+ sed "s|gitdir: ${git_com_dir}/|gitdir: /repo-common/|" "${KAS_REPO_DIR}/.git" > "${KAS_GIT_OVERLAY_FILE}"

+ set -- "$@" -v "${git_com_dir}:/repo-common:${KAS_REPO_MOUNT_OPT}" \

+ -v "${KAS_GIT_OVERLAY_FILE}:/repo/.git:ro"

+ # if the workdir is the same as the repo dir, it is the same shared worktree
+ if [ "${KAS_WORK_DIR}" = "${KAS_REPO_DIR}" ]; then
+ set -- "$@" -v "${KAS_GIT_OVERLAY_FILE}:/work/.git:ro"
+ fi

+fi
+
if [ -n "${KAS_SSH_DIR}" ] ; then
if [ ! -d "${KAS_SSH_DIR}" ]; then
fatal_error "passed KAS_SSH_DIR '${KAS_SSH_DIR}' is not a directory"

--
2.45.2

[Jörg Sommer]

Felix Moessbauer schrieb am Do 02. Jan, 09:11 (+0100):
> From: Jörg Sommer <joerg....@navimatix.de>

Nearly all of this work was done by you, Felix. I think it's better when you
take the ownership.

> +KAS_GIT_OVERLAY_FILE=""
> +kas_container_cleanup()
> +{
> + if [ -f "${KAS_GIT_OVERLAY_FILE}" ]; then
> + trace rm -f "${KAS_GIT_OVERLAY_FILE}"
> + fi
> +}
> +trap kas_container_cleanup EXIT

I think you should add INT and TERM. At me, the EXIT trap is not run when
using these signals:

```
% dash -xc 'trap "echo bye" EXIT; true'
+ trap echo bye EXIT
+ true
+ echo bye
bye

% dash -xc 'trap "echo bye" EXIT; kill -INT $$'
+ trap echo bye EXIT
+ kill -INT 140297

% dash -xc 'trap "echo bye" EXIT INT; kill -INT $$'
+ trap echo bye EXIT INT
+ kill -INT 140497
+ echo bye
bye
+ echo bye
bye

% dash -xc 'trap "trap - EXIT; echo bye" EXIT INT; kill -INT $$'
+ trap trap - EXIT; echo bye EXIT INT
+ kill -INT 140629
+ trap - EXIT
+ echo bye
bye
```

> +
> set_container_image_var() {
> KAS_IMAGE_VERSION="${KAS_IMAGE_VERSION:-${KAS_IMAGE_VERSION_DEFAULT}}"
> KAS_CONTAINER_IMAGE_NAME="${KAS_CONTAINER_IMAGE_NAME:-${KAS_CONTAINER_IMAGE_NAME_DEFAULT}}"
> @@ -461,6 +470,23 @@ set -- "$@" -v "${KAS_REPO_DIR}:/repo:${KAS_REPO_MOUNT_OPT}" \
> -e KAS_BUILD_DIR=/build \
> -e USER_ID="$(id -u)" -e GROUP_ID="$(id -g)" --rm --init
>
> +if git_com_dir=$(git -C "${KAS_REPO_DIR}" rev-parse --git-common-dir 2>/dev/null) \
> + && [ "$git_com_dir" != "$(git -C "${KAS_REPO_DIR}" rev-parse --git-dir)" ]; then
> + # If (it's a git repo) and the common dir isn't the git-dir, it is shared worktree and
> + # we have to mount the common dir in the container to make git work
> + # The mount path inside the container is different from the host path. Hence, we over-mount
> + # the .git file to point to the correct path.
> + warning "Shared git worktree detected. Support is experimental."

BTW: Is it somehow possible to suppress this message (while keeping
warnings)? I saw no way to filter, and I suspect seeing a warning every time
makes it a little annoying. Maybe downgrade to info?


Regards Jörg

--
Macht besitzen und nicht ausüben ist wahre Größe.
(Friedl Beutelrock)

Navimatix GmbH T: 03641 - 327 99 0
Tatzendpromenade 2 F: 03641 - 526 306
07745 Jena www.navimatix.de

Geschäftsführer: Steffen Späthe, Jan Rommeley
Registergericht: Amtsgericht Jena, HRB 501480

[MOESSBAUER, Felix]

On Thu, 2025-01-02 at 10:56 +0100, Jörg Sommer wrote:
> Felix Moessbauer schrieb am Do 02. Jan, 09:11 (+0100):
> > From: Jörg Sommer <joerg....@navimatix.de>
>
> Nearly all of this work was done by you, Felix. I think it's better
> when you
> take the ownership.

Ok, fine for me.

>
> > +KAS_GIT_OVERLAY_FILE=""
> > +kas_container_cleanup()
> > +{
> > + if [ -f "${KAS_GIT_OVERLAY_FILE}" ]; then
> > + trace rm -f "${KAS_GIT_OVERLAY_FILE}"
> > + fi
> > +}
> > +trap kas_container_cleanup EXIT
>
> I think you should add INT and TERM. At me, the EXIT trap is not run
> when
> using these signals:

Good catch. This unfortunately depends on the shell. ash and dash don't
trap on EXIT via signals, bash does. Actually I'm surprised that
shellcheck did not catch this.

Anways. I'll include INT and TERM as well.

>
> ```
> % dash -xc 'trap "echo bye" EXIT; true'
> + trap echo bye EXIT
> + true
> + echo bye
> bye
>
> % dash -xc 'trap "echo bye" EXIT; kill -INT $$'
> + trap echo bye EXIT
> + kill -INT 140297
>
> % dash -xc 'trap "echo bye" EXIT INT; kill -INT $$'
> + trap echo bye EXIT INT
> + kill -INT 140497
> + echo bye
> bye
> + echo bye
> bye
>
> % dash -xc 'trap "trap - EXIT; echo bye" EXIT INT; kill -INT $$'
> + trap trap - EXIT; echo bye EXIT INT
> + kill -INT 140629
> + trap - EXIT
> + echo bye
> bye
> ```
>
> > +
> >  set_container_image_var() {

> >   KAS_IMAGE_VERSION="${KAS_IMAGE_VERSION:-
> > ${KAS_IMAGE_VERSION_DEFAULT}}"
> >   KAS_CONTAINER_IMAGE_NAME="${KAS_CONTAINER_IMAGE_NAME:-

> > ${KAS_CONTAINER_IMAGE_NAME_DEFAULT}}"
> > @@ -461,6 +470,23 @@ set -- "$@" -v
> > "${KAS_REPO_DIR}:/repo:${KAS_REPO_MOUNT_OPT}" \
> >   -e KAS_BUILD_DIR=/build \
> >   -e USER_ID="$(id -u)" -e GROUP_ID="$(id -g)" --rm --init
> >  
> > +if git_com_dir=$(git -C "${KAS_REPO_DIR}" rev-parse --git-common-
> > dir 2>/dev/null) \
> > + && [ "$git_com_dir" != "$(git -C "${KAS_REPO_DIR}" rev-
> > parse --git-dir)" ]; then
> > + # If (it's a git repo) and the common dir isn't the git-
> > dir, it is shared worktree and
> > + # we have to mount the common dir in the container to make
> > git work
> > + # The mount path inside the container is different from
> > the host path. Hence, we over-mount
> > + # the .git file to point to the correct path.
> > + warning "Shared git worktree detected. Support is
> > experimental."
>
> BTW: Is it somehow possible to suppress this message (while keeping
> warnings)? I saw no way to filter, and I suspect seeing a warning
> every time
> makes it a little annoying. Maybe downgrade to info?

Many people tested this by now, so I'll prefer to just remove this
message. I'll add a note to the documentation regarding the known
limitations (like better run no git worktree commands on the checked-
out worktree).

Best regards,
Felix

>
>
> Regards Jörg
>

[Felix Moessbauer]

Git support sharing the .git directory across multiple worktrees each having
its own HEAD, index, rebase-todo and so on. The worktree directory (created
with `git worktree`) contains no directory .git, but a file .git with a
reference to the main git directory. Therefore, this directory must also be

mounted in the container to make git operable.

Git uses an absolute path to anchor the worktrees to the common
repository. As we do not want to recreate the full host path inside the
container, we mount the common repo as /repo-common and patch the .git
file of the worktree to point to that instead of the path on the host.
This is implemented using a temporary .git file with the patched path.
By that, the worktree on the host is kept unchanged.

Closes: #107

Proposed-by: Jörg Sommer <joerg....@navimatix.de>
Tested-by: Quirin Gylstorff <quirin.g...@siemens.com>

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

Changes since v4:

- trap on EXIT, INT and TERM to cleanup temporary file (required for dash)
- move warning about experimental support to the documentation

Changes since v3:

- only mount .git in /work if /work and /repo are the same directory

Changes since v2:

- use trap to cleanup temporary file
- make KAS_GIT_OVERLAY_FILE uppercase and move to global scope
- removed the second 2>/dev/null check as proposed by Jörg
- also over-mount the /work/.git file
- move git worktree detection after /repo and /work are mounted
(on podman that does not seem to matter, but on docker it does)
- add warning about experimental support (@Jan feel free to drop this)

I tested this with kas menu, kas build (no-config), kas build (with config)
and kas shell.

Best regards,
Felix Moessbauer

docs/userguide/kas-container-description.inc | 5 ++++
kas-container | 25 ++++++++++++++++++++
2 files changed, 30 insertions(+)

diff --git a/docs/userguide/kas-container-description.inc b/docs/userguide/kas-container-description.inc
index ceab78706..e17359988 100644
--- a/docs/userguide/kas-container-description.inc
+++ b/docs/userguide/kas-container-description.inc
@@ -3,6 +3,11 @@ It gives fine grained control over the data that is mapped into the build and
decouples the build environment from the host system. The wrapper also takes care of
mounting the necessary directories and setting up the environment variables.

+.. note::
+ The ``kas-container`` script has experimental support for Git worktrees. Regular
+ Git operations on the checked-out repository are supported. However, executing
+ any ``git worktree ...`` command inside the container is not allowed.
+
By default ``kas-container`` uses the official images provided by the kas project:
``ghcr.io/siemens/kas/kas[-isar]:<version>``. To specify your own image set the
``KAS_CONTAINER_IMAGE`` environment variable. As container backends, Docker and
diff --git a/kas-container b/kas-container
index 0437381ab..aea7be8bd 100755

--- a/kas-container
+++ b/kas-container
@@ -153,6 +153,15 @@ run_clean() {
fi
}

+KAS_GIT_OVERLAY_FILE=""
+kas_container_cleanup()
+{
+ if [ -f "${KAS_GIT_OVERLAY_FILE}" ]; then
+ trace rm -f "${KAS_GIT_OVERLAY_FILE}"
+ fi
+}

+trap kas_container_cleanup EXIT INT TERM
+
set_container_image_var() {

KAS_IMAGE_VERSION="${KAS_IMAGE_VERSION:-${KAS_IMAGE_VERSION_DEFAULT}}"
KAS_CONTAINER_IMAGE_NAME="${KAS_CONTAINER_IMAGE_NAME:-${KAS_CONTAINER_IMAGE_NAME_DEFAULT}}"

@@ -461,6 +470,22 @@ set -- "$@" -v "${KAS_REPO_DIR}:/repo:${KAS_REPO_MOUNT_OPT}" \

-e KAS_BUILD_DIR=/build \
-e USER_ID="$(id -u)" -e GROUP_ID="$(id -g)" --rm --init

+if git_com_dir=$(git -C "${KAS_REPO_DIR}" rev-parse --git-common-dir 2>/dev/null) \
+ && [ "$git_com_dir" != "$(git -C "${KAS_REPO_DIR}" rev-parse --git-dir)" ]; then
+ # If (it's a git repo) and the common dir isn't the git-dir, it is shared worktree and

+ # we have to mount the common dir in the container to make git work
+ # The mount path inside the container is different from the host path. Hence, we over-mount
+ # the .git file to point to the correct path.

[Jan Kiszka]

s/experimental/limited/

> + Git operations on the checked-out repository are supported. However, executing
> + any ``git worktree ...`` command inside the container is not allowed.

Should we mount /repo-common/worktrees read-only so that any
modification attempt of worktree configuration is blocked, rather than
causing inconsistencies when repo-common is r/w mounted?

Jan

[MOESSBAUER, Felix]

Hi, please change this while merging (except if we need a v6).

>
> > +    Git operations on the checked-out repository are supported.
> > However, executing
> > +    any ``git worktree ...`` command inside the container is not
> > allowed.
>
> Should we mount /repo-common/worktrees read-only so that any
> modification attempt of worktree configuration is blocked, rather
> than
> causing inconsistencies when repo-common is r/w mounted?

Isn't that the directory where the git part of the worktree lives? This
should remain mutable. Anyways, I see no clear benefit in trying to
harden this. Now the limitation is documented and that should suffice.

IMHO this patch is ready to be merged.

Felix

>
> Jan
>
> > +
> >  By default ``kas-container`` uses the official images provided by
> > the kas project:
> >  ``ghcr.io/siemens/kas/kas[-isar]:<version>``. To specify your own
> > image set the
> >  ``KAS_CONTAINER_IMAGE`` environment variable. As container
> > backends, Docker and
> > diff --git a/kas-container b/kas-container
> > index 0437381ab..aea7be8bd 100755
> > --- a/kas-container
> > +++ b/kas-container
> > @@ -153,6 +153,15 @@ run_clean() {
> >   fi
> >  }
> >
> > +KAS_GIT_OVERLAY_FILE=""
> > +kas_container_cleanup()
> > +{
> > + if [ -f "${KAS_GIT_OVERLAY_FILE}" ]; then
> > + trace rm -f "${KAS_GIT_OVERLAY_FILE}"
> > + fi
> > +}
> > +trap kas_container_cleanup EXIT INT TERM
> > +
> >  set_container_image_var() {

> >   KAS_IMAGE_VERSION="${KAS_IMAGE_VERSION:-
> > ${KAS_IMAGE_VERSION_DEFAULT}}"
> >   KAS_CONTAINER_IMAGE_NAME="${KAS_CONTAINER_IMAGE_NAME:-

[Jan Kiszka]

Seems git actually keeps not just the references to the actual worktree
checkout folder ("gitdir") in the base .git/worktrees but also all the
metadata. That is a bit ugly, and it indeed forces us to keep this
folder writable when we want to allow updating its head inside the
container.

>
> IMHO this patch is ready to be merged.
>

I've updated docs as proposed and merged it.

Thanks,
Jan