kas(-docker): Issue with git submodules.

[Sudler, Simon]

Hi,

(copy from github issue: https://github.com/siemens/kas/issues/11)

When trying to build a isar based project, I noticed, that kas-docker has a problem with git submodules.

When executing the same kas-docker build in a "normal" git repository, everything works fine. Executing the command in a git submodule fails to detect the proper git-root:

$ kas-docker --isar -v build kas/boards/some-project.yml
+ mkdir -p /home/user/isar/isar
+ docker run -v /home/user/isar/isar:/repo:ro -v /home/user/isar/isar:/work:rw --workdir=/work -e USER_ID=1000 -e GROUP_ID=1000 --rm -t -i -e SHELL='/bin/bash' -e TERM='xterm-256color' --cap-add=SYS_ADMIN --cap-add=MKNOD --privileged --device /dev/loop20 kasproject/kas-isar build /repo/kas/boards/some-project.yml
2019-08-13 11:10:04 - INFO - kas 1.0 started
2019-08-13 11:10:04 - INFO - /repo/kas$ git rev-parse --show-toplevel
2019-08-13 11:10:04 - INFO - /repo/kas$ hg root
2019-08-13 11:10:04 - INFO - /repo/kas/board$ git rev-parse --show-toplevel
2019-08-13 11:10:04 - INFO - /repo/kas/board$ hg root
2019-08-13 11:10:04 - INFO - /repo/kas$ git rev-parse --show-toplevel
2019-08-13 11:10:04 - INFO - /repo/kas$ hg root
2019-08-13 11:10:04 - INFO - Using /repo/kas as root for repository isar
2019-08-13 11:10:04 - INFO - /work/isar$ git cat-file -t xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

"git rev-parse --show-toplevel" command cannot work with reverence in the .git file, since the top-level repository is not mounted inside the docker container.

Why does kas require the "git rev-parse --show-toplevel" in the project root folder? It seems unnecessary for proper cloning of the meta-layers defined in the kas-file.

With best regards,
Simon

[Jan Kiszka]

And to repeat my answer: It's needed to mount the repo, and only the repo. And
only read-only. When you just specify an absolute path to a kas yml file, it's
unclear where that repo start.

Counter-question: Why do you need submodules, when you have kas as CM tool?
Legacy reasons?

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Sudler, Simon]

> >
> > "git rev-parse --show-toplevel" command cannot work with reverence in
> > the .git file, since the top-level repository is not mounted inside the
> > docker container.
> >
> > Why does kas require the "git rev-parse --show-toplevel" in the project
> > root folder? It seems unnecessary for proper cloning of the meta-layers
> > defined in the kas-file.
> >
> > With best regards,
> > Simon
> >
>
> And to repeat my answer: It's needed to mount the repo, and only the repo.
> And only read-only. When you just specify an absolute path to a kas yml
> file, it's unclear where that repo start.
>
> Counter-question: Why do you need submodules, when you have kas as CM
> tool?
> Legacy reasons?

Not really, I usually organizing all projects with submodules. For setting ab
multiple targets, I needed some scripting around that kas build process.
We also use a top-level git repository where the "Jenkinsfile" are located.
This allows as to do project specific builds/tests without having to modify
upstream repositories.

I am using kas primarily to configure the isar project, the cloning of the
meta-layers repositories are a nice addition, but I wouldn't mind including
them in submodules as well.

So, since kas needs the "toplevel" from git to determine the (even when
called wit eh relative path to kas-file) repo start. Are you suggesting to not
use kas for configuration, since it does not fit into a larger submodule
structure?

Regards,
Simon

[Jan Kiszka]

I'm suggesting to rethink CM here again: If you are calling a kas build within a
submodule, you request to build that submodule, only, not the toplevel project.
If you try the latter, you are doing something fundamentally wrong.

If you put the kas file in the toplevel repo in order to build something that
also includes submodules, you should rather use kas to define the dependencies
and their revisions and let do kas the checkout. kas can include other kas files
from dependencies, so that toplevel kas file can be really small.

Our user story is

kas[-docker] [--isar] build target.yml[:option.yml]

Any wrapper around it is too much - well, except you what to add some
interactive "UI" (which could even become a kas feature at some point as well,
if there is enough interest). We intentionally model CI this way as well so that
we are not testing different things than the user would do.

[Sudler, Simon]

>>> And to repeat my answer: It's needed to mount the repo, and only the repo.
>>> And only read-only. When you just specify an absolute path to a kas
>>> yml file, it's unclear where that repo start.
>>>
>>> Counter-question: Why do you need submodules, when you have kas as CM
>>> tool?
>>> Legacy reasons?
>>
>> Not really, I usually organizing all projects with submodules. For
>> setting ab multiple targets, I needed some scripting around that kas build process.
>> We also use a top-level git repository where the "Jenkinsfile" are located.
>> This allows as to do project specific builds/tests without having to
>> modify upstream repositories.
>>
>> I am using kas primarily to configure the isar project, the cloning of
>> the meta-layers repositories are a nice addition, but I wouldn't mind
>> including them in submodules as well.
>>
>> So, since kas needs the "toplevel" from git to determine the (even
>> when called wit eh relative path to kas-file) repo start. Are you
>> suggesting to not use kas for configuration, since it does not fit
>> into a larger submodule structure?
>
> I'm suggesting to rethink CM here again: If you are calling a kas build
> within a submodule, you request to build that submodule, only, not
> the toplevel project.
> If you try the latter, you are doing something fundamentally wrong.

Depends on the use case: If you are just "building" using kas as CM might
be enough. If, however, you need to deploy and test the results, it's not
good enough.
Since the deployment and test corresponds directly to a specific commit,
it is necessary to store it in a corresponding repository.

Also, I cannot store e.g. Jenkins files in dummy-meta layer, because
they would only be available after the kas execution. But since the
kas execution must happen in the Jenkins file, it creates a chicken-or-egg
problem.

> If you put the kas file in the toplevel repo in order to build something
> that also includes submodules, you should rather use kas to define

> the dependencies and their revisions and let do kas the checkout. Kas

> can include other kas files from dependencies, so that toplevel kas file
> can be really small.
>
> Our user story is
>
> kas[-docker] [--isar] build target.yml[:option.yml]
>
> Any wrapper around it is too much - well, except you what to add some
> interactive "UI" (which could even become a kas feature at some point
> as well, if there is enough interest). We intentionally model CI this way
> as well so that we are not testing different things than the user would do.

I don't see as use case for an "interactive UI", since I only want reproducible
builds and _tests_. I suppose if there would be a

kas[-docker] [--isar] execute test.yml[:option.yml]

which allows the execution of custom commands on the build results would
help. On the other hand, why add CI/CD-support to a CM-only-tool.

I found a solution that works for my use case, but I still believe it would be
a good thing for kas to be git submodule agnostic.

Regards,
Simon

[Jan Kiszka]

It doesn't - unless I'm overseeing a fundamental limitation of Jenkins (we are
using gitlab-ci).

>
>> If you put the kas file in the toplevel repo in order to build something
>> that also includes submodules, you should rather use kas to define
>> the dependencies and their revisions and let do kas the checkout. Kas
>> can include other kas files from dependencies, so that toplevel kas file
>> can be really small.
>>
>> Our user story is
>>
>> kas[-docker] [--isar] build target.yml[:option.yml]
>>
>> Any wrapper around it is too much - well, except you what to add some
>> interactive "UI" (which could even become a kas feature at some point
>> as well, if there is enough interest). We intentionally model CI this way
>> as well so that we are not testing different things than the user would do.
>
> I don't see as use case for an "interactive UI", since I only want reproducible
> builds and _tests_. I suppose if there would be a
>
> kas[-docker] [--isar] execute test.yml[:option.yml]
>
> which allows the execution of custom commands on the build results would
> help. On the other hand, why add CI/CD-support to a CM-only-tool.

Right, there is no need for that. You do just two commands in your ci script,
one of them being the kas-based build, the other the execution of the artifacts
(including invocation of scripts or usage of configurations coming from kas CM
steps). Or we deploy artifacts afterwards. Or we kick-off a complete LAVA test
ran from there. Or, or, or... We did all that already, and there was zero need
for "kas execute". See eg.
https://github.com/siemens/meta-iot2000/blob/master/.gitlab-ci.yml

Maybe you can be more specific about what you would need from the kas build
environment in order to execute or deploy tests after the build. Then I may be
able to explain where to model things more elegantly.