Enable repo URL override

[Bogdan Prădatu]

Hello,

I have a use case where I am maintaining a meta-layer which could use public or private repositories and I was thinking that, the same as commit hashes can be overridden from yaml config files, the same could be done for the urls.

I have attached a minimal working example patch (no tests) just to demonstrate the PoC and get some initial feedback.

Let me know what you think,

Bogdan Prădatu

[MOESSBAUER, Felix]

On Tue, 2025-02-25 at 11:53 +0200, Bogdan Prădatu wrote:
> Hello,
>
> I have a use case where I am maintaining a meta-layer which could use
> public or private repositories and I was thinking that, the same as
> commit hashes can be overridden from yaml config files, the same
> could be done for the urls.

Hi,

can't you already achieve the same with the KAS_PREMIRRORS environment
variable? [1]

If you need the url switching to build in CI systems, I recommend to
just add the git+ssh format and let our CI detection logic take care of
rewriting this on the fly to https (e.g. on GitLab CI [2]).

[1]
https://kas.readthedocs.io/en/latest/command-line.html#variables-glossary
[2]
https://kas.readthedocs.io/en/latest/userguide/credentials.html#gitlab-ci

Best regards,
Felix

>
> I have attached a minimal working example patch (no tests) just to
> demonstrate the PoC and get some initial feedback.
>
> Let me know what you think,
> Bogdan Prădatu

--
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

[Bogdan Prădatu]

Is there any way to set the environment from a yaml config file?

I was trying to use the `env:` key, but realized they are only passed to bitbake:
https://kas.readthedocs.io/en/latest/userguide/project-configuration.html

env: dict [optional]
Environment variables to forward and their default values (set to nulltype to only forward if set). These variables are made available to bitbake via BB_ENV_PASSTHROUGH_ADDITIONS (BB_ENV_EXTRAWHITE in older Bitbake versions) and can be overwritten by the variables of the environment in which kas is started. Either a string or nothing (null) can be assigned as value. The former one serves as a default value whereas the latter one will lead to add the variable only to BB_ENV_PASSTHROUGH_ADDITIONS and not to the environment where kas is started. Please note, that null needs to be assigned as the nulltype (e.g. MYVAR: null), not as ‘null’.

I would have preferred to have a config file for internal use with everything in there that will override stuff and apply new configs and whatnot.

Thanks.

[MOESSBAUER, Felix]

On Tue, 2025-02-25 at 06:11 -0800, Bogdan Prădatu wrote:
> Is there any way to set the environment from a yaml config file?
>
> I was trying to use the `env:` key, but realized they are only passed
> to bitbake:
> https://kas.readthedocs.io/en/latest/userguide/project-configuration.html
>

Yes, the env section is only affecting the bitbake environment. The
reason for that is that kas first builds the (final) config and during
that cannot use information of the config itself.

> env: dict [optional]
> Environment variables to forward and their default values (set to
> nulltype to only forward if set). These variables are made available
> to bitbake via BB_ENV_PASSTHROUGH_ADDITIONS (BB_ENV_EXTRAWHITE in
> older Bitbake versions) and can be overwritten by the variables of
> the environment in which kas is started. Either a string or nothing
> (null) can be assigned as value. The former one serves as a default
> value whereas the latter one will lead to add the variable only to
> BB_ENV_PASSTHROUGH_ADDITIONS and not to the environment where kas is
> started. Please note, that null needs to be assigned as the nulltype
> (e.g. MYVAR: null), not as ‘null’.
>
> I would have preferred to have a config file for internal use with
> everything in there that will override stuff and apply new configs
> and whatnot.

Maybe we should go one step back: What exactly is your use-case? If you
could give an example, it might help to come up with a solution. I'm
also wondering if simple config drop-in with repo settings does the
job:

Example:

main.yml
repos:
foo:
url: https://example.com/foo
tag: v1

override.yml
repos:
foo:
url: https://local.domain/foo

kas checkout main.yml:override.yml

Best regards,
Felix

[Bogdan Prădatu]

I have multiple yaml files for different type of targets, images, whatever. Some of them need a meta-layer, some don't. If I over an override.yml, that will just add the repo, if it does not exist. The overrides functionality pretty much does exactly what I want: if the attribute is defined, it overrides it, otherwise it does nothing.

What I want to achieve is one or multiple repositories which point to public repositories and an override.yml file for CI and local development that will point to our internal repositories. Like these guys are doing in here, for example: https://github.com/nxp-auto-linux/auto_yocto_bsp/blob/release/bsp43.0/default-bitbucket.xml.

Yes, I could set the KAS_PREMIRRORS environment variable in CI, but for local developers, I feel that it's easier to just have everything in the kas configs and not bother them with setting environment variables.

[Bogdan Prădatu]

So, I guess it's not desired to allow repo url overrides?

[Jan Kiszka]

On 07.03.25 09:26, Bogdan Prădatu wrote:
> So, I guess it's not desired to allow repo url overrides?
>

Well, if there is a use case, we can discuss potential solutions. But be
warned that this may not be as simple as it looks initially.

> On Wednesday, 26 February 2025 at 11:42:18 UTC+2 Bogdan Prădatu wrote:
>
> I have multiple yaml files for different type of targets, images,
> whatever. Some of them need a meta-layer, some don't. If I over an
> override.yml, that will just add the repo, if it does not exist. The
> overrides functionality pretty much does exactly what I want: if the
> attribute is defined, it overrides it, otherwise it does nothing.
>
> What I want to achieve is one or multiple repositories which point
> to public repositories and an override.yml file for CI and local
> development that will point to our internal repositories. Like these
> guys are doing in here, for example: https://github.com/nxp-auto-
> linux/auto_yocto_bsp/blob/release/bsp43.0/default-bitbucket.xml

> <https://github.com/nxp-auto-linux/auto_yocto_bsp/blob/release/
> bsp43.0/default-bitbucket.xml>.

>
> Yes, I could set the KAS_PREMIRRORS environment variable in CI, but
> for local developers, I feel that it's easier to just have
> everything in the kas configs and not bother them with setting
> environment variables.

So, your demand is to have KAS_PREMIRRORS semantics carried by a kas
config snippet file? How would your use case look like concretely? Can
you sketch a set config files and their origin, similar to what Felix
tried? Would help to find the right design. Because I don't think it
would help you to simply add the url key to overwrite, just like the
commit key.

Jan

--
Siemens AG, Foundational Technologies
Linux Expert Center

[Bogdan Prădatu]

I don't demand anything, I came with a proposal. Ok, I've tried to sketch a simplified example of the configs.

I our meta layer we have something like this:
baseConfig.yaml
```
header:
  version: 14
local_conf_header:
  base: |
    CONF_VERSION = "2"
    EXTRA_IMAGE_FEATURES ?= "debug-tweaks"
    USER_CLASSES ?= "buildstats"
    PATCHRESOLVE = "noop"
    PACKAGECONFIG:append:pn-qemu-system-native = " sdl"    
repos:
  poky:
    url: "https://git.yoctoproject.org/git/poky"
    commit: 755632c2fcab43aa05cdcfa529727064b045073c
    layers:
      meta:
      meta-poky:
      meta-yocto-bsp:
```

someOtherConfig.yaml
```
header:
  version: 14
  includes:
    - kas/baseConfig.yml
```

target_1_config.yaml
```
header:
  version: 14
  includes:
    - kas/someOtherConfig.yml
machine: machine_1
target:
  - target_1
  - target_2
repos:
  meta-foo:
      url: "https://github.com/someProject/meta-foo.git"
      commit: f67e7141a92506a8b6897756fe5df969c4242453
      branch: release/someVersion
  meta-bar:
      url: "https://git.yoctoproject.org/git/meta-bar"
      commit: f67e7141a92506a8b6897756fe5df969c4242453
      branch: release/someVersion
```

target_2_config.yaml
```
header:
  version: 14
  includes:
    - kas/baseConfig.yml
machine: machine_2
target:
  - target_3
  - target_4
repos:
  meta-spam:
      url: "https://git.yoctoproject.org/git/meta-spam"
      commit: f67e7141a92506a8b6897756fe5df969c4242453
      branch: release/someVersion
```

In our dev meta layer that we only use internally for development and CI there is another config, something like:

devConfig.yaml
```
header:
  version: 14
repos:
  meta-dev:
     url: ssh://g...@bitbucket.com/project/meta-dev.git
     branch: develop
```

in this devConfig.yaml I would like to add the override for meta-foo that has a public and an internal version.
so devConfig.yaml will be something like:
```
header:
  version: 14
repos:
  meta-dev:
     url: ssh://g...@bitbucket.com/project/meta-dev.git
     branch: develop
overrides:
  repos:
    meta-foo:
      url: "ssh://g...@bitbucket.com/project/meta-foo.git"
```

The developers will always apply devConfig.yaml when building dev builds and the meta-foo override will only apply for target_1_config.yaml, but not for target_2_config.yaml, because there is no meta-foo in there.

Yes, the same thing can be achieved with the KAS_PREMIRROS, but why use an environment variable, when there is already a mechanism for applying overrides? And we use the dev config anyway.

[Bogdan Prădatu]

Sorry, meta-foo was supposed to be in someOtherConfig.yaml.

[Jan Kiszka]

Ok, the example above looks reasonable from the distance. It will become
fairly verbose to manage URL rewrites this way, specifically if they are
rather uniform (git.yoctoproject.org -> git.internal-mirror.net). OTOH,
overrides is also a fairly intuitive mechanism.

Now the problem with overrides is that it was introduced for being
managed by the dump plugin. It has to be carefully checked if user
provided (and managed) overrides would "just work" and we could drop this:

"Note, that all entries below this key are reserved for auto-generation
using kas plugins. Do not manually add entries."

Felix, thoughts?

[MOESSBAUER, Felix]

Actually, there should be no conflict. The lock plugin only operates on
".lock.(yml|json)" files (which are overwritten on lock --update). As
long as the url override is added to a non-lock file, this will work.

>
> "Note, that all entries below this key are reserved for auto-
> generation
> using kas plugins. Do not manually add entries."
>

This requirement needs to be changed anyways. Since ddb921757
("locking: generalize locking to all included files") we search for
lock files across the whole include tree, whereby the lockfiles for
includes are likely created manually.

We probably should change that requirement to "The overrides.commit
entries must only be set in lockfiles". Once we have the infrastructure
from "Backend changes for distributed inplace lockfile updates" we can
also enforce that rule.

By that, I'm fine with the proposed change. We just need to document,
that the override.url happens prior to the KAS_PREMIRRORS logic and by
that can be overwritten again by KAS_PREMIRRORS:

Felix

>
> Jan

[Jan Kiszka]

Good point.

[Bogdan Prădatu]

So, if I read this correctly, you would agree to the suggested change. How should it be done, I must admit I don't have much experience in contributing to open source (never sent patches anywhere else :) ), so I might need some guidance. Should I prepare a patch and post it for review or will someone else take care of this?

[Pedro @ Elk]

Hello,

I have a similar use case.

For local development, I would like to override some of the repositories.
Instead of using a branch / commit and relying on kas for the checkout I
would like to use a local repo that I check out and control by hand.

This is supported as of now, any repository that doesn't include the url field
is treated as a "local repo".

My current workflow involves commenting out any url and commit fields. It would
be much nicer if a url could be overridden with "null" from a config fragment.

Best,
Pedro 

[Jan Kiszka]

On 09.07.26 19:18, Pedro @ Elk wrote:
> Hello,
>
> I have a similar use case.
>
> For local development, I would like to override some of the repositories.
> Instead of using a branch / commit and relying on kas for the checkout I
> would like to use a local repo that I check out and control by hand.
>
> This is supported as of now, any repository that doesn't include the url
> field
> is treated as a "local repo".
>
> My current workflow involves commenting out any url and commit fields.
> It would
> be much nicer if a url could be overridden with "null" from a config
> fragment.
>

I guess we would need something similar to
https://github.com/siemens/kas/commit/25307f7bc0536860b0322583d912ba76979259a4