Docker image SHA instead of version in kas-container

[Jörg Sommer]

Hello,

is it intended that kas-container pins the image by the version tag and not
the SHA ID? Some proxies might mess with the tag. Wouldn't it be better to
use the SHA ID?

Regards Jörg

--

[Jan Kiszka]

How do you want to resolve the issue that the SHA is only known after I
tagged, built and released? We would have to split up kas-container into
a separate repo to allow its own, slightly delayed release cycle.

But you can always set your own local KAS_IMAGE_VERSION using a SHA instead.

BTW, which kind of proxies could actually cause troubles here? GitHub
itself? Or some fishy middle boxes which you enabled / had to enable to
break your E2E encryption?

Jan

--
Siemens AG, Technology
Linux Expert Center

[Jörg Sommer]

Jan Kiszka schrieb am Mo 03. Jun, 13:09 (GMT):

> On 03.06.24 12:45, 'Jörg Sommer' via kas-devel wrote:
> > Hello,
> >
> > is it intended that kas-container pins the image by the version tag and not
> > the SHA ID? Some proxies might mess with the tag. Wouldn't it be better to
> > use the SHA ID?
> How do you want to resolve the issue that the SHA is only known after I
> tagged, built and released? We would have to split up kas-container into
> a separate repo to allow its own, slightly delayed release cycle.

Oh yes, I didn't thought of the chicken-egg-problem.

But would it be possible to sign the images?

https://www.howtogeek.com/devops/how-to-sign-your-docker-images-to-increase-trust/

% docker trust inspect ghcr.io/siemens/kas/kas:4.4
[]
no signatures or cannot access ghcr.io/siemens/kas/kas:4.4

> But you can always set your own local KAS_IMAGE_VERSION using a SHA
> instead.

Sounds good.

> BTW, which kind of proxies could actually cause troubles here? GitHub
> itself? Or some fishy middle boxes which you enabled / had to enable to
> break your E2E encryption?

A self-build solution for image caching and pinning and fancy foobar. It
took me hours for another image to find out that the tag in this proxy of
our customer did not match the tag online.

--

[Jan Kiszka]

On 04.06.24 08:25, 'Jörg Sommer' via kas-devel wrote:
> Jan Kiszka schrieb am Mo 03. Jun, 13:09 (GMT):
>> On 03.06.24 12:45, 'Jörg Sommer' via kas-devel wrote:
>>> Hello,
>>>
>>> is it intended that kas-container pins the image by the version tag and not
>>> the SHA ID? Some proxies might mess with the tag. Wouldn't it be better to
>>> use the SHA ID?
>> How do you want to resolve the issue that the SHA is only known after I
>> tagged, built and released? We would have to split up kas-container into
>> a separate repo to allow its own, slightly delayed release cycle.
>
> Oh yes, I didn't thought of the chicken-egg-problem.
>
> But would it be possible to sign the images?
>
> https://www.howtogeek.com/devops/how-to-sign-your-docker-images-to-increase-trust/
>
> % docker trust inspect ghcr.io/siemens/kas/kas:4.4
> []
> no signatures or cannot access ghcr.io/siemens/kas/kas:4.4
>

This looks architecturally broken to me, asking me to load my private
key into docker (docker trust key load). But maybe my colleagues have
better understanding here.

>
>> But you can always set your own local KAS_IMAGE_VERSION using a SHA
>> instead.
>
> Sounds good.
>
>> BTW, which kind of proxies could actually cause troubles here? GitHub
>> itself? Or some fishy middle boxes which you enabled / had to enable to
>> break your E2E encryption?
>
> A self-build solution for image caching and pinning and fancy foobar. It
> took me hours for another image to find out that the tag in this proxy of
> our customer did not match the tag online.
>

OK, I see.

[Cirujano Cuesta, Silvano]

> On 4. Jun 2024, at 10:35, Kiszka, Jan (T CED) <jan.k...@siemens.com> wrote:
>
> On 04.06.24 08:25, 'Jörg Sommer' via kas-devel wrote:
>> Jan Kiszka schrieb am Mo 03. Jun, 13:09 (GMT):
>>> On 03.06.24 12:45, 'Jörg Sommer' via kas-devel wrote:
>>>> Hello,
>>>>
>>>> is it intended that kas-container pins the image by the version tag and not
>>>> the SHA ID? Some proxies might mess with the tag. Wouldn't it be better to
>>>> use the SHA ID?
>>> How do you want to resolve the issue that the SHA is only known after I
>>> tagged, built and released? We would have to split up kas-container into
>>> a separate repo to allow its own, slightly delayed release cycle.
>>
>> Oh yes, I didn't thought of the chicken-egg-problem.
>>
>> But would it be possible to sign the images?
>>
>> https://www.howtogeek.com/devops/how-to-sign-your-docker-images-to-increase-trust/
>>
>> % docker trust inspect ghcr.io/siemens/kas/kas:4.4
>> []
>> no signatures or cannot access ghcr.io/siemens/kas/kas:4.4
>>
>
> This looks architecturally broken to me, asking me to load my private
> key into docker (docker trust key load). But maybe my colleagues have
> better understanding here.

Jan, is your concern adding your private key to your local Docker instance?

Or is your concern that “docker trust” might send your private key to the Docker Hub?

In both cases some investigation would be need to:
- an alternative without using docker toolchain could be investigated
- understand what the tool itself does (hopefully OSS) to gain some trust on it

Being container images built up as Merkle trees, in theory you only need to sign the top-level manifest.
You shouldn’t need to pull the whole image, but I have to admit I don’t know how the Docker CLI does it.

>
>>
>>> But you can always set your own local KAS_IMAGE_VERSION using a SHA
>>> instead.
>>
>> Sounds good.
>>
>>> BTW, which kind of proxies could actually cause troubles here? GitHub
>>> itself? Or some fishy middle boxes which you enabled / had to enable to
>>> break your E2E encryption?
>>
>> A self-build solution for image caching and pinning and fancy foobar. It
>> took me hours for another image to find out that the tag in this proxy of
>> our customer did not match the tag online.
>>
>
> OK, I see.

A self-built solution for image caching? That sounds scary…

Jörg, did you considered the usage of a local container registry in proxy mode?
It should take care of it all transparently and reliably.
The CNCF registry can only proxy from one registry, but Harbor appears to support multiple upstream registries.

Tags are not as reliable as SHAs, that’s right.
Still I would say that having tags being modified by the proxy should be considered a bug

Silvano

[Jan Kiszka]

On 05.06.24 10:49, Cirujano Cuesta, Silvano (T CED OES-DE) wrote:
>
>
>> On 4. Jun 2024, at 10:35, Kiszka, Jan (T CED) <jan.k...@siemens.com> wrote:
>>
>> On 04.06.24 08:25, 'Jörg Sommer' via kas-devel wrote:
>>> Jan Kiszka schrieb am Mo 03. Jun, 13:09 (GMT):
>>>> On 03.06.24 12:45, 'Jörg Sommer' via kas-devel wrote:
>>>>> Hello,
>>>>>
>>>>> is it intended that kas-container pins the image by the version tag and not
>>>>> the SHA ID? Some proxies might mess with the tag. Wouldn't it be better to
>>>>> use the SHA ID?
>>>> How do you want to resolve the issue that the SHA is only known after I
>>>> tagged, built and released? We would have to split up kas-container into
>>>> a separate repo to allow its own, slightly delayed release cycle.
>>>
>>> Oh yes, I didn't thought of the chicken-egg-problem.
>>>
>>> But would it be possible to sign the images?
>>>
>>> https://www.howtogeek.com/devops/how-to-sign-your-docker-images-to-increase-trust/
>>>
>>> % docker trust inspect ghcr.io/siemens/kas/kas:4.4
>>> []
>>> no signatures or cannot access ghcr.io/siemens/kas/kas:4.4
>>>
>>
>> This looks architecturally broken to me, asking me to load my private
>> key into docker (docker trust key load). But maybe my colleagues have
>> better understanding here.
>
> Jan, is your concern adding your private key to your local Docker instance?
>

Sure. That cannot work with mine (HSM...). Where is the PGP integration?
Or smartcard support? That's why I mean this is broken (not well thought
through).

[Jörg Sommer]

Cirujano Cuesta, Silvano schrieb am Mi 05. Jun, 08:49 (GMT):

> > On 4. Jun 2024, at 10:35, Kiszka, Jan (T CED) <jan.k...@siemens.com> wrote:
> > On 04.06.24 08:25, 'Jörg Sommer' via kas-devel wrote:
> >> Jan Kiszka schrieb am Mo 03. Jun, 13:09 (GMT):

> >>> BTW, which kind of proxies could actually cause troubles here? GitHub
> >>> itself? Or some fishy middle boxes which you enabled / had to enable to
> >>> break your E2E encryption?
> >>
> >> A self-build solution for image caching and pinning and fancy foobar. It
> >> took me hours for another image to find out that the tag in this proxy of
> >> our customer did not match the tag online.
> >>
> >
> > OK, I see.
>
> A self-built solution for image caching? That sounds scary…
>
> Jörg, did you considered the usage of a local container registry in proxy mode?

Unfortunately, I can't change anything on their setup.

> Tags are not as reliable as SHAs, that’s right.
> Still I would say that having tags being modified by the proxy should be considered a bug

The proxy did not modify the tag. It was changed upstream and the proxy
didn't update and still delivered the old version.

Since then, I have had a certain mistrust of tags. ;-)


Jörg

--

[Cirujano Cuesta, Silvano]

TL;DR: the setup for image signing is much too complex for KAS, I’d discard it.

Docker relies on Notary for signing, which is inspired on the TUF specification.
Therefore images are not being signed with your private key, but so-called tagging keys.
Tagging keys are delegations of your HSM private key.

Here you can get some superficial understanding of the concept:
https://docs.docker.com/engine/security/trust/#docker-content-trust-keys

I don’t want to get into the details. Because in fact I doubt that the pretty complex setup is worth for KAS alone.
In notary (as well as with any TUF implementation) you need a service to manage the metadata.
Docker Hub provides one for those images hosted there, but AFAIK GitHub doesn’t.

Another possible approach is the usage of CoSign, which also needs an additional service to work (there is a publicly available Fulcio instance) and is not well integrated into Docker, as Notary.
It is like the 2-step (download, verify) workflow that you follow when checking the signature of a manually downloaded file.
This set-up is less complex than the Notary one, but still somehow complex and less elegant.

Silvano

—

[Cirujano Cuesta, Silvano]

> On 5. Jun 2024, at 11:55, Jörg Sommer <joerg....@navimatix.de> wrote:
>
> Cirujano Cuesta, Silvano schrieb am Mi 05. Jun, 08:49 (GMT):
>>> On 4. Jun 2024, at 10:35, Kiszka, Jan (T CED) <jan.k...@siemens.com> wrote:
>>> On 04.06.24 08:25, 'Jörg Sommer' via kas-devel wrote:
>>>> Jan Kiszka schrieb am Mo 03. Jun, 13:09 (GMT):
>>>>> BTW, which kind of proxies could actually cause troubles here? GitHub
>>>>> itself? Or some fishy middle boxes which you enabled / had to enable to
>>>>> break your E2E encryption?
>>>>
>>>> A self-build solution for image caching and pinning and fancy foobar. It
>>>> took me hours for another image to find out that the tag in this proxy of
>>>> our customer did not match the tag online.
>>>>
>>>
>>> OK, I see.
>>
>> A self-built solution for image caching? That sounds scary…
>>
>> Jörg, did you considered the usage of a local container registry in proxy mode?
>
> Unfortunately, I can't change anything on their setup.

Bad luck then on that end.

>
>> Tags are not as reliable as SHAs, that’s right.
>> Still I would say that having tags being modified by the proxy should be considered a bug
>
> The proxy did not modify the tag. It was changed upstream and the proxy
> didn't update and still delivered the old version.
>
> Since then, I have had a certain mistrust of tags. ;-)
>

Tags are human-friendly, mutable references without any guarantee on content.
Digests are human-unfriendly, immutable references with content guarantee.

If you are focused on the content and you know what you want to get, digests are better.
But if you want to let “others” direct you to the “right” content, then tags are better.

In the case of KAS, using tags helps you to rely on what the KAS-community tells to be the best fitting content for a tag.
If you use digests, the KAS-community won’t be able to “redirect” you new bug-fixed images for a specific tag.

In the end, it’s like referring to Git commits by digest or by tag.
Local and remote tags might loose synchronization ¯\_(ツ)_/¯

Something like the Renovate-Bot can help you “pinning” by digest, remaining aware of changes in the tags that you might want to get.

Silvano

—

Siemens AG, Technology
Linux Expert Center

>

> Jörg
>
> --