[PATCH 1/3] introduce container registry login
B. Niedermayr
Create a container registry login file that is currently compatible with
docker, podman and skopeo.
When REGISTRY_AUTH_FILE is set then use that file and copy it into the
kas home directory.
If CI_REGISTRY, CI_REGISTRY_USER and CI_JOB_TOKEN is set then
the required login data is added to the login file.
Signed-off-by: Benedikt Niedermayr <benedikt....@siemens.com>
---
kas/libcmds.py | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/kas/libcmds.py b/kas/libcmds.py
index db28eef7344f..09d55fe3a9d5 100644
--- a/kas/libcmds.py
+++ b/kas/libcmds.py
@@ -29,6 +29,8 @@ import shutil
import os
import pprint
import configparser
+import json
+import base64
from git.config import GitConfigParser
from .libkas import (ssh_cleanup_agent, ssh_setup_agent, ssh_no_host_key_check,
get_build_environ, repos_fetch, repos_apply_patches)
@@ -168,6 +170,7 @@ class SetupHome(Command):
'AWS_SHARED_CREDENTIALS_FILE',
'AWS_WEB_IDENTITY_TOKEN_FILE',
'NETRC_FILE',
+ 'REGISTRY_AUTH_FILE',
]
def __init__(self):
@@ -220,6 +223,25 @@ class SetupHome(Command):
'login gitlab-ci-token\n'
'password ' + os.environ['CI_JOB_TOKEN'] + '\n')
+ def _setup_registry_auth(self):
+ if os.environ.get('REGISTRY_AUTH_FILE', False):
+ os.mkdir(self.tmpdirname + "/.docker")
+ shutil.copy(os.environ['REGISTRY_AUTH_FILE'],
+ self.tmpdirname + "/.docker/config.json")
+ if os.environ.get('CI_REGISTRY', False) \
+ and os.environ.get('CI_JOB_TOKEN', False) \
+ and os.environ.get('CI_REGISTRY_USER', False):
+ with open(self.tmpdirname + '/.docker/config.json', 'r+') as fds:
+ data = json.loads(fds.read())
+ token = os.environ['CI_JOB_TOKEN']
+ base64_token = base64.b64encode(token.encode()).decode()
+ auths = data.get('auths', {})
+ auths.update({os.environ['CI_REGISTRY']: {"auth": base64_token}})
+ data['auths'] = auths
+ fds.seek(0)
+ fds.write(json.dumps(data, indent=4))
+ fds.truncate()
+
def _setup_aws_creds(self):
aws_dir = self.tmpdirname + "/.aws"
conf_file = aws_dir + "/config"
@@ -290,6 +312,7 @@ class SetupHome(Command):
logging.info(f'Running on {ci}')
def_umask = os.umask(0o077)
self._setup_netrc()
+ self._setup_registry_auth()
self._setup_gitconfig()
self._setup_aws_creds()
os.umask(def_umask)
--
2.34.1
MOESSBAUER, Felix
> From: Benedikt Niedermayr <benedikt....@siemens.com>
>
> Create a container registry login file that is currently compatible
> with
> docker, podman and skopeo.
> When REGISTRY_AUTH_FILE is set then use that file and copy it into
> the
> kas home directory.
>
> If CI_REGISTRY, CI_REGISTRY_USER and CI_JOB_TOKEN is set then
> the required login data is added to the login file.
Hi,
thanks for these patches. To keep things atomic, please also move the
documentation changes into this commit.
Shouldn't this be an either or? Only if the file is not present, check
for the gitlab env. That's at least how it is done with the
CI_JOB_TOKEN in setup_gitconfig().
> + if os.environ.get('CI_REGISTRY', False) \
> + and os.environ.get('CI_JOB_TOKEN', False) \
> + and os.environ.get('CI_REGISTRY_USER', False):
> + with open(self.tmpdirname + '/.docker/config.json',
> 'r+') as fds:
> + data = json.loads(fds.read())
> + token = os.environ['CI_JOB_TOKEN']
> + base64_token =
> base64.b64encode(token.encode()).decode()
> + auths = data.get('auths', {})
> + auths.update({os.environ['CI_REGISTRY']: {"auth":
> base64_token}})
> + data['auths'] = auths
> + fds.seek(0)
> + fds.write(json.dumps(data, indent=4))
> + fds.truncate()
As we truncate here, we likely need to open the file with 'w+', IIRC.
Felix
> +
> def _setup_aws_creds(self):
> aws_dir = self.tmpdirname + "/.aws"
> conf_file = aws_dir + "/config"
> @@ -290,6 +312,7 @@ class SetupHome(Command):
> logging.info(f'Running on {ci}')
> def_umask = os.umask(0o077)
> self._setup_netrc()
> + self._setup_registry_auth()
> self._setup_gitconfig()
> self._setup_aws_creds()
> os.umask(def_umask)
> --
> 2.34.1
>
--
Siemens AG, Technology
Linux Expert Center
B. Niedermayr
Create a container registry authentication file that is
When REGISTRY_AUTH_FILE is set then use that file and copy it into the
kas home directory.
If CI_REGISTRY, CI_REGISTRY_USER and CI_JOB_TOKEN is set then
the required login data is added to the login file.
docs/command-line/environment-variables.inc | 17 ++++++++++++---
docs/userguide/credentials.rst | 24 +++++++++++++++++++--
kas/libcmds.py | 23 ++++++++++++++++++++
3 files changed, 59 insertions(+), 5 deletions(-)
diff --git a/docs/command-line/environment-variables.inc b/docs/command-line/environment-variables.inc
index fce53e69229a..044310b9a571 100644
--- a/docs/command-line/environment-variables.inc
+++ b/docs/command-line/environment-variables.inc
@@ -132,14 +132,25 @@ Variables Glossary
| ``NETRC_FILE`` | Path to a .netrc file which will be copied to |
| (K,C) | the kas home dir as .netrc. |
+--------------------------+--------------------------------------------------+
+| ``REGISTRY_AUTH_FILE`` | Path to a container registry authentication file.|
+| (K,C) | |
++--------------------------+--------------------------------------------------+
| ``CI_SERVER_HOST`` | Environment variables from GitLab CI, if set |
| ``CI_JOB_TOKEN`` | .netrc is configured to allow fetching from |
| ``CI_JOB_URL`` | the GitLab instance. An entry will be appended |
-| (K) | in case ``NETRC_FILE`` was given as well. Note |
-| | that if the file already contains an entry for |
-| | that host most tools would probably take that |
+| ``CI_REGISTRY`` | in case ``NETRC_FILE`` was given as well. Note |
+| ``CI_REGISTRY_USER`` | that if the file already contains an entry for |
+| (K) | that host most tools would probably take that |
| | first one. The job url is added to the |
| | provenance attestation (if enabled). |
+| | If ``CI_REGISTRY`` and ``CI_REGISTRY_USER`` is |
+| | also set, a container registry login file is |
+| | created, which is used by docker, podman and |
+| | skopeo. In case ``REGISTRY_AUTH_FILE`` was given |
+| | as well, the CI login data will be appended to |
+| | that file. |
+| | The required base64 encoded login data can be |
+| | generated by kas. |
+--------------------------+--------------------------------------------------+
| ``GITHUB_ACTIONS`` | Environment variables from GitHub actions or |
| ``GITLAB_CI`` | GitLab CI. If set to `true`, `.gitconfig` is |
diff --git a/docs/userguide/credentials.rst b/docs/userguide/credentials.rst
index 1ce84e7e4fb6..7423352f0f7f 100644
--- a/docs/userguide/credentials.rst
+++ b/docs/userguide/credentials.rst
@@ -49,8 +49,9 @@ GitLab CI
~~~~~~~~~
When running in the GitLab CI, the ``CI_JOB_TOKEN`` can be used to access
-git repositories via https. kas automatically adds this token to the
-``.netrc`` file, where it is picked up by git. Further, kas configures git
+git repositories via https. If ``CI_SERVER_HOST`` is also set,
+kas automatically adds this token to the ``.netrc`` file,
+where it is picked up by git. Further, kas configures git
to automatically rewrite the urls of the repositories to clone via https
for repos stored on the same server. Technically this is achieved by adding
`insteadof` entries to the ``.gitconfig`` file.
@@ -58,10 +59,29 @@ for repos stored on the same server. Technically this is achieved by adding
For backwards compatibility, the git rewrite rules are only added if
``.gitconfig`` does not exists and no SSH configuration is provided.
+If the ``CI_REGISTRY``, ``CI_REGISTRY_USER`` and ``CI_JOB_TOKEN`` variables
+are set, kas automatically creates a login file for the container
+registry at ``~/.docker/config.json``. This file is compatible with
+docker, podman and even skopeo.
+
+In other words, if ``CI_REGISTRY``, ``CI_REGISTRY_USER`` and ``CI_JOB_TOKEN`` are
+set, kas creates the container registry login file.
+If ``CI_JOB_TOKEN`` and ``CI_SERVER_HOST`` are set, kas creates the ``.netrc`` file.
+
.. note::
Make sure to assign the correct permissions to the ``CI_JOB_TOKEN``.
For details, see `GitLab CI/CD job token <https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html>`_.
+Container Registry Authentication File
+--------------------------------------
+
+A file that kas saves as ``.docker/config.json`` in the kas home directory.
+It contains credentials for the container registry login.
+The syntax is specified `here <https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md>`_.
+The authentication file is compatible with docker, podman and skopeo.
+When running in the GitLab CI, the ``CI_JOB_TOKEN`` is appended to
+automatically grant access to the container registry.
+
Netrc File
----------
+ and os.environ.get('CI_JOB_TOKEN', False) \
+ and os.environ.get('CI_REGISTRY_USER', False):
+ with open(self.tmpdirname + '/.docker/config.json', 'r+') as fds:
+ data = json.loads(fds.read())
+ token = os.environ['CI_JOB_TOKEN']
+ base64_token = base64.b64encode(token.encode()).decode()
+ auths = data.get('auths', {})
+ auths.update({os.environ['CI_REGISTRY']: {"auth": base64_token}})
+ data['auths'] = auths
+ fds.seek(0)
+ fds.write(json.dumps(data, indent=4))
+ fds.truncate()
B. Niedermayr
Create a container registry authentication file that is
currently compatible with docker, podman and skopeo.
The format is specified here [1].
When REGISTRY_AUTH_FILE is set then use that file and copy it into the
kas home directory.
If CI_REGISTRY, CI_REGISTRY_USER and CI_JOB_TOKEN is set then
the required login data is added to the login file.
[1] https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md
Signed-off-by: Benedikt Niedermayr <benedikt....@siemens.com>
---
docs/userguide/credentials.rst | 24 ++++++++++++++++--
kas/libcmds.py | 28 +++++++++++++++++++++
3 files changed, 64 insertions(+), 5 deletions(-)
+++ b/kas/libcmds.py
@@ -29,6 +29,8 @@ import shutil
import os
import pprint
import configparser
+import json
+import base64
from git.config import GitConfigParser
from .libkas import (ssh_cleanup_agent, ssh_setup_agent, ssh_no_host_key_check,
get_build_environ, repos_fetch, repos_apply_patches)
@@ -168,6 +170,7 @@ class SetupHome(Command):
'AWS_SHARED_CREDENTIALS_FILE',
'AWS_WEB_IDENTITY_TOKEN_FILE',
'NETRC_FILE',
+ 'REGISTRY_AUTH_FILE',
]
def __init__(self):
'password ' + os.environ['CI_JOB_TOKEN'] + '\n')
+ def _setup_registry_auth(self):
+ with open(self.tmpdirname + "/.docker/config.json", 'w') as fds:
+ fds.write("{}")
+
+ self.tmpdirname + "/.docker/config.json")
+ if os.environ.get('CI_REGISTRY', False) \
+ and os.environ.get('CI_JOB_TOKEN', False) \
+ and os.environ.get('CI_REGISTRY_USER', False):
+ with open(self.tmpdirname + '/.docker/config.json', 'r+') as fds:
+ data = json.loads(fds.read())
+ token = os.environ['CI_JOB_TOKEN']
+ base64_token = base64.b64encode(token.encode()).decode()
+ auths = data.get('auths', {})
+ auths.update({os.environ['CI_REGISTRY']: {"auth": base64_token}})
+ data['auths'] = auths
+ fds.seek(0)
+ fds.write(json.dumps(data, indent=4))
+ fds.truncate()
+
def _setup_aws_creds(self):
aws_dir = self.tmpdirname + "/.aws"
conf_file = aws_dir + "/config"
Jan Kiszka
> +It contains credentials for the container registry login.
> +The syntax is specified `here <https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md>`_.
"here" is not a nice URL description
> +The authentication file is compatible with docker, podman and skopeo.
> +When running in the GitLab CI, the ``CI_JOB_TOKEN`` is appended to
> +automatically grant access to the container registry.
first and then overwrite it with the provided content.
Niedermayr, BENEDIKT
>
> > +It contains credentials for the container registry login.
> > +The syntax is specified `here
> > <https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md>`_.
>
> "The syntax follows the `containers-auth.json specification <url>`_,
> "here" is not a nice URL description
>
> > +The authentication file is compatible with docker, podman and skopeo.
> > +When running in the GitLab CI, the ``CI_JOB_TOKEN`` is appended to
> > +automatically grant access to the container registry.
>
> "...to automatically grant access according to the job permissions."
>
Ok. Actually copied it from the netrc section...
That was exactly what I fixed in V3. Because in case "REGISTRY_AUTH_FILE" was not set before,
open(..., 'r+') raised an "FileNotFoundError".
Changing the access mode from 'r+' to 'a+' behaves strange when I write to the file.
It discards any seek operation and continues to write at the end, so not usable even though it
creates the file if not present. If I use 'w+' then persistent content is always deleted, so I can't
load the content into my "data" variable...
What I can change here is to create the file only if it doesn't exist and if any of the CI_*
variables is set.
Like this:
if os.environ.get('CI_REGISTRY', False) \
and os.environ.get('CI_JOB_TOKEN', False) \
and os.environ.get('CI_REGISTRY_USER', False):
if not os.path.exists(self.tmpdirname + "/.docker/config.json"):
with open(self.tmpdirname + "/.docker/config.json", 'w') as fds:
fds.write("{}")
with open(...) as fds:
...
Would that be better?
Benedikt
MOESSBAUER, Felix
wrote:
Ok, then please also fix that there as well (in a dedicated cleanup
commit). Having the URLs in the text is anyways better, as these
otherwise are hidden in the manpage output.
Felix
--
Jan Kiszka
[...]
if os.environ.get('REGISTRY_AUTH_FILE', False):
self.tmpdirname + "/.docker/config.json")
elif not os.path.exists(self.tmpdirname + "/.docker/config.json"):
fds.write("{}")
Niedermayr, BENEDIKT
Seems I got you wrong.
Benedikt
> Jan
>
Niedermayr, BENEDIKT
below the respective sections (or bottom of the page)?
Benedikt
MOESSBAUER, Felix
In case the URL is not super-long, I prefer inline.
Felix
--