Please drive traffic to registry.k8s.io (instead of k8s.gcr.io)

[Davanum Srinivas]

Hi Folks,

Context: As a way to reduce costs for our GCR repositories, we have embarked on a project at sig-k8s-infra to spread the load/cost across cloud providers, we are starting with AWS. Please see [1] and [2] for more info.

Ask: If you are setting up new clusters or in charge of CI jobs for your organization, could you please help drive traffic to registry.k8s.io? For example kubeadm has a `--image-repository` [3]. So essentially wherever you see "k8s.gcr.io" that can be "registry.k8s.io" instead.

Why: we want to see how our new OCI proxy holds up under load [4]

Thanks,

Dims

[1] https://github.com/kubernetes/k8s.io/wiki/New-Registry-url-for-Kubernetes-(registry.k8s.io)

[2] https://github.com/orgs/kubernetes/projects/77/views/2

[3] https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/

[4] https://console.cloud.google.com/monitoring/dashboards/builder/07365a2f-9d9b-41af-860e-a3fa5e4164ad?project=k8s-infra-oci-proxy-prod&dashboardBuilderState=%257B%2522editModeEnabled%2522:false%257D&timeDomain=1m

--

Davanum Srinivas :: https://twitter.com/dims

[Owen Tuz]

Hi Dims,

Thanks for this - it looks like a great initiative and I'm really pleased to see it :)

Do you have a view on whether we should start to migrate public resources (such as Helm charts or tools that reference images on k8s.gcr.io) yet?

Or are we treating this as purely a load test at this stage - ie, please opt-in your own resources if you can, but on the understanding that it's under development?

Thanks,

Owen

--
You received this message because you are subscribed to the Google Groups "dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@kubernetes.io.
To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/CANw6fcHjZNffxZhu9F%2BQMQ9QSB7iAjXADK6Mu_L8UB3zMXXyfQ%40mail.gmail.com.

[Davanum Srinivas]

Owen,

the latter right now. "purely a load test at this stage - ie, please opt-in your own resources if you can, but on the understanding that it's under development"

thanks,

Dims

[Lars Ekman]

Hi,

I get a problem with coredns using this;

kubeadm > docker pull registry.k8s.io/coredns:v1.8.6
Error response from daemon: manifest for registry.k8s.io/coredns:v1.8.6 not found: manifest unknown: Failed to fetch "v1.8.6" from request "/v2/coredns/manifests/v1.8.6".

This is for K8s v1.24.0-rc.0

Best Regards,

Lars Ekman

[Davanum Srinivas]

Lars,

good rule of thumb, if it is NOT in k8s.gcr.io, the registry.k8s.io won't work. So if you check a docker pull against k8s.gcr.io you can see that it is missing there as well.

$ docker pull k8s.gcr.io/coredns:v1.8.6
Error response from daemon: manifest for k8s.gcr.io/coredns:v1.8.6 not found: manifest unknown: Failed to fetch "v1.8.6" from request "/v2/coredns/manifests/v1.8.6".

thanks,

Dims

[Lars Ekman]

Hi,

The problem is weirder:

kubeadm > kubeadm config images list --kubernetes-version v1.24.0-rc.0
k8s.gcr.io/kube-apiserver:v1.24.0-rc.0
k8s.gcr.io/kube-controller-manager:v1.24.0-rc.0
k8s.gcr.io/kube-scheduler:v1.24.0-rc.0
k8s.gcr.io/kube-proxy:v1.24.0-rc.0
k8s.gcr.io/pause:3.7
k8s.gcr.io/etcd:3.5.3-0
k8s.gcr.io/coredns/coredns:v1.8.6

But:

kubeadm > kubeadm config images list --image-repository registry.k8s.io --kubernetes-version v1.24.0-rc.0
registry.k8s.io/kube-apiserver:v1.24.0-rc.0
registry.k8s.io/kube-controller-manager:v1.24.0-rc.0
registry.k8s.io/kube-scheduler:v1.24.0-rc.0
registry.k8s.io/kube-proxy:v1.24.0-rc.0
registry.k8s.io/pause:3.7
registry.k8s.io/etcd:3.5.3-0
registry.k8s.io/coredns:v1.8.6

When "--image-repository registry.k8s.io" is used the coredns url loses a "coredns/" item.

Regards,

L Ekman

[Marko Mudrinić]

This is the expected behavior.

From the Kubernetes 1.22 changelog[1]:

> Kubeadm: when using a custom image repository for CoreDNS kubeadm now will append the coredns image name instead of coredns/coredns, thus restoring the behaviour existing before the v1.21 release. Users who rely on nested folder for the coredns image should set the clusterConfiguration.dns.imageRepository value including the nested path name (e.g using registry.company.xyz/coredns will force kubeadm to use registry.company.xyz/coredns/coredns image). No action is needed if using the default registry (k8s.gcr.io). (#102502, @ykakarap)

[1]: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#bug-or-regression-9

To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/3cad0553-616d-4c2d-b289-b0bf39e49b97n%40kubernetes.io.

[Davanum Srinivas]

Folks,

With the KEP [1] merging and PR 109938 [1] merged as well (and in CI for about 5 days). It's time to call for folks who build tools/installers etc who use k8s.gcr.io to switch to registry.k8s.io.

The older "k8s.gcr.io" will continue to be available for a while to help transition and avoid breaking folks, So please don't panic :) However, please do start switching to the new registry.k8s.io wherever you can, and definitely for k8s 1.25, that will be the default.

If you have tools that have k8s.gcr.io (yamls, manifests, scripts), please pay special attention when 1.25 ships (best to do it before!)

thanks,

Dims

[1] https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/3000-artifact-distribution

[2] https://github.com/kubernetes/kubernetes/pull/109938

[Davanum Srinivas]

wrong image path junel

$ docker pull registry.k8s.io/coredns/coredns:v1.8.4
v1.8.4: Pulling from coredns/coredns
c6568d217a00: Pull complete
bc38a22c706b: Pull complete
Digest: sha256:6e5a02c21641597998b4be7cb5eb1e7b02c0d8d23cce4dd09f4682d463798890
Status: Downloaded newer image for registry.k8s.io/coredns/coredns:v1.8.4
registry.k8s.io/coredns/coredns:v1.8.4

On Sun, Dec 4, 2022 at 4:51 AM junel domingo <juneldo...@gmail.com> wrote:

Hi, how can I point from k8s.gcr.io to registry.k8s.io im getting error in pulling image. Im using cluster API declarative approach. Now I'm wondering were to modify the registry pointing. I couldnt see any registry in my declarative yaml file.

[junel domingo]

Hi Dav, thanks for your reply. That image path was automatically pulled by the api. I didnt mention it. But when I change back the repository in configmap to k8s.grc.io it works really fine. Btw Im using k8s version 1.22 as of the moment. Does this repository issue resolve in the latest version?

[Marko Mudrinić]

What are you seeing is the expected behavior in kubeadm. The CoreDNS image is overridden in the following way when a custom/non-default registry is used: `k8s.gcr.io -> <custom-registry>/coredns` (one `/coredns` is dropped).

`registry.k8s.io` is considered as a custom registry, so the CoreDNS image is overridden to `registry.k8s.io/coredns` instead of `registry.k8s.io/coredns/coredns`.

The upcoming December patch releases will use `registry.k8s.io` as the default registry, so there will be no need to change the registry and therefore you'll not be affected by this behavior.

However, if you want to use the new registry with an earlier patch release that uses `k8s.gcr.io` as the default registry, you can mitigate this issue by adding this to your kubeadm configuration. This will enforce kubeadm to use `registry.k8s.io/coredns/coredns` instead of `registry.k8s.io/coredns`.

```

kind: ClusterConfiguration

apiVersion: kubeadm.k8s.io/v1beta3

...

imageRepository: registry.k8s.io

dns:
  imageRepository: registry.k8s.io/coredns

```

This will also be an issue for users that want to change back to `k8s.gcr.io` for patch releases that use `registry.k8s.io` by default and they can use the same mitigation steps but with `k8s.gcr.io`.

[junel domingo]

Hi Team, thanks for your inputs. I got my problem fixed by downgrading the cluster API components from 1.3.0 to 1.2.0 after reading the CAPI docs.

1.       For Kubernetes version 1.21.x to 1.24.x you should use cluster API components version of 1.0.x to 1.2.x

2.       For Kubernetes version 1.25.x and above you should use cluster API components version of 1.3