Please drive traffic to registry.k8s.io (instead of k8s.gcr.io)

1770 views
Skip to first unread message

Davanum Srinivas

unread,
Apr 20, 2022, 10:54:28 AM4/20/22
to d...@kubernetes.io
Hi Folks,

Context: As a way to reduce costs for our GCR repositories, we have embarked on a project at sig-k8s-infra to spread the load/cost across cloud providers, we are starting with AWS. Please see [1] and [2] for more info.

Ask: If you are setting up new clusters or in charge of CI jobs for your organization, could you please help drive traffic to registry.k8s.io? For example kubeadm has a `--image-repository` [3]. So essentially wherever you see "k8s.gcr.io" that can be "registry.k8s.io" instead.

Why: we want to see how our new OCI proxy holds up under load [4]

Thanks,
Dims

Owen Tuz

unread,
Apr 20, 2022, 11:23:12 AM4/20/22
to dav...@gmail.com, d...@kubernetes.io
Hi Dims,

Thanks for this - it looks like a great initiative and I'm really pleased to see it :)

Do you have a view on whether we should start to migrate public resources (such as Helm charts or tools that reference images on k8s.gcr.io) yet?

Or are we treating this as purely a load test at this stage - ie, please opt-in your own resources if you can, but on the understanding that it's under development?

Thanks,

Owen

--
You received this message because you are subscribed to the Google Groups "dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@kubernetes.io.
To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/CANw6fcHjZNffxZhu9F%2BQMQ9QSB7iAjXADK6Mu_L8UB3zMXXyfQ%40mail.gmail.com.

Davanum Srinivas

unread,
Apr 20, 2022, 11:26:41 AM4/20/22
to Owen Tuz, d...@kubernetes.io
Owen,

the latter right now. "purely a load test at this stage - ie, please opt-in your own resources if you can, but on the understanding that it's under development"

thanks,
Dims

Lars Ekman

unread,
Apr 21, 2022, 3:24:29 AM4/21/22
to dev, dav...@gmail.com, d...@kubernetes.io, owe...@gmail.com
Hi,

I get a problem with coredns using this;

kubeadm > docker pull registry.k8s.io/coredns:v1.8.6
Error response from daemon: manifest for registry.k8s.io/coredns:v1.8.6 not found: manifest unknown: Failed to fetch "v1.8.6" from request "/v2/coredns/manifests/v1.8.6".

This is for K8s v1.24.0-rc.0

Best Regards,
Lars Ekman

Davanum Srinivas

unread,
Apr 21, 2022, 6:54:05 AM4/21/22
to Lars Ekman, d...@kubernetes.io, owe...@gmail.com
Lars,

good rule of thumb, if it is NOT in k8s.gcr.io, the registry.k8s.io won't work. So if you check a docker pull against k8s.gcr.io you can see that it is missing there as well.

$ docker pull k8s.gcr.io/coredns:v1.8.6
Error response from daemon: manifest for k8s.gcr.io/coredns:v1.8.6 not found: manifest unknown: Failed to fetch "v1.8.6" from request "/v2/coredns/manifests/v1.8.6".

thanks,
Dims

Lars Ekman

unread,
Apr 21, 2022, 7:27:17 AM4/21/22
to dev, dav...@gmail.com, d...@kubernetes.io, owe...@gmail.com, Lars Ekman
Hi,

The problem is weirder:


But:


When "--image-repository registry.k8s.io" is used the coredns url loses a "coredns/" item.

Regards,
L Ekman

Marko Mudrinić

unread,
Apr 21, 2022, 7:30:37 AM4/21/22
to lars.g...@est.tech, dav...@gmail.com, d...@kubernetes.io, owe...@gmail.com
This is the expected behavior.

From the Kubernetes 1.22 changelog[1]:

> Kubeadm: when using a custom image repository for CoreDNS kubeadm now will append the coredns image name instead of coredns/coredns, thus restoring the behaviour existing before the v1.21 release. Users who rely on nested folder for the coredns image should set the clusterConfiguration.dns.imageRepository value including the nested path name (e.g using registry.company.xyz/coredns will force kubeadm to use registry.company.xyz/coredns/coredns image). No action is needed if using the default registry (k8s.gcr.io). (#102502, @ykakarap)



To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/3cad0553-616d-4c2d-b289-b0bf39e49b97n%40kubernetes.io.

Davanum Srinivas

unread,
Jun 5, 2022, 4:42:41 PM6/5/22
to d...@kubernetes.io
Folks,

With the KEP [1] merging and PR 109938 [1] merged as well (and in CI for about 5 days). It's time to call for folks who build tools/installers etc who use k8s.gcr.io to switch to registry.k8s.io.

The older "k8s.gcr.io" will continue to be available for a while to help transition and avoid breaking folks, So please don't panic :) However, please do start switching to the new registry.k8s.io wherever you can, and definitely for k8s 1.25, that will be the default.

If you have tools that have k8s.gcr.io (yamls, manifests, scripts), please pay special attention when 1.25 ships (best to do it before!)

thanks,
Dims

Davanum Srinivas

unread,
Dec 4, 2022, 8:06:50 AM12/4/22
to junel domingo, dev
wrong image path junel

$ docker pull registry.k8s.io/coredns/coredns:v1.8.4
v1.8.4: Pulling from coredns/coredns
c6568d217a00: Pull complete
bc38a22c706b: Pull complete
Digest: sha256:6e5a02c21641597998b4be7cb5eb1e7b02c0d8d23cce4dd09f4682d463798890
Status: Downloaded newer image for registry.k8s.io/coredns/coredns:v1.8.4
registry.k8s.io/coredns/coredns:v1.8.4

On Sun, Dec 4, 2022 at 4:51 AM junel domingo <juneldo...@gmail.com> wrote:
Hi, how can I point from k8s.gcr.io to registry.k8s.io im getting error in pulling image. Im using cluster API declarative approach. Now I'm wondering were to modify the registry pointing. I couldnt see any registry in my declarative yaml file.
error_pull_image.jpg

junel domingo

unread,
Dec 4, 2022, 8:55:58 AM12/4/22
to Davanum Srinivas, dev
Hi Dav, thanks for your reply. That image path was automatically pulled by the api. I didnt mention it. But when I change back the repository in configmap to k8s.grc.io it works really fine. Btw Im using k8s version 1.22 as of the moment. Does this repository issue resolve in the latest version?

Marko Mudrinić

unread,
Dec 5, 2022, 7:25:17 AM12/5/22
to dev, junel domingo, dev, dav...@gmail.com
What are you seeing is the expected behavior in kubeadm. The CoreDNS image is overridden in the following way when a custom/non-default registry is used: `k8s.gcr.io -> <custom-registry>/coredns` (one `/coredns` is dropped).
`registry.k8s.io` is considered as a custom registry, so the CoreDNS image is overridden to `registry.k8s.io/coredns` instead of `registry.k8s.io/coredns/coredns`.

The upcoming December patch releases will use `registry.k8s.io` as the default registry, so there will be no need to change the registry and therefore you'll not be affected by this behavior.

However, if you want to use the new registry with an earlier patch release that uses `k8s.gcr.io` as the default registry, you can mitigate this issue by adding this to your kubeadm configuration. This will enforce kubeadm to use `registry.k8s.io/coredns/coredns` instead of `registry.k8s.io/coredns`.

```
kind: ClusterConfiguration
...
imageRepository: registry.k8s.io
dns:
  imageRepository: registry.k8s.io/coredns
```

This will also be an issue for users that want to change back to `k8s.gcr.io` for patch releases that use `registry.k8s.io` by default and they can use the same mitigation steps but with `k8s.gcr.io`.

junel domingo

unread,
Dec 6, 2022, 12:31:59 AM12/6/22
to Marko Mudrinić, dev, dav...@gmail.com
Hi Team, thanks for your inputs. I got my problem fixed by downgrading the cluster API components from 1.3.0 to 1.2.0 after reading the CAPI docs.

1.       For Kubernetes version 1.21.x to 1.24.x you should use cluster API components version of 1.0.x to 1.2.x

2.       For Kubernetes version 1.25.x and above you should use cluster API components version of 1.3 

Reply all
Reply to author
Forward
0 new messages