[ACTION REQUIRED] k8s.gcr.io moving from gcr.io/google-containers to gcr.io/k8s-artifacts-prod in early April

[Linus Arver]

You can stop reading now if you do not own or depend on any images in k8s.gcr.io.

Hello,

As you may know, the k8s.gcr.io name is currently a facade for gcr.io/google-containers, which is only writeable by Google employees.

On April 1, 2020* the k8s.gcr.io vanity domain will flip to gcr.io/k8s-artifacts-prod, which is community owned. Just before the flip, gcr.io/google-containers will enter a read-only freeze.

For a seamless transition for existing Kubernetes configurations, all existing images in the google-containers project will be backfilled into gcr.io/k8s-artifacts-prod. Any new ones that pop up in google-containers leading up to the flip will also be backfilled.

*This is a tentative date and may be shifted depending on findings from the Release Managers group. 

Why are you doing this?

The domain flip transitions from Google-operated to community-operated infrastructure for official Kubernetes image hosting.

How can I prepare?

If you own images that belong to the Kubernetes OSS community, please follow the directions to create your own staging repo if you have not already. This is because ALL images promoted into gcr.io/k8s-artifacts-prod ARE REQUIRED to use the promotion process described here (and configured here).

I need more time, help!

Please feel free to ask questions on any of the following forums:

• Slack: #wg-k8s-infra

• wg-k8s-infra Mailing list

• k8s.io Issues

 

Can I see this happen live?

The domain flip event will be done live in a “War Room”. This event will be broadcasted in a Zoom meeting. Links to the meeting will be sent to the Kubernetes Slack channel #wg-k8s-infra on the day of the flip. 

--

Linus Arver

Cloud Release Team

[Stephen Augustus]

Thanks so much, Linus.

Looking forward to working with you on this! :)

-- Stephen

--
You received this message because you are subscribed to the Google Groups "Kubernetes Release Team" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-releas...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-release-team/CAD61eHxDNT%3DP_2JqXePK7TkdRPJmv72OrM%3Dmgmzj8ZJdVjW8QQ%40mail.gmail.com.

[Stephen Augustus]

p.s. The original tracking issue for this is here: https://github.com/kubernetes/release/issues/270

If you have any questions, comments, or concerns, please drop them in the issue, so we can track in one place.

-- Stephen

[Kris Nova]

If like to volunteer some time on this if anyone needs any one off issues worked on - thanks for doing this - it’s been a long time coming!

Takk Fyrir,

Nóva

On 3 Mar 2020, at 18:28, Stephen Augustus <steph...@agst.us> wrote:



You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAOqU-DTwN6X5Z82JmWE7ic8Jz_rxewJkjLyDpLbnowVm41wknQ%40mail.gmail.com.

[lin...@google.com]

Pinging this thread again to remind folks that this is still on track for early April (roughly 2 weeks from now).

As Stephen suggested, please go to https://github.com/kubernetes/release/issues/270 for questions/comments/concerns.

Thanks!

On Tuesday, March 3, 2020 at 5:15:17 PM UTC-8 Kris Nova wrote:

If like to volunteer some time on this if anyone needs any one off issues worked on - thanks for doing this - it’s been a long time coming!

Takk Fyrir,

Nóva

On 3 Mar 2020, at 18:28, Stephen Augustus <steph...@agst.us> wrote:



p.s. The original tracking issue for this is here: https://github.com/kubernetes/release/issues/270

If you have any questions, comments, or concerns, please drop them in the issue, so we can track in one place.

-- Stephen

On Tue, Mar 3, 2020, 18:49 Stephen Augustus <steph...@agst.us> wrote:

Thanks so much, Linus.

Looking forward to working with you on this! :)

-- Stephen

On Tue, Mar 3, 2020 at 5:35 PM 'Linus Arver' via Kubernetes Release Team <kubernetes-release-team@googlegroups.com> wrote:

You can stop reading now if you do not own or depend on any images in k8s.gcr.io.

Hello,

As you may know, the k8s.gcr.io name is currently a facade for gcr.io/google-containers, which is only writeable by Google employees.

On April 1, 2020* the k8s.gcr.io vanity domain will flip to gcr.io/k8s-artifacts-prod, which is community owned. Just before the flip, gcr.io/google-containers will enter a read-only freeze.

For a seamless transition for existing Kubernetes configurations, all existing images in the google-containers project will be backfilled into gcr.io/k8s-artifacts-prod. Any new ones that pop up in google-containers leading up to the flip will also be backfilled.

*This is a tentative date and may be shifted depending on findings from the Release Managers group. 

Why are you doing this?

The domain flip transitions from Google-operated to community-operated infrastructure for official Kubernetes image hosting.

How can I prepare?

If you own images that belong to the Kubernetes OSS community, please follow the directions to create your own staging repo if you have not already. This is because ALL images promoted into gcr.io/k8s-artifacts-prod ARE REQUIRED to use the promotion process described here (and configured here).

I need more time, help!

Please feel free to ask questions on any of the following forums:

• Slack: #wg-k8s-infra

• wg-k8s-infra Mailing list

• k8s.io Issues

 

Can I see this happen live?

The domain flip event will be done live in a “War Room”. This event will be broadcasted in a Zoom meeting. Links to the meeting will be sent to the Kubernetes Slack channel #wg-k8s-infra on the day of the flip. 

--

Linus Arver

Cloud Release Team

--
You received this message because you are subscribed to the Google Groups "Kubernetes Release Team" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-release-team+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-release-team/CAD61eHxDNT%3DP_2JqXePK7TkdRPJmv72OrM%3Dmgmzj8ZJdVjW8QQ%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.

[Linus Arver]

Pinging this thread again to remind folks that this is still on track for early April (roughly 2 weeks from now).

As Stephen suggested, please go to https://github.com/kubernetes/release/issues/270 for questions/comments/concerns.

Thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/E4AAA9FE-4F5D-42B7-8D60-FA7E9488D6B8%40nivenly.com.

[Linus Arver]

The domain flip has commenced! The flip will happen over a course of ~3.5 days to fully complete.

Attached are some graphs that show evidence of the change:

 

As you can see there is a large uptick in the US region for k8s-artifacts-prod. This is expected as the change is first being pushed out to US regions.

I will post an update again tomorrow (Thurs) and Friday.

[Linus Arver]

The rollouts have been proceeding smoothly so far. Below is another graph. There has been an uptick for the EU region because the rollout has been triggered as well.

[Linus Arver]

Rollout still in progress. So far so good. Updated graph:

[Linus Arver]

Hi all,

We found a hard-coded internal reference that we had missed which caused some large fraction of traffic from certain regions to be sent to the wrong place. Because it is Friday afternoon, we will halt the rollout, do a fast rollback, and try again early next week.  Sorry!

Stay tuned!

[Davanum Srinivas]

thanks for the heads up Linus!

You received this message because you are subscribed to the Google Groups "kubernetes-sig-release" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-re...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-release/CAD61eHz5sXTDPSwJd6vc81CuJ0bE_HAHa7pxG5rTpFXD2vyG3w%40mail.gmail.com.

--

Davanum Srinivas :: https://twitter.com/dims

[Linus Arver]

The k8s.gcr.io vanity-domain flip has been fully rolled back for now (k8s.gcr.io == {asia,eu,us}.gcr.io/google-containers).

We found several internal couplings that we didn’t know about, which this flip broke. Mid-way through the rollout these were brought to our attention and we rolled back immediately. We are in the process of disentangling these, which we expect this to be complete within 2 weeks. We will update everyone as it progresses.

While we intend to never do another flip, such assertions have proven wrong before. In order to test this sort of flip better, we’ll be adding a second vanity domain - canary-k8s.gcr.io - which will always be changed BEFORE k8s.gcr.io. Anyone who wants to run tests before the real name is flipped can do pulls against this new name.

Thanks for your patience!

[Linus Arver]

Hello all,

We are in the middle of disentangling the internal dependencies on the old google-containers project. Due to the age of the project and the size of Google’s codebase, we are encountering new findings as we perform a deep search and contact teams regarding each find. We expect to finish the introspection by the end of the month, and the flip to happen in May.

Stay tuned for more updates.

Thanks!

[Stephen Augustus]

Thanks for the update (and your continued work on this), Linus!!

-- Stephen

[Linus Arver]

Hello all,

We are wrapping up what we believe to be the final pieces of work to allow us to re-attempt the vanity domain flip. We are now targeting June 15th for the next attempt.

Thank you for your patience!

--

Linus Arver

AMP Release Engineering

[Stephen Augustus]

Thank you Linus for continuing to shepherd this on the Google side! Looking forward to the cutover.

-- Stephen

[Stephen Augustus]

My apologies for not realizing this earlier...

I think we're going to need to push VDF until after the next scheduled patch releases (tentative 6/17).

We've got a few changes on those branches, including (already disclosed) CVE fixes, and it would worry me to have our container repository in a transitive state during that time.

Perhaps we can target the week after?

-- Stephen

[Linus Arver]

Sounds good. Let's target Monday June 22 instead.

[Stephen Augustus]

Great. Thanks again, Linus!

-- Stephen

[Linus Arver]

Hello all,

We are still targeting Monday, June 22, for the next flip attempt.

We are not planning on organizing a live Zoom meeting as we did previously on the first attempt, as there is no new information to disseminate. Updates will be posted on this thread on the day of the flip, and daily after that while the changes are rolled out to production. The rollout is expected to finish by the end of next week.

[Davanum Srinivas]

Thanks for the heads up Linus!

You received this message because you are subscribed to the Google Groups "kubernetes-sig-release" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-re...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-release/CAD61eHyHYJ0t74oJEhMCJi2hYBD-PGXFZu9pjpQws9DAx5%3DUSg%40mail.gmail.com.

[Linus Arver]

Hello all,

The domain flip was initiated earlier this morning, but it was rolled back after ~5 hours due to a possible billing issue. More updates to come later this week.

[Linus Arver]

Hello all,

The billing issues we had last time have been resolved! We are now targeting Monday July 13 for the next flip attempt. Please contact lin...@google.com if you have any questions (@listx on slack.k8s.io).

[Stephen Augustus]

Hey Linus,

We have patch releases on 7/15, so we'll need to schedule this flip to happen the week afterwards (Monday, 7/20).

-- Stephen

[Linus Arver]

Thanks for the reminder Stephen. However, I don't think that is a strict blocker. As part of the patch releases on 7/15 we can perform another backfill of everything in google-containers (including images for the new patch releases) into k8s-artifacts-prod.

The backfills are a straightforward process and will ensure that clients hitting either the old or new project will see the same set of images. Speaking of backfills, this PR (to be merged before we start the flip on Monday) is something similar to what you can expect on 7/15, where I'd be happy to create another backfill PR.

If you still object, I will postpone the flip; it's just that I see an easy workaround and it seems reasonable to run with it.

Thoughts?

[Stephen Augustus]

(Spoke with Linus on Slack, but sending here in more detail...)

The cutover also requires work from the Release Engineering team to ensure that containers can be written to the new location during our stage/release process.

I have a PR opened here, which is a revert of the first time we did this, but it needs to be rebased, slightly modified, and tested again against the current k/release code.

That combined with the patch releases and impending Go security releases make this a less-than-ideal week to add another infra shift to the mix.

Let's target next week.

-- Stephen

[Linus Arver]

Moving this thread here. Please follow up on the new thread, thanks!