manual upgrade/downgrade testing for Kubernetes 1.5

[David Oppenheimer]

We didn't quite finish automating the manual upgrade/downgrade tests in time for 1.5, so we're doing manual upgrade/downgrade testing again for 1.5.

We're distributing the work across Google sites and the community. If you're on the Cc list of this email (except Saad), then you have signed up to do one or more of the tests. (Nobody was "volunteered" without their accepting the job.) The assignments are recorded in a spreadsheet which you should also use to record a summary of the results (in addition to recording results in the instructions doc, as last time).

The instructions doc is here.

There area couple of changes this time around, mentioned at the top of the doc, which I will copy here:

In 1.4 when testing minion upgrade we only tested 1.3 ContainerVM (CVM) -> 1.4 CVM. For 1.5 we are testing for minions

• 1.4 CVM -> 1.5 CVM (and downgrade)

• 1.4 CVM -> 1.5 GCI (and downgrade)

• 1.4 GCI -> 1.5 GCI (and downgrade)

Also, we now have a spreadsheet for recording test result summaries. Please record a summary of your results in this spreadsheet in addition to writing the notes in this doc.

Saad has informed me that the beta to test should be available by the end of this week. Once it is available, please run the tests and record your results, finishing before next Thursday if at all possible.

If you have any questions, please let me know.

Thanks to everyone for their help!

[Saad Ali]

> Saad has informed me that the beta to test should be available by the end of this week. Once it is available, please run the tests and record your results, finishing before next Thursday if at all possible.

I

​ will​ reply to this thread once the 1.5 beta is cut. Currently blocked on this issue.

[Tomasz 'Zen' Napierala]

Hi David,

Is the automation effort tracked somewhere? We might try to help here, but it would be good what the overall progress is and where the gaps are.

REgards,

> --
> You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
> To post to this group, send email to kuberne...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdPWU9S2eY4hytXrqm9EA-xXkr2guG64tkzu1%3DbH7YZ3g%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

--
Tomasz 'Zen' Napierala
Kubernetes Engineering - Poland

[David Oppenheimer]

https://github.com/kubernetes/kubernetes/pull/35697

is a PR with the first round.

https://docs.google.com/spreadsheets/d/1BBriwwyAtp4EzjE53lwyKXcvUNE808DozXiNEZfjtFI/edit#gid=1876580779&vpid=A1

is a spreadsheet with the overall plan and status, Quinton owns the spreadsheet and I've asked him to share it with the list (I can't do it).

On Fri, Nov 18, 2016 at 3:45 AM, Tomasz 'Zen' Napierala <tnapi...@mirantis.com> wrote:

Hi David,

Is the automation effort tracked somewhere? We might try to help here, but it would be good what the overall progress is and where the gaps are.

REgards,

> On 17 Nov 2016, at 22:09, 'David Oppenheimer' via Kubernetes developer/contributor discussion <kubernetes-dev@googlegroups.com> wrote:
>
> We didn't quite finish automating the manual upgrade/downgrade tests in time for 1.5, so we're doing manual upgrade/downgrade testing again for 1.5.
>
> We're distributing the work across Google sites and the community. If you're on the Cc list of this email (except Saad), then you have signed up to do one or more of the tests. (Nobody was "volunteered" without their accepting the job.) The assignments are recorded in a spreadsheet which you should also use to record a summary of the results (in addition to recording results in the instructions doc, as last time).
>
> The instructions doc is here.
>
> There area couple of changes this time around, mentioned at the top of the doc, which I will copy here:
>
> In 1.4 when testing minion upgrade we only tested 1.3 ContainerVM (CVM) -> 1.4 CVM. For 1.5 we are testing for minions
>       • 1.4 CVM -> 1.5 CVM (and downgrade)
>       • 1.4 CVM -> 1.5 GCI (and downgrade)
>       • 1.4 GCI -> 1.5 GCI (and downgrade)
>
> Also, we now have a spreadsheet for recording test result summaries. Please record a summary of your results in this spreadsheet in addition to writing the notes in this doc.
>
> Saad has informed me that the beta to test should be available by the end of this week. Once it is available, please run the tests and record your results, finishing before next Thursday if at all possible.
>
> If you have any questions, please let me know.
>
> Thanks to everyone for their help!
>
>

> --
> You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
> To post to this group, send email to kubernetes-dev@googlegroups.com.

[Saad Ali]

>> Saad has informed me that the beta to test should be available by the end of this week. Once it is available, please run the tests and record your results, finishing before next Thursday if at all possible.

> 

> I​ will​ reply to this thread once the 1.5 beta is cut. Currently blocked on this issue.

The v1.5.0-beta.1 release is live. You can start testing against it. If you discover any issues, create a GitHub issue, make sure it has the 1.5 milestone, and add a blocker or non-blocker label.

> To post to this group, send email to kuberne...@googlegroups.com.

[Rajat Ramesh Koujalagi]

Has anyone been able to successfully run the cluster/gce/upgrade.sh script ?

It's taking a long time for me (~40 mins) and it hasn't yet completed.

I ran the script from 1.5 branch.

cluster/gce/upgrade.sh -M v1.5.0-beta.1

[Rajat Ramesh Koujalagi]

Created an issue:

https://github.com/kubernetes/kubernetes/issues/37257

[Robert Bailey]

I'm going to start looking into this today. 

To post to this group, send email to kubernetes-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CA%2Bd%3DP%2B%2BptC5H-LSB3mJJh-X2dKRYJWtdK%3DON%2BcQP8L9ijDxQuw%40mail.gmail.com.

[David Oppenheimer]

Thanks!

On Tue, Nov 22, 2016 at 3:21 PM, Robert Bailey <robert...@google.com> wrote:

I'm going to start looking into this today. 

[Saad Ali]

The v1.5.0-beta.2 has been cut. It should have Robert's fix. Please give it a shot.

[Robert Bailey]

Master upgrades now work. Node upgrades are broken. See https://github.com/kubernetes/kubernetes/issues/37474.

[David Oppenheimer]

Hi folks. Just a reminder that we urgently need someone to work on

https://github.com/kubernetes/kubernetes/issues/37474

as it is blocking manual upgrade testing.

[Robert Bailey]

I believe that I found the issue I'd encountered (and that it is mac-specific). If anyone has time to test my PR and/or test the upgrade path on linux that'd be really helpful. 

To post to this group, send email to kubernetes-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdq1%2BnKoROz68nvDpfM2K1L4BSiz_XFNAL5fFotTh3KLg%40mail.gmail.com.

[David Oppenheimer]

I tried to build the latest release-1.4 to test your theory and almost finished but got the following from 'make quick-release'

+++ [1128 11:07:19] Building go targets for linux/amd64:

    vendor/github.com/onsi/ginkgo/ginkgo

    test/e2e_node/e2e_node.test

+++ [1128 11:08:31] Running build command....

+++ [1128 11:08:32] Output directory is local.  No need to copy results out.

+++ [1128 11:08:32] Building tarball: src

+++ [1128 11:08:32] Building tarball: manifests

+++ [1128 11:08:32] Building tarball: salt

+++ [1128 11:08:32] Starting tarball: client linux-amd64

+++ [1128 11:08:32] Building tarball: server linux-amd64

+++ [1128 11:08:32] Waiting on tarballs

+++ [1128 11:08:42] Starting Docker build for image: kube-apiserver

+++ [1128 11:08:42] Starting Docker build for image: kube-controller-manager

+++ [1128 11:08:42] Starting Docker build for image: kube-scheduler

+++ [1128 11:08:42] Starting Docker build for image: kube-proxy

Error response from daemon: unexpected EOF

Error response from daemon: unexpected EOF

Error response from daemon: unexpected EOF

+++ [1128 11:09:02] Deleting docker image gcr.io/google_containers/kube-proxy:74b27d56d4717d0c3ba3e46d4944353b

Untagged: gcr.io/google_containers/kube-proxy:74b27d56d4717d0c3ba3e46d4944353b

Deleted: sha256:a417305ac8d8a27e43bb6b841011adbdaa86b1b3239c04ec66bad4daa1bd49c3

Deleted: sha256:7c58b8614d826974e6a6ae060bb4e44818b53162f8509c15a15c589418055cab

!!! [1128 11:09:03] previous Docker build failed

!!! [1128 11:09:03] previous tarball phase failed

I'm running Docker 1.12.3 on Ubuntu

On Mon, Nov 28, 2016 at 10:21 AM, Robert Bailey <robert...@google.com> wrote:

I believe that I found the issue I'd encountered (and that it is mac-specific). If anyone has time to test my PR and/or test the upgrade path on linux that'd be really helpful. 

[David Oppenheimer]

@piosz suggested to do

sudo docker rmi -f $(docker images -q)

before building, which solved the problem.

I'm going to check if upgrade.sh works now.

[David Oppenheimer]

Node upgrade worked for me on Ubuntu. So this appears to be a mac-specific problem as Robby theorized.

[Prashanth B]

>1.4 CVM -> 1.5 GCI (and downgrade)

Running the command in the doc (`cluster/gce/upgrade.sh -N v1.5.0-beta.2 -o gci`) from the 1.5 branch still leaves my nodes running "osImage: Debian GNU/Linux 7 (wheezy)". PBKAC? 

[David Oppenheimer]

You're right, there's a bug in the instructions. Thanks for noticing! (And thanks to Vish for figuring out what the problem is.)

It should be "KUBE_NODE_OS_DISTRIBUTION=XXX; cluster/gce/upgrade.sh -N v1.5.0-beta.2 -o"

I've updated the instructions. Anyone who already ran the tests will unfortunately need to do them again for the CVM->GCI upgrade/downgrade only. (The CVM->CVM and GCI->GCI paths happen to work correctly using the original instructions.)

[Prashanth B]

Actually that doesn't work either. I already tried it, as well as hard coding a bunch of env vars into config-default for NODE_OS_DISTRIBUTION, KUBE_OS_DISTRIBUTION and KUBE_NODE_OS_DISTRIBUTION (basically shotgun everything that has even been assigned a os value in the setup scripts).

There are various ways to detect a gci node, but probably the most straightforward is to: 

```

$ kubectl get no e2e-test-beeps-minion-group-i3fk -o yaml | grep -i image

  images:

    osImage: Google Container-VM Image

```

To post to this group, send email to kubernetes-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzf6MK5EzF1PHGH1h8SYjgoTcD_31mwnVByMy8T6Yq6w%2BQ%40mail.gmail.com.

[Michael Taufen]

The upgrade script also prints the os images of the master and nodes before and after the upgrade.

On Wed, Nov 30, 2016 at 10:08 AM, 'Prashanth B' via Kubernetes developer/contributor discussion <kuberne...@googlegroups.com> wrote:

Actually that doesn't work either. I already tried it, as well as hard coding a bunch of env vars into config-default for NODE_OS_DISTRIBUTION, KUBE_OS_DISTRIBUTION and KUBE_NODE_OS_DISTRIBUTION (basically shotgun everything that has even been assigned a os value in the setup scripts).

There are various ways to detect a gci node, but probably the most straightforward is to: 

```

$ kubectl get no e2e-test-beeps-minion-group-i3fk -o yaml | grep -i image

  images:

    osImage: Google Container-VM Image

```

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzf6MK5EzF1PHGH1h8SYjgoTcD_31mwnVByMy8T6Yq6w%2BQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAD3YSY_-RSG5r5CGLKpT_CO1G7G1MSU139JmqyfX7kFovMXFDA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Michael Taufen

MTV SWE

[Jeff Peeler]

FWIW, I can confirm that the new node instructions aren't working. (I
also updated the spreadsheet with the other scenario results completed
so far.)

>>>>>>>>>>>>>>>>> > from it, send an email to kubernetes-de...@googlegroups.com.

>>>>>>>>>>>>>>>>> > To post to this group, send email to
>>>>>>>>>>>>>>>>> > kuberne...@googlegroups.com.
>>>>>>>>>>>>>>>>> > To view this discussion on the web visit
>>>>>>>>>>>>>>>>> > https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdPWU9S2eY4hytXrqm9EA-xXkr2guG64tkzu1%3DbH7YZ3g%40mail.gmail.com.
>>>>>>>>>>>>>>>>> > For more options, visit
>>>>>>>>>>>>>>>>> > https://groups.google.com/d/optout.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Tomasz 'Zen' Napierala
>>>>>>>>>>>>>>>>> Kubernetes Engineering - Poland
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>> Google Groups "Kubernetes developer/contributor discussion" group.
>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from

>>>>>>>>>>>>> it, send an email to kubernetes-de...@googlegroups.com.

>>>>>>>>>>>>> To post to this group, send email to
>>>>>>>>>>>>> kuberne...@googlegroups.com.
>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CA%2Bd%3DP%2B%2BptC5H-LSB3mJJh-X2dKRYJWtdK%3DON%2BcQP8L9ijDxQuw%40mail.gmail.com.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,

>>>>>>>> send an email to kubernetes-de...@googlegroups.com.

>>>>>>>> To post to this group, send email to
>>>>>>>> kuberne...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdq1%2BnKoROz68nvDpfM2K1L4BSiz_XFNAL5fFotTh3KLg%40mail.gmail.com.
>>>>>>>>
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Kubernetes developer/contributor discussion" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to kubernetes-de...@googlegroups.com.

>> To post to this group, send email to kuberne...@googlegroups.com.
>> To view this discussion on the web visit

[David Oppenheimer]

OK we're going to look into it, very sorry for the trouble.

>>>>>>>>>>>>>>>>> > from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.

>>>>>>>>>>>>>>>>> > To post to this group, send email to

>>>>>>>>>>>>>>>>> > kubernetes-dev@googlegroups.com.

>>>>>>>>>>>>>>>>> > To view this discussion on the web visit
>>>>>>>>>>>>>>>>> > https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdPWU9S2eY4hytXrqm9EA-xXkr2guG64tkzu1%3DbH7YZ3g%40mail.gmail.com.
>>>>>>>>>>>>>>>>> > For more options, visit
>>>>>>>>>>>>>>>>> > https://groups.google.com/d/optout.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Tomasz 'Zen' Napierala
>>>>>>>>>>>>>>>>> Kubernetes Engineering - Poland
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>> Google Groups "Kubernetes developer/contributor discussion" group.
>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from

>>>>>>>>>>>>> it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.

>>>>>>>>>>>>> To post to this group, send email to

>>>>>>>>>>>>> kubernetes-dev@googlegroups.com.

>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CA%2Bd%3DP%2B%2BptC5H-LSB3mJJh-X2dKRYJWtdK%3DON%2BcQP8L9ijDxQuw%40mail.gmail.com.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,

>>>>>>>> send an email to kubernetes-dev+unsubscribe@googlegroups.com.

>>>>>>>> To post to this group, send email to

>>>>>>>> kubernetes-dev@googlegroups.com.

>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdq1%2BnKoROz68nvDpfM2K1L4BSiz_XFNAL5fFotTh3KLg%40mail.gmail.com.
>>>>>>>>
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Kubernetes developer/contributor discussion" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to kubernetes-dev+unsubscribe@googlegroups.com.
>> To post to this group, send email to kubernetes-dev@googlegroups.com.

[Michael Taufen]

Here's an even easier way to inspect the os images and some other info; this is essentially what the upgrade script uses to print them:

kubectl get nodes -o=jsonpath='{range .items[*]}name: "{.metadata.name}", osImage: "{.status.nodeInfo.osImage}", kubeletVersion: "{.status.nodeInfo.kubeletVersion}"{"\n"}{end}'

>>>>>>>>>>>>>>>>> > kuberne...@googlegroups.com.

>>>>>>>>>>>>>>>>> > To view this discussion on the web visit
>>>>>>>>>>>>>>>>> > https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdPWU9S2eY4hytXrqm9EA-xXkr2guG64tkzu1%3DbH7YZ3g%40mail.gmail.com.
>>>>>>>>>>>>>>>>> > For more options, visit
>>>>>>>>>>>>>>>>> > https://groups.google.com/d/optout.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Tomasz 'Zen' Napierala
>>>>>>>>>>>>>>>>> Kubernetes Engineering - Poland
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>> Google Groups "Kubernetes developer/contributor discussion" group.
>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>>>> it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
>>>>>>>>>>>>> To post to this group, send email to

>>>>>>>>>>>>> kuberne...@googlegroups.com.

>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CA%2Bd%3DP%2B%2BptC5H-LSB3mJJh-X2dKRYJWtdK%3DON%2BcQP8L9ijDxQuw%40mail.gmail.com.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to kubernetes-dev+unsubscribe@googlegroups.com.
>>>>>>>> To post to this group, send email to

>>>>>>>> kuberne...@googlegroups.com.

>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdq1%2BnKoROz68nvDpfM2K1L4BSiz_XFNAL5fFotTh3KLg%40mail.gmail.com.
>>>>>>>>
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Kubernetes developer/contributor discussion" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to kubernetes-dev+unsubscribe@googlegroups.com.

>> To post to this group, send email to kuberne...@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzf6MK5EzF1PHGH1h8SYjgoTcD_31mwnVByMy8T6Yq6w%2BQ%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzeJvjdCmc_x-1owCPKLV0KVT9X-K9cOuit7-C-FAP0bOA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Michael Taufen

MTV SWE

[Anthony Yeh]

Has anyone tried the following:

export KUBE_NODE_OS_DISTRIBUTION=gci

kube-upgrade.sh -No v1.5.0-beta.2

Note the "o" flag is required to make KUBE_NODE_OS_DISTRIBUTION take effect, and it must come before the positional arguments (the version specifier).

>>>>>>>>>>>>>>>>> > from it, send an email to kubernetes-de...@googlegroups.com.

>>>>>>>>>>>>>>>>> > To post to this group, send email to
>>>>>>>>>>>>>>>>> > kuberne...@googlegroups.com.
>>>>>>>>>>>>>>>>> > To view this discussion on the web visit
>>>>>>>>>>>>>>>>> > https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdPWU9S2eY4hytXrqm9EA-xXkr2guG64tkzu1%3DbH7YZ3g%40mail.gmail.com.
>>>>>>>>>>>>>>>>> > For more options, visit
>>>>>>>>>>>>>>>>> > https://groups.google.com/d/optout.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Tomasz 'Zen' Napierala
>>>>>>>>>>>>>>>>> Kubernetes Engineering - Poland
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>> Google Groups "Kubernetes developer/contributor discussion" group.
>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from

>>>>>>>>>>>>> it, send an email to kubernetes-de...@googlegroups.com.

>>>>>>>>>>>>> To post to this group, send email to
>>>>>>>>>>>>> kuberne...@googlegroups.com.
>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CA%2Bd%3DP%2B%2BptC5H-LSB3mJJh-X2dKRYJWtdK%3DON%2BcQP8L9ijDxQuw%40mail.gmail.com.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,

>>>>>>>> send an email to kubernetes-de...@googlegroups.com.

>>>>>>>> To post to this group, send email to
>>>>>>>> kuberne...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdq1%2BnKoROz68nvDpfM2K1L4BSiz_XFNAL5fFotTh3KLg%40mail.gmail.com.
>>>>>>>>
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Kubernetes developer/contributor discussion" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to kubernetes-de...@googlegroups.com.

>> To post to this group, send email to kuberne...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzf6MK5EzF1PHGH1h8SYjgoTcD_31mwnVByMy8T6Yq6w%2BQ%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.

To post to this group, send email to kuberne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzeJvjdCmc_x-1owCPKLV0KVT9X-K9cOuit7-C-FAP0bOA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Michael Taufen

MTV SWE

--

You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.

To post to this group, send email to kuberne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAHJadQLux%2B4X9bjg22QfOR1bYqfuXuQa4035bO11vt51646zxQ%40mail.gmail.com.

[David Oppenheimer]

Sorry I should have mentioned on this thread, we have an issue open with discussion of how to fix this:

https://github.com/kubernetes/kubernetes/issues/37715

On Wed, Nov 30, 2016 at 3:12 PM, Anthony Yeh <eni...@google.com> wrote:

Has anyone tried the following:

export KUBE_NODE_OS_DISTRIBUTION=gci

kube-upgrade.sh -No v1.5.0-beta.2

Note the "o" flag is required to make KUBE_NODE_OS_DISTRIBUTION take effect, and it must come before the positional arguments (the version specifier).

On Wed, Nov 30, 2016 at 11:48 AM 'Michael Taufen' via Kubernetes developer/contributor discussion <kubernetes-dev@googlegroups.com> wrote:

Here's an even easier way to inspect the os images and some other info; this is essentially what the upgrade script uses to print them:

kubectl get nodes -o=jsonpath='{range .items[*]}name: "{.metadata.name}", osImage: "{.status.nodeInfo.osImage}", kubeletVersion: "{.status.nodeInfo.kubeletVersion}"{"\n"}{end}'

>>>>>>>>>>>>>>>>> > from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.

>>>>>>>>>>>>>>>>> > To post to this group, send email to

>>>>>>>>>>>>>>>>> > kubernetes-dev@googlegroups.com.

>>>>>>>>>>>>>>>>> > To view this discussion on the web visit
>>>>>>>>>>>>>>>>> > https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdPWU9S2eY4hytXrqm9EA-xXkr2guG64tkzu1%3DbH7YZ3g%40mail.gmail.com.
>>>>>>>>>>>>>>>>> > For more options, visit
>>>>>>>>>>>>>>>>> > https://groups.google.com/d/optout.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Tomasz 'Zen' Napierala
>>>>>>>>>>>>>>>>> Kubernetes Engineering - Poland
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>> Google Groups "Kubernetes developer/contributor discussion" group.
>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from

>>>>>>>>>>>>> it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.

>>>>>>>>>>>>> To post to this group, send email to

>>>>>>>>>>>>> kubernetes-dev@googlegroups.com.

>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CA%2Bd%3DP%2B%2BptC5H-LSB3mJJh-X2dKRYJWtdK%3DON%2BcQP8L9ijDxQuw%40mail.gmail.com.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,

>>>>>>>> send an email to kubernetes-dev+unsubscribe@googlegroups.com.

>>>>>>>> To post to this group, send email to

>>>>>>>> kubernetes-dev@googlegroups.com.

>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdq1%2BnKoROz68nvDpfM2K1L4BSiz_XFNAL5fFotTh3KLg%40mail.gmail.com.
>>>>>>>>
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Kubernetes developer/contributor discussion" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to kubernetes-dev+unsubscribe@googlegroups.com.
>> To post to this group, send email to kubernetes-dev@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzf6MK5EzF1PHGH1h8SYjgoTcD_31mwnVByMy8T6Yq6w%2BQ%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzeJvjdCmc_x-1owCPKLV0KVT9X-K9cOuit7-C-FAP0bOA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Michael Taufen

MTV SWE

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.

[David Oppenheimer]

tl;dr: Manual upgrade testing works now, just follow the slightly-modified instructions in the doc.

We found the problem, and it doesn't require any code changes, just an update to the instructions on how to run the manual upgrade tests. I've updates the instructions and it should work now if you follow the instructions as they appear in the doc.

Thanks and sorry for the inconvenience.

>>>>>>>>>>>>>>>>> > kuberne...@googlegroups.com.

>>>>>>>>>>>>>>>>> > To view this discussion on the web visit
>>>>>>>>>>>>>>>>> > https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdPWU9S2eY4hytXrqm9EA-xXkr2guG64tkzu1%3DbH7YZ3g%40mail.gmail.com.
>>>>>>>>>>>>>>>>> > For more options, visit
>>>>>>>>>>>>>>>>> > https://groups.google.com/d/optout.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Tomasz 'Zen' Napierala
>>>>>>>>>>>>>>>>> Kubernetes Engineering - Poland
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>> Google Groups "Kubernetes developer/contributor discussion" group.
>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>>>> it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
>>>>>>>>>>>>> To post to this group, send email to

>>>>>>>>>>>>> kuberne...@googlegroups.com.

>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CA%2Bd%3DP%2B%2BptC5H-LSB3mJJh-X2dKRYJWtdK%3DON%2BcQP8L9ijDxQuw%40mail.gmail.com.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to kubernetes-dev+unsubscribe@googlegroups.com.
>>>>>>>> To post to this group, send email to

>>>>>>>> kuberne...@googlegroups.com.

>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdq1%2BnKoROz68nvDpfM2K1L4BSiz_XFNAL5fFotTh3KLg%40mail.gmail.com.
>>>>>>>>
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Kubernetes developer/contributor discussion" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to kubernetes-dev+unsubscribe@googlegroups.com.

>> To post to this group, send email to kuberne...@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzf6MK5EzF1PHGH1h8SYjgoTcD_31mwnVByMy8T6Yq6w%2BQ%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.

To post to this group, send email to kuberne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzeJvjdCmc_x-1owCPKLV0KVT9X-K9cOuit7-C-FAP0bOA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Michael Taufen

MTV SWE

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.

To post to this group, send email to kuberne...@googlegroups.com.

[Jeff Peeler]

(Apologies for emailing everybody directly, but these tests take so
much time to run confirming all is well quickly is necessary.)

I left a question on the test instructions document (verifying results
section) asking about the expected pod status after a node
upgrade/downgrade. I'm seeing some pods still running after a node
upgrade, but more often they aren't. My testing notes are available
here:

https://docs.google.com/document/d/1iWyeDmP0Sx2xIATydzYlkhHvD5R_eFG5QmQXmKRXz6Y/edit?usp=sharing

If you scroll to the bottom and look at the pod survival summary, you
can see the pod outcomes thus far. (Good news is the master
upgrade/downgrades seem to be good!)

Hopefully there aren't any mistakes, but the matrix of combinations
has made it hard to juggle all the details.

>>>>>> > developer/contributor discussion <kuberne...@googlegroups.com>

>>>>>> >>>>>>> <kuberne...@googlegroups.com> wrote:
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> Hi folks. Just a reminder that we urgently need someone to
>>>>>> >>>>>>>> work on
>>>>>> >>>>>>>> https://github.com/kubernetes/kubernetes/issues/37474
>>>>>> >>>>>>>> as it is blocking manual upgrade testing.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> On Fri, Nov 25, 2016 at 12:00 AM, Robert Bailey
>>>>>> >>>>>>>> <robert...@google.com> wrote:
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Master upgrades now work. Node upgrades are broken. See
>>>>>> >>>>>>>>> https://github.com/kubernetes/kubernetes/issues/37474.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> On Thu, Nov 24, 2016 at 10:37 PM, Saad Ali
>>>>>> >>>>>>>>> <saa...@google.com>
>>>>>> >>>>>>>>> wrote:
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> The v1.5.0-beta.2 has been cut. It should have Robert's
>>>>>> >>>>>>>>>> fix.
>>>>>> >>>>>>>>>> Please give it a shot.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> On Tue, Nov 22, 2016 at 3:33 PM, David Oppenheimer
>>>>>> >>>>>>>>>> <davi...@google.com> wrote:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Thanks!
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> On Tue, Nov 22, 2016 at 3:21 PM, Robert Bailey
>>>>>> >>>>>>>>>>> <robert...@google.com> wrote:
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> I'm going to start looking into this today.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> On Mon, Nov 21, 2016 at 5:53 PM, 'Rajat Ramesh Koujalagi'
>>>>>> >>>>>>>>>>>> via
>>>>>> >>>>>>>>>>>> Kubernetes developer/contributor discussion

>>>>>> >>>>>>>>>>>>>>>>> > kubernetes-de...@googlegroups.com.

>>>>>> >>>>>>>>>>>>>>>>> > To post to this group, send email to
>>>>>> >>>>>>>>>>>>>>>>> > kuberne...@googlegroups.com.
>>>>>> >>>>>>>>>>>>>>>>> > To view this discussion on the web visit
>>>>>> >>>>>>>>>>>>>>>>> >
>>>>>> >>>>>>>>>>>>>>>>> > https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdPWU9S2eY4hytXrqm9EA-xXkr2guG64tkzu1%3DbH7YZ3g%40mail.gmail.com.
>>>>>> >>>>>>>>>>>>>>>>> > For more options, visit
>>>>>> >>>>>>>>>>>>>>>>> > https://groups.google.com/d/optout.
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>> --
>>>>>> >>>>>>>>>>>>>>>>> Tomasz 'Zen' Napierala
>>>>>> >>>>>>>>>>>>>>>>> Kubernetes Engineering - Poland
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> --
>>>>>> >>>>>>>>>>>>> You received this message because you are subscribed to
>>>>>> >>>>>>>>>>>>> the
>>>>>> >>>>>>>>>>>>> Google Groups "Kubernetes developer/contributor
>>>>>> >>>>>>>>>>>>> discussion" group.
>>>>>> >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails
>>>>>> >>>>>>>>>>>>> from
>>>>>> >>>>>>>>>>>>> it, send an email to

>>>>>> >>>>>>>>>>>>> kubernetes-de...@googlegroups.com.

>>>>>> >>>>>>>>>>>>> To post to this group, send email to
>>>>>> >>>>>>>>>>>>> kuberne...@googlegroups.com.
>>>>>> >>>>>>>>>>>>> To view this discussion on the web visit
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CA%2Bd%3DP%2B%2BptC5H-LSB3mJJh-X2dKRYJWtdK%3DON%2BcQP8L9ijDxQuw%40mail.gmail.com.
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> For more options, visit
>>>>>> >>>>>>>>>>>>> https://groups.google.com/d/optout.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> --
>>>>>> >>>>>>>> You received this message because you are subscribed to the
>>>>>> >>>>>>>> Google
>>>>>> >>>>>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>>>> >>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>> >>>>>>>> it,

>>>>>> >>>>>>>> send an email to kubernetes-de...@googlegroups.com.

>>>>>> >>>>>>>> To post to this group, send email to
>>>>>> >>>>>>>> kuberne...@googlegroups.com.
>>>>>> >>>>>>>> To view this discussion on the web visit
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdq1%2BnKoROz68nvDpfM2K1L4BSiz_XFNAL5fFotTh3KLg%40mail.gmail.com.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>
>>>>>> >> --
>>>>>> >> You received this message because you are subscribed to the Google
>>>>>> >> Groups
>>>>>> >> "Kubernetes developer/contributor discussion" group.
>>>>>> >> To unsubscribe from this group and stop receiving emails from it,
>>>>>> >> send an

>>>>>> >> email to kubernetes-de...@googlegroups.com.

>>>>>> >> To post to this group, send email to
>>>>>> >> kuberne...@googlegroups.com.
>>>>>> >> To view this discussion on the web visit
>>>>>> >>
>>>>>> >> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzf6MK5EzF1PHGH1h8SYjgoTcD_31mwnVByMy8T6Yq6w%2BQ%40mail.gmail.com.
>>>>>> >>
>>>>>> >> For more options, visit https://groups.google.com/d/optout.
>>>>>> >
>>>>>> >
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>>> an email to kubernetes-de...@googlegroups.com.

>>>>> To post to this group, send email to kuberne...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzeJvjdCmc_x-1owCPKLV0KVT9X-K9cOuit7-C-FAP0bOA%40mail.gmail.com.
>>>>>
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Michael Taufen
>>>> MTV SWE
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>> an email to kubernetes-de...@googlegroups.com.

[Rajat Ramesh Koujalagi]

I might be wrong but..

The pod gets deleted if during the upgrade the node is not able to come up fast enough. This would cause the master controller to see that there is an orphaned pod (no node belongs to it) and the master deletes it.

A workaround that I did is to stub out the podgc garbage collection code and nodeController pod deletion code. This will ensure that your pod stays up post the upgrade and you can verify behavior (secrets in your case).

In terms of user experience I think the pod getting deleted is the expected behavior. Of course if they have a replication controller then that's a different story.

>>>>>> > developer/contributor discussion <kubernetes-dev@googlegroups.com>

>>>>>> >>>>>>> <kubernetes-dev@googlegroups.com> wrote:
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> Hi folks. Just a reminder that we urgently need someone to
>>>>>> >>>>>>>> work on
>>>>>> >>>>>>>> https://github.com/kubernetes/kubernetes/issues/37474
>>>>>> >>>>>>>> as it is blocking manual upgrade testing.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> On Fri, Nov 25, 2016 at 12:00 AM, Robert Bailey
>>>>>> >>>>>>>> <robert...@google.com> wrote:
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Master upgrades now work. Node upgrades are broken. See
>>>>>> >>>>>>>>> https://github.com/kubernetes/kubernetes/issues/37474.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> On Thu, Nov 24, 2016 at 10:37 PM, Saad Ali
>>>>>> >>>>>>>>> <saa...@google.com>
>>>>>> >>>>>>>>> wrote:
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> The v1.5.0-beta.2 has been cut. It should have Robert's
>>>>>> >>>>>>>>>> fix.
>>>>>> >>>>>>>>>> Please give it a shot.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> On Tue, Nov 22, 2016 at 3:33 PM, David Oppenheimer
>>>>>> >>>>>>>>>> <davi...@google.com> wrote:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Thanks!
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> On Tue, Nov 22, 2016 at 3:21 PM, Robert Bailey
>>>>>> >>>>>>>>>>> <robert...@google.com> wrote:
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> I'm going to start looking into this today.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> On Mon, Nov 21, 2016 at 5:53 PM, 'Rajat Ramesh Koujalagi'
>>>>>> >>>>>>>>>>>> via
>>>>>> >>>>>>>>>>>> Kubernetes developer/contributor discussion

>>>>>> >>>>>>>>>>>>>>>>> > kubernetes-dev+unsubscribe@googlegroups.com.

>>>>>> >>>>>>>>>>>>>>>>> > To post to this group, send email to

>>>>>> >>>>>>>>>>>>>>>>> > kubernetes-dev@googlegroups.com.

>>>>>> >>>>>>>>>>>>>>>>> > To view this discussion on the web visit
>>>>>> >>>>>>>>>>>>>>>>> >
>>>>>> >>>>>>>>>>>>>>>>> > https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdPWU9S2eY4hytXrqm9EA-xXkr2guG64tkzu1%3DbH7YZ3g%40mail.gmail.com.
>>>>>> >>>>>>>>>>>>>>>>> > For more options, visit
>>>>>> >>>>>>>>>>>>>>>>> > https://groups.google.com/d/optout.
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>> --
>>>>>> >>>>>>>>>>>>>>>>> Tomasz 'Zen' Napierala
>>>>>> >>>>>>>>>>>>>>>>> Kubernetes Engineering - Poland
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> --
>>>>>> >>>>>>>>>>>>> You received this message because you are subscribed to
>>>>>> >>>>>>>>>>>>> the
>>>>>> >>>>>>>>>>>>> Google Groups "Kubernetes developer/contributor
>>>>>> >>>>>>>>>>>>> discussion" group.
>>>>>> >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails
>>>>>> >>>>>>>>>>>>> from
>>>>>> >>>>>>>>>>>>> it, send an email to

>>>>>> >>>>>>>>>>>>> kubernetes-dev+unsubscribe@googlegroups.com.

>>>>>> >>>>>>>>>>>>> To post to this group, send email to

>>>>>> >>>>>>>>>>>>> kubernetes-dev@googlegroups.com.

>>>>>> >>>>>>>>>>>>> To view this discussion on the web visit
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CA%2Bd%3DP%2B%2BptC5H-LSB3mJJh-X2dKRYJWtdK%3DON%2BcQP8L9ijDxQuw%40mail.gmail.com.
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> For more options, visit
>>>>>> >>>>>>>>>>>>> https://groups.google.com/d/optout.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> --
>>>>>> >>>>>>>> You received this message because you are subscribed to the
>>>>>> >>>>>>>> Google
>>>>>> >>>>>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>>>> >>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>> >>>>>>>> it,

>>>>>> >>>>>>>> send an email to kubernetes-dev+unsubscribe@googlegroups.com.

>>>>>> >>>>>>>> To post to this group, send email to

>>>>>> >>>>>>>> kubernetes-dev@googlegroups.com.

>>>>>> >>>>>>>> To view this discussion on the web visit
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdq1%2BnKoROz68nvDpfM2K1L4BSiz_XFNAL5fFotTh3KLg%40mail.gmail.com.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>
>>>>>> >> --
>>>>>> >> You received this message because you are subscribed to the Google
>>>>>> >> Groups
>>>>>> >> "Kubernetes developer/contributor discussion" group.
>>>>>> >> To unsubscribe from this group and stop receiving emails from it,
>>>>>> >> send an

>>>>>> >> email to kubernetes-dev+unsubscribe@googlegroups.com.

>>>>>> >> To post to this group, send email to

>>>>>> >> kubernetes-dev@googlegroups.com.

>>>>>> >> To view this discussion on the web visit
>>>>>> >>
>>>>>> >> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzf6MK5EzF1PHGH1h8SYjgoTcD_31mwnVByMy8T6Yq6w%2BQ%40mail.gmail.com.
>>>>>> >>
>>>>>> >> For more options, visit https://groups.google.com/d/optout.
>>>>>> >
>>>>>> >
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>>> an email to kubernetes-dev+unsubscribe@googlegroups.com.
>>>>> To post to this group, send email to kubernetes-dev@googlegroups.com.

>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzeJvjdCmc_x-1owCPKLV0KVT9X-K9cOuit7-C-FAP0bOA%40mail.gmail.com.
>>>>>
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Michael Taufen
>>>> MTV SWE
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>> an email to kubernetes-dev+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to kubernetes-dev@googlegroups.com.

[David Oppenheimer]

Yes, what Rajat said is right.

1) Node upgrade on GCE deletes the VM (which of course kills everything -- the containers and all the local data) and re-creates the VM

2) If a node is down for < 5 minutes and it comes back up with the same name, then Kubernetes will re-create the pods that were running there previously (the data will still be gone)

3) If a node is down for > 5 minutes or it comes back up with a different name, then the pod is not re-created. If it was managed by a controller, then the controller should create a replacement elsewhere.

Ideally we would force the node upgrades in these manual tests to take > 5 minutes so the result would always be the same, but we don't do that.

>>>>>> >>>>>>>>>>>>>>>>> > kuberne...@googlegroups.com.

>>>>>> >>>>>>>>>>>>>>>>> > To view this discussion on the web visit
>>>>>> >>>>>>>>>>>>>>>>> >
>>>>>> >>>>>>>>>>>>>>>>> > https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdPWU9S2eY4hytXrqm9EA-xXkr2guG64tkzu1%3DbH7YZ3g%40mail.gmail.com.
>>>>>> >>>>>>>>>>>>>>>>> > For more options, visit
>>>>>> >>>>>>>>>>>>>>>>> > https://groups.google.com/d/optout.
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>> --
>>>>>> >>>>>>>>>>>>>>>>> Tomasz 'Zen' Napierala
>>>>>> >>>>>>>>>>>>>>>>> Kubernetes Engineering - Poland
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> --
>>>>>> >>>>>>>>>>>>> You received this message because you are subscribed to
>>>>>> >>>>>>>>>>>>> the
>>>>>> >>>>>>>>>>>>> Google Groups "Kubernetes developer/contributor
>>>>>> >>>>>>>>>>>>> discussion" group.
>>>>>> >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails
>>>>>> >>>>>>>>>>>>> from
>>>>>> >>>>>>>>>>>>> it, send an email to
>>>>>> >>>>>>>>>>>>> kubernetes-dev+unsubscribe@googlegroups.com.
>>>>>> >>>>>>>>>>>>> To post to this group, send email to

>>>>>> >>>>>>>>>>>>> kuberne...@googlegroups.com.

>>>>>> >>>>>>>>>>>>> To view this discussion on the web visit
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CA%2Bd%3DP%2B%2BptC5H-LSB3mJJh-X2dKRYJWtdK%3DON%2BcQP8L9ijDxQuw%40mail.gmail.com.
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> For more options, visit
>>>>>> >>>>>>>>>>>>> https://groups.google.com/d/optout.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> --
>>>>>> >>>>>>>> You received this message because you are subscribed to the
>>>>>> >>>>>>>> Google
>>>>>> >>>>>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>>>> >>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>> >>>>>>>> it,
>>>>>> >>>>>>>> send an email to kubernetes-dev+unsubscribe@googlegroups.com.
>>>>>> >>>>>>>> To post to this group, send email to

>>>>>> >>>>>>>> kuberne...@googlegroups.com.

>>>>>> >>>>>>>> To view this discussion on the web visit
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzdq1%2BnKoROz68nvDpfM2K1L4BSiz_XFNAL5fFotTh3KLg%40mail.gmail.com.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>
>>>>>> >> --
>>>>>> >> You received this message because you are subscribed to the Google
>>>>>> >> Groups
>>>>>> >> "Kubernetes developer/contributor discussion" group.
>>>>>> >> To unsubscribe from this group and stop receiving emails from it,
>>>>>> >> send an
>>>>>> >> email to kubernetes-dev+unsubscribe@googlegroups.com.
>>>>>> >> To post to this group, send email to

>>>>>> >> kuberne...@googlegroups.com.

>>>>>> >> To view this discussion on the web visit
>>>>>> >>
>>>>>> >> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzf6MK5EzF1PHGH1h8SYjgoTcD_31mwnVByMy8T6Yq6w%2BQ%40mail.gmail.com.
>>>>>> >>
>>>>>> >> For more options, visit https://groups.google.com/d/optout.
>>>>>> >
>>>>>> >
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to kubernetes-dev+unsubscribe@googlegroups.com.

>>>>> To post to this group, send email to kuberne...@googlegroups.com.

>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzeJvjdCmc_x-1owCPKLV0KVT9X-K9cOuit7-C-FAP0bOA%40mail.gmail.com.
>>>>>
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Michael Taufen
>>>> MTV SWE
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Kubernetes developer/contributor discussion" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to kubernetes-dev+unsubscribe@googlegroups.com.

>>>> To post to this group, send email to kuberne...@googlegroups.com.