Cluster ID API

[Morgan Bauer]

Hello,


In the past I was seeking guidance on approaches to establish a cluster
identifier that could be used to tell to different clusters apart.


I have gone forward with an implementation. I would like to get some
feedback on what you can see below.


The links below are for a short 5 minute discussion and demo.


Currently is installed as part of service catalog API Group, it can be moved.

Once the apiserver and controller are installed and the ID resource is automatically provisioned and set up with a random value.

API machinery can't prevent it from being deleted. I do not currently see a way to implement this without changes in API machinery.

Validation can prevent changes and enforce the singleton by only
allowing one specific name. This is a storage layer function, and is probably the inappropriate place to do this for production. Some API machinery changes may be needed to do this at a higher REST layer.

Discussion on how we can manually set it in theory by controller flag,
or by setting the object before the controller starts.


First link should be a downloadable, second is youtube

https://ibm.box.com/s/znkz2n3hnhcep2rat04q3n0tn9wev1bk

https://www.youtube.com/watch?v=eqI0JqCmxug



I can attend a meeting if further discussion is required, or we can raise it in a service-catalog meeting if that is more helpful.

Thank you

[Stefan Schimanski]

Just to be clear about your setting:

- you are in an aggregated apiserver, this is not CRD based

- you are asking for a way to forbid deletion of objects, but not via RBAC, but through the implemented registry for the resource.

> API machinery can't prevent it from being deleted. I do not currently see a way to implement this without changes in API machinery.

I guess you have tried implement a special Delete method returning a "Cannot be deleted" 40x error?

  https://github.com/kubernetes/kubernetes/blob/2452afffe09e1ced9487e5a701beb1443a92b741/staging/src/k8s.io/apiserver/pkg/registry/rest/rest.go#L147

Best regards,

   Stefan

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/eac255e2-db7e-a35f-c754-4abdf5715b21%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Smith]

Better yet, just don't implement the DeleteStrategy.

Best regards,

   Stefan

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.

To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/eac255e2-db7e-a35f-c754-4abdf5715b21%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFtQ%3DdoN5qXrJ1%3DnA05juxDrecg-Mz%3DmdLxA3oN70syuxmaU_A%40mail.gmail.com.

[Stefan Schimanski]

On Thu, Feb 1, 2018 at 5:35 PM, Daniel Smith <dbs...@google.com> wrote:

Better yet, just don't implement the DeleteStrategy.

Depending on whether he uses the generic registry code. But for this special purpose resource maybe that's reasonable.

 

[Daniel Smith]

If you don't want DELETE to ever be an option, then the verb shouldn't installed for the path. Otherwise the OpenAPI spec will be lying to users. Probably POST, PUT, PATCH & WATCH shouldn't be installed either, if it is a system generated immutable read-only informational resource.

I will also admit I'm not super excited about having a singleton resource like this, but that's the nice thing about aggregated APIs / CRDs-- nobody has to be on board for you to experiment :)

Something we're thinking about lately is how to make the control plane "latch" at correct versions, CRDs, aggregated apis, and feature flags. This is likely going to involve some sort of coordination between apiservers in the storage layer. This coordination point would be a natural place to record a unique cluster ID. (It may not be exposed to users directly from there, of course!)

[Christian Bell]

On Thu, Feb 1, 2018 at 9:23 AM, 'Daniel Smith' via kubernetes-sig-multicluster <kubernetes-si...@googlegroups.com> wrote:

I will also admit I'm not super excited about having a singleton resource like this, but that's the nice thing about aggregated APIs / CRDs-- nobody has to be on board for you to experiment :)

Experimentation is great. But let's face it -- this is a lot of work to retrofit a per-cluster UUID that should have been made available from day 0 ;-).

[DL duglin]

big +1 - I was really surprised we didn't already have a UUID we could leverage.

-Doug

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CABnu8_4ojQ1cPWHLkhDKUvesb0kjN9Dp3SNPtxTcJwAJc-%2BysA%40mail.gmail.com.

[Morgan Bauer]

If this kind of resource is better implemented at a different level that abandons some of the existing infrastructure, I've not got that understanding yet.

As far as not installing various verbs, I'm not sure I've ever explicitly chosen or not chosen to install them. I honestly have no idea what mechanism is involved that they end up exposed or not when building the apiserver that we do have.

On 2/1/18 9:23 AM, 'Daniel Smith' via kubernetes-sig-architecture wrote:

If you don't want DELETE to ever be an option, then the verb shouldn't installed for the path. Otherwise the OpenAPI spec will be lying to users. Probably POST, PUT, PATCH & WATCH shouldn't be installed either, if it is a system generated immutable read-only informational resource.

I will also admit I'm not super excited about having a singleton resource like this, but that's the nice thing about aggregated APIs / CRDs-- nobody has to be on board for you to experiment :)

Exactly so. Let the "api economy" decide.

You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3bb9zh3uD-GB_wNTUms5eyHGJ6Og6f6v%3D%3D4NkXsBGdUd-w%40mail.gmail.com.

[bryan...@gmail.com]

I have considered using the `uid` field of the kube-system namespace as a cluster identifier. From what I understand, all clusters will have this.

Best regards,

   Stefan

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/eac255e2-db7e-a35f-c754-4abdf5715b21%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFtQ%3DdoN5qXrJ1%3DnA05juxDrecg-Mz%3DmdLxA3oN70syuxmaU_A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsub...@googlegroups.com.

[DL duglin]

I might be remembering incorrectly, but I think that namespace can be deleted. It'll then be recreated with a new uid. No?

-Doug

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

To post to this group, send email to kubernetes-si...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/26df07e1-320c-4bb0-8185-041711b8a816%40googlegroups.com.

[Joe Beda]

My feeling is that we should work to pick something that is always that isn't likely to change.  There are any number of things that people can do to screw up a cluster.  While it may be possible to delete/create kube-system, practically it'll never happen.  On Linux, you can always regenerate your ssh host key. You can always `dd of=/dev/sda`.  Just don't do that.

In my mind using the uid kube-system is good enough and requires ~no code changes to anything.

(Also -- head nod to JustinSB for suggesting that in the first place)

Joe

Best regards,

   Stefan

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.

To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/eac255e2-db7e-a35f-c754-4abdf5715b21%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.

To post to this group, send email to kubernetes-sig...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFtQ%3DdoN5qXrJ1%3DnA05juxDrecg-Mz%3DmdLxA3oN70syuxmaU_A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3bb9zh3uD-GB_wNTUms5eyHGJ6Og6f6v%3D%3D4NkXsBGdUd-w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/26df07e1-320c-4bb0-8185-041711b8a816%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/C908ED97-A636-4FC9-8530-87832DF5AFAC%40gmail.com.

[Jordan Liggitt]

kube-system is an immortal namespace. It can't be deleted via the API. You'd have to mess with etcd directly to remove it. 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAEAFPY%2BAzTRiNq3%3DMm8Ei524fLEg%2BgGZfFNrsYc9BftLQjUiHg%40mail.gmail.com.

[Paul Morie]

This is a very interesting idea re: kube-system ns uid

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CALbx6sZzcZKu6aESmbxKL3Ld9gNgLYyFHcAWyJ8Sy6WovqQhSA%40mail.gmail.com.

[DL duglin]

Is there an option to set the uid of kube-system namespace during init time? I'm asking because while the ClusterID should be immutable, for the most part, I can image cases where you do need to set it because remote systems already know your Kube cluster by a certain ID.  For example, you're trying to reconstruct a cluster and don't have the freedom to modify the remote system's ClusterID it knows you by. In cases like that we may need to force a certain cluserID on things. If we can do that, even going directly to etcd, that's ok - as long as its possible somehow.

-Doug

[Jordan Liggitt]

> On Feb 8, 2018, at 8:23 PM, DL duglin <dug...@gmail.com> wrote:
>
> Is there an option to set the uid of kube-system namespace during init time?

No, uids are allocated at object creation time deep in storage.

> If we can do that, even going directly to etcd, that's ok - as long as its possible somehow.

That's what you would have to do

[Mo Khan]

IIRC admission allows you to override the UID on create by injecting it into the request context.  Though I am not sure you could do that for kube-system.  Direct to ETCD is probably simpler.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-architecture@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CALbx6sZgg4%3DWQ4qD_qNrKscSL_xisaQajfnaH25h8%3DZ10DTdjg%40mail.gmail.com.

[Daniel Smith]

We should probably have a test that verifies that you can set the UID via mutating webhook, or that you get an error if you try, preferably the latter--setting non-unique UIDs is a foot-gun: a good way to completely break your cluster subtly enough that you won't notice for quite some time. 

Either way it won't help on kube-system, since it must already have been created before you can even add in the webhook configuration. :)

On Sat, Feb 10, 2018 at 9:48 PM, Mo Khan <mo...@redhat.com> wrote:

IIRC admission allows you to override the UID on create by injecting it into the request context.  Though I am not sure you could do that for kube-system.  Direct to ETCD is probably simpler.

On Feb 8, 2018 8:36 PM, "Jordan Liggitt" <jlig...@redhat.com> wrote:

> On Feb 8, 2018, at 8:23 PM, DL duglin <dug...@gmail.com> wrote:
>
> Is there an option to set the uid of kube-system namespace during init time?

No, uids are allocated at object creation time deep in storage.

> If we can do that, even going directly to etcd, that's ok - as long as its possible somehow.


That's what you would have to do

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsubs...@googlegroups.com.

To post to this group, send email to kubernetes-sig-architecture@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CALbx6sZgg4%3DWQ4qD_qNrKscSL_xisaQajfnaH25h8%3DZ10DTdjg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-architecture@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAKT2rag3nKEkVfrh70EZW04d7eX%3DYQ8xrdXjv%3Do8sCD5VsX2OA%40mail.gmail.com.

[Mo Khan]

Seems like we should just remove the UID override completely.  The only safe way to use it is to create a very random + long UID (since you cannot tell what UIDs you have used unless you are tracking it via an external system), so just let the system do it correctly for you.

On Mon, Feb 12, 2018 at 12:19 PM, Daniel Smith <dbs...@google.com> wrote:

We should probably have a test that verifies that you can set the UID via mutating webhook, or that you get an error if you try, preferably the latter--setting non-unique UIDs is a foot-gun: a good way to completely break your cluster subtly enough that you won't notice for quite some time. 

Either way it won't help on kube-system, since it must already have been created before you can even add in the webhook configuration. :)

On Sat, Feb 10, 2018 at 9:48 PM, Mo Khan <mo...@redhat.com> wrote:

IIRC admission allows you to override the UID on create by injecting it into the request context.  Though I am not sure you could do that for kube-system.  Direct to ETCD is probably simpler.

On Feb 8, 2018 8:36 PM, "Jordan Liggitt" <jlig...@redhat.com> wrote:

> On Feb 8, 2018, at 8:23 PM, DL duglin <dug...@gmail.com> wrote:
>
> Is there an option to set the uid of kube-system namespace during init time?

No, uids are allocated at object creation time deep in storage.

> If we can do that, even going directly to etcd, that's ok - as long as its possible somehow.


That's what you would have to do

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig-architecture@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CALbx6sZgg4%3DWQ4qD_qNrKscSL_xisaQajfnaH25h8%3DZ10DTdjg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig-architecture@googlegroups.com.

[Daniel Smith]

I agree, would you like to file an issue to track this?

[Joe Beda]

Taking a step back -- anyone see any reason not to use UID as a cluster ID?

On Mon, Feb 12, 2018 at 11:18 AM Daniel Smith <dbs...@google.com> wrote:

I agree, would you like to file an issue to track this?

On Mon, Feb 12, 2018 at 11:02 AM, Mo Khan <mo...@redhat.com> wrote:

Seems like we should just remove the UID override completely.  The only safe way to use it is to create a very random + long UID (since you cannot tell what UIDs you have used unless you are tracking it via an external system), so just let the system do it correctly for you.

On Mon, Feb 12, 2018 at 12:19 PM, Daniel Smith <dbs...@google.com> wrote:

We should probably have a test that verifies that you can set the UID via mutating webhook, or that you get an error if you try, preferably the latter--setting non-unique UIDs is a foot-gun: a good way to completely break your cluster subtly enough that you won't notice for quite some time. 

Either way it won't help on kube-system, since it must already have been created before you can even add in the webhook configuration. :)

On Sat, Feb 10, 2018 at 9:48 PM, Mo Khan <mo...@redhat.com> wrote:

IIRC admission allows you to override the UID on create by injecting it into the request context.  Though I am not sure you could do that for kube-system.  Direct to ETCD is probably simpler.

On Feb 8, 2018 8:36 PM, "Jordan Liggitt" <jlig...@redhat.com> wrote:

> On Feb 8, 2018, at 8:23 PM, DL duglin <dug...@gmail.com> wrote:
>
> Is there an option to set the uid of kube-system namespace during init time?

No, uids are allocated at object creation time deep in storage.

> If we can do that, even going directly to etcd, that's ok - as long as its possible somehow.


That's what you would have to do

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CALbx6sZgg4%3DWQ4qD_qNrKscSL_xisaQajfnaH25h8%3DZ10DTdjg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.

[Quinton Hoole]

I had assumed that to be implied by:

"Once the apiserver and controller are installed and the ID resource is automatically provisioned and set up with a random value."

Q

On Mon, Feb 12, 2018 at 11:47 AM, Joe Beda <j...@heptio.com> wrote:

Taking a step back -- anyone see any reason not to use UID as a cluster ID?

On Mon, Feb 12, 2018 at 11:18 AM Daniel Smith <dbs...@google.com> wrote:

I agree, would you like to file an issue to track this?

On Mon, Feb 12, 2018 at 11:02 AM, Mo Khan <mo...@redhat.com> wrote:

Seems like we should just remove the UID override completely.  The only safe way to use it is to create a very random + long UID (since you cannot tell what UIDs you have used unless you are tracking it via an external system), so just let the system do it correctly for you.

On Mon, Feb 12, 2018 at 12:19 PM, Daniel Smith <dbs...@google.com> wrote:

We should probably have a test that verifies that you can set the UID via mutating webhook, or that you get an error if you try, preferably the latter--setting non-unique UIDs is a foot-gun: a good way to completely break your cluster subtly enough that you won't notice for quite some time. 

Either way it won't help on kube-system, since it must already have been created before you can even add in the webhook configuration. :)

On Sat, Feb 10, 2018 at 9:48 PM, Mo Khan <mo...@redhat.com> wrote:

IIRC admission allows you to override the UID on create by injecting it into the request context.  Though I am not sure you could do that for kube-system.  Direct to ETCD is probably simpler.

On Feb 8, 2018 8:36 PM, "Jordan Liggitt" <jlig...@redhat.com> wrote:

> On Feb 8, 2018, at 8:23 PM, DL duglin <dug...@gmail.com> wrote:
>
> Is there an option to set the uid of kube-system namespace during init time?

No, uids are allocated at object creation time deep in storage.

> If we can do that, even going directly to etcd, that's ok - as long as its possible somehow.


That's what you would have to do

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-architecture@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CALbx6sZgg4%3DWQ4qD_qNrKscSL_xisaQajfnaH25h8%3DZ10DTdjg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-architecture@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAKT2rag3nKEkVfrh70EZW04d7eX%3DYQ8xrdXjv%3Do8sCD5VsX2OA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-architecture@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAEAFPYLhwJxyEtNYyDT3USx-t5v59DtjyZASD1iy8X6wHvg-Nw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Quinton Hoole
qui...@hoole.biz