New types in old APIs

[Tim Hockin]

Suppose we have a v1 apigroup "foo". We want to add an entirely new
concept and type, and as such we need to go thru alpha a beta. There
is no longer a v1alpha1 for "foo".

What do we do?

1) Resurrect v1alpha1 "foo" ?
2) Spawn v2alpha1 "foo" ?
3) Make an entirely new apigroup?
4) Something else?

These all seem unfortunately clumsy. Guidance requested.

Tim

[Clayton Coleman]

Resurrect seems the most consistent to a user (ignoring for a moment we have some bugs there in how we handle objects moving singly across resource versions).  New version seems more consistent with the current "disable/enable alpha/beta via a version+group" concept for admins.  Creating lots of similarly named groups seems to violate the spirit of groups (it's a group... for one resource due to a release?).

It seems harder to explain to a new user or contributor why "you need to go create v1alpha15 because v1alpha14 is closed in 1.34" vs "new api?  always goes into v1alpha1.  ready for beta?  goes to v1beta1".  I feel like I'd like to get this right for end users over admins (go here to see things that are new as a user that you shouldn't use).

I could maybe argue that good naming for versions might address the lots of api versions problem `v1alpha1-<feature_name>`, but i also think that would pollute user visible lists without adding clarity.  It might add clarity to admins though.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAO_Rewa%3Do2qMt-Szwe99EOmee4hE3V2jFd7ZsygD-wh%3DwPNpKw%40mail.gmail.com.

[Daniel Smith]

On Mon, Jan 24, 2022 at 1:03 PM 'Tim Hockin' via K8s API Machinery SIG <kubernetes-sig...@googlegroups.com> wrote:

Suppose we have a v1 apigroup "foo".  We want to add an entirely new
concept and type, and  as such we need to go thru alpha a beta.  There
is  no longer a v1alpha1 for "foo".

What do we do?

1) Resurrect v1alpha1 "foo" ?

1.1) Make a new v1alpha2 "foo", to avoid confusing any client that knew about v1alpha1.

 

2) Spawn v2alpha1 "foo" ?
3) Make an entirely new apigroup?
4) Something else?

These all seem unfortunately clumsy.  Guidance requested.

Tim

[David Eads]

The old client could not have known about a type that didn't exist yet, so re-using v1alpha1.foo for NewKind should be safe.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAB_J3baDU9pK2%2B3LoZWuoOhhoy6dC-cwh5waqkk5zUTaGPz1Gg%40mail.gmail.com.

[Daniel Smith]

Right, the error mode to avoid is such a client finding the v1alpha1 group-- but type(s) it knew about aren't there anymore.

(It would probably be fine. But just in case...)

[Tim Hockin]

If we use a v1alphaN, does that imply all the existing types that are
available in v1 also get cloned into v1alphaN ?

I realize I am getting back into the same old discussion about sparse groups.

Concretely, I used to have;

foo/v1alpha1: resources "aardvarks" and "birds"
foo/v1beta1: resources "aardvarks" and "birds"
foo/v1: resources "aardvarks" and "birds"

alpha1 and beta1 aged out (and in the future, maybe they would have
been deleted).

Now I want to add a "cats" resource. I'll spawn a foo/v1alpha2 with
"cats" - does that ALSO have "aardvarks" and "birds" ?

On Mon, Jan 24, 2022 at 2:03 PM David Eads <de...@redhat.com> wrote:
>

[Clayton Coleman]

Should clients be making blanket assumptions about groups like that?

“I’m a client and i expect these 30 resources to be here so I can use Bar?”

Group is an organizing concept, why can’t a resource be selectively enabled within it?

On Jan 24, 2022, at 5:06 PM, 'Daniel Smith' via K8s API Machinery SIG <kubernetes-sig...@googlegroups.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAB_J3bZ8WKH7qQHbvnYj3gCx32tG-WCwJUoNxbSjef5mwNNUjA%40mail.gmail.com.

[Daniel Smith]

On Mon, Jan 24, 2022 at 2:11 PM Tim Hockin <tho...@google.com> wrote:

If we use a v1alphaN, does that imply all the existing types that are
available in v1 also get cloned into v1alphaN ?

I realize I am getting back into the same old discussion about sparse groups.

Concretely, I used to have;

foo/v1alpha1: resources "aardvarks" and "birds"
foo/v1beta1: resources "aardvarks" and "birds"
foo/v1: resources "aardvarks" and "birds"

alpha1 and beta1 aged out (and in the future, maybe they would have
been deleted).

Wait, in the original hypothetical you said the alpha group didn't contain the types any more? If the group currently exists but doesn't have the types (how did this happen?) then IMO your option 1 is fine, just reuse it.

 

Now I want to add a "cats" resource.  I'll spawn a foo/v1alpha2 with
"cats"  - does that ALSO have "aardvarks" and "birds" ?

It doesn't have to. 

[Tim Hockin]

On Mon, Jan 24, 2022 at 2:13 PM Clayton Coleman <ccol...@redhat.com> wrote:
>
> Should clients be making blanket assumptions about groups like that?
>
> “I’m a client and i expect these 30 resources to be here so I can use Bar?”
>
> Group is an organizing concept, why can’t a resource be selectively enabled within it?

This is what I meant "same old" topic. My personal feeling is that
groups should not be sparse, but I think I lost that fight.

:)

> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAH16ShJge0ggPrk5TxEuceQ9LUV_7N5%2BX%3DJigeeL7%3DhqySYb3w%40mail.gmail.com.

[Daniel Smith]

Clients should not be making assumptions like that, however they *could*, and it's easy for us to not do the thing that would lead such clients down the garden path.

On Mon, Jan 24, 2022 at 2:13 PM Clayton Coleman <ccol...@redhat.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAH16ShJge0ggPrk5TxEuceQ9LUV_7N5%2BX%3DJigeeL7%3DhqySYb3w%40mail.gmail.com.

[Clayton Coleman]

It feels like we are debating technology when we should be debating:

“what is the purpose of an api version and an api group, and who are
we communicating to?”

Version indicates maturity and specific minimum levels of capability.
Group creates a human readable association between related resources.

Then there are all the extra biases and limitations of technology we
bring to the discussion that might modify those arguments (sparse vs
full).

I still believe we should be prioritizing the user of the API over the
admin of the cluster or the engineers implementing it, so a relevant
argument is “a human <prefers|hates> v1alpha2 because <reason>” that
the sparsists vs the duplicatists camps should make a case for.

> On Jan 24, 2022, at 5:11 PM, 'Tim Hockin' via K8s API Machinery SIG <kubernetes-sig...@googlegroups.com> wrote:
>
> If we use a v1alphaN, does that imply all the existing types that are

> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAO_RewbaUEB6_sKpJUoDBYV1TarBxa0cZC2OcXuHkjinpMEpQw%40mail.gmail.com.
>

[Tim Hockin]

On Mon, Jan 24, 2022 at 2:14 PM Daniel Smith <dbs...@google.com> wrote:
>
>
>
> On Mon, Jan 24, 2022 at 2:11 PM Tim Hockin <tho...@google.com> wrote:
>>
>> If we use a v1alphaN, does that imply all the existing types that are
>> available in v1 also get cloned into v1alphaN ?
>>
>> I realize I am getting back into the same old discussion about sparse groups.
>>
>> Concretely, I used to have;
>>
>> foo/v1alpha1: resources "aardvarks" and "birds"
>> foo/v1beta1: resources "aardvarks" and "birds"
>> foo/v1: resources "aardvarks" and "birds"
>>
>> alpha1 and beta1 aged out (and in the future, maybe they would have
>> been deleted).
>
>
> Wait, in the original hypothetical you said the alpha group didn't contain the types any more? If the group currently exists but doesn't have the types (how did this happen?) then IMO your option 1 is fine, just reuse it.

Do we ever do that? Leave the group but remove the types?

>> Now I want to add a "cats" resource. I'll spawn a foo/v1alpha2 with
>> "cats" - does that ALSO have "aardvarks" and "birds" ?
>
>
> It doesn't have to.

So it's OK to have a sparse group and to make a newalpha group after a GA.

[Tim Hockin]

On Mon, Jan 24, 2022 at 2:18 PM Clayton Coleman <ccol...@redhat.com> wrote:
>
> It feels like we are debating technology when we should be debating:
>
> “what is the purpose of an api version and an api group, and who are
> we communicating to?”
>
> Version indicates maturity and specific minimum levels of capability.
> Group creates a human readable association between related resources.

Agree. In my mind, a human would expect the members of a group to be
present in all subsequent versions of that group. As such, the answer
I would hope for my query was "make a v2alpha1, and import all v1
types into it" (and that our machinery would make that easy). But I
know it doesn't make that easy and that's NOT how we have operated in
the past. Right?

GVK = (GV)+K or G+(VK)

> I still believe we should be prioritizing the user of the API over the
> admin of the cluster or the engineers implementing it, so a relevant
> argument is “a human <prefers|hates> v1alpha2 because <reason>” that
> the sparsists vs the duplicatists camps should make a case for.

That's "sparcicists", TYVM. :)

[James Munnelly]

From a correctness point of view, I would argue that a given object Foo (that is at v1) doesn’t have a representation at v1alpha2, and copying the resource into the other version really does not achieve anything bar confusion.

Some examples I can think of:

1) given this is an alpha API, would it be okay for me to make breaking changes to the schema for a resource that has otherwise GA’d?

2) would it be possible for this resource to graduate? Would it graduate to v2 eventually? This means we then have resources in v1alpha2 that are all destined for different future api versions

3) we’re also creating a lot more endpoints that’ll eventually stop working, for the sake of ‘tidiness’. This could lead to clients utilising v1alphaN for resources that have corresponding v1 versions already.

So I think that makes me a ‘sparcicist’ 😄

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAO_Rewbpk%3D0TenXbH6v6YEyNHAEscAeM1aOj1UsGL3XucG7xyw%40mail.gmail.com.

[Jordan Liggitt]

I'd expect the new type to start as v1alpha1 (and be the only resource in that alpha version if we've already stopped serving the other graduated types), mature, and eventually join its peers in v1

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAHD0B%3D9g7cpL2BhYBz43cJ5Eh5xLrs6Pam%3DYfjxN13pFi5hX_w%40mail.gmail.com.

[Tim Hockin]

OK, So as long as we're OK with resurrecting old API versions, I'll
suggest we proceed on that paath, sparsely.

Thanks for the opinions!

[Clayton Coleman]

I can take a stab at proposing this in a PR in docs. I do think it’s
possible we have some small sparse bugs - if you hit any let me know
since i vaguely remember which bodies were buried where.

> On Jan 25, 2022, at 3:16 PM, Tim Hockin <tho...@google.com> wrote:
>
> OK, So as long as we're OK with resurrecting old API versions, I'll

[Tim Hockin]

In the near term we have a larger problem - can't mix apigroups for
buitins and CRDs :(

[Clayton Coleman]

Don’t tempt me to rewrite the entire handler chain…

> On Jan 25, 2022, at 7:34 PM, Tim Hockin <tho...@google.com> wrote:
>
> In the near term we have a larger problem - can't mix apigroups for

[Antonio Ojea]

I hit a problem implementing this, let me add some context

1. add a new type ServiceCIDRConfig, resurrecting v1alpha1,  on the group networking.k8s.io that only has v1 right now.

2. wire it (I think that this is ok, apiserver boots if I comment the new types) and enable the version in the apiserver (TL;DR https://github.com/kubernetes/kubernetes/pull/110470)

3.the apiserver fails trying to create the storage with the following error

E0623 09:53:42.745504 2777558 run.go:74] "command failed" err="problem initializing API group \"networking.k8s.io\" : storage codec doesn't seem to match given type: Internal type not encodable: no kind \"ServiceCIDRConfig\" is registered for version \"networking.k8s.io/v1\" in scheme \"pkg/api/legacyscheme/scheme.go:30\""

The whole stacktrace is

goroutine 1 [running]:
runtime/debug.Stack()
    /usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
    /usr/local/go/src/runtime/debug/stack.go:16 +0x19
k8s.io/apiserver/pkg/registry/generic/registry.StorageWithCacher.func1(0xc004959b00, {0xc00074a2d0, 0x13}, 0xc001c52ca8, 0x4dd9d70, 0x4dd9d78, 0x4dd5368, 0x0, 0x0)
    vendor/k8s.io/apiserver/pkg/registry/generic/registry/storage_factory.go:73 +0x4d9
k8s.io/apiserver/pkg/registry/generic/registry.(*Store).CompleteWithOptions(0xc004bf2000, 0xc0005ce698)
    vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go:1442 +0x7e3
k8s.io/kubernetes/pkg/registry/networking/servicecidr/storage.NewREST({0x5422f80, 0xc000567b00})
    pkg/registry/networking/servicecidr/storage/storage.go:49 +0x359
k8s.io/kubernetes/pkg/registry/networking/rest.RESTStorageProvider.v1alpha1Storage({}, {0x542f100, 0xc001106150}, {0x5422f80, 0xc000567b00})
    pkg/registry/networking/rest/storage_settings.go:95 +0x10e
k8s.io/kubernetes/pkg/registry/networking/rest.RESTStorageProvider.NewRESTStorage({}, {0x542f100, 0xc001106150}, {0x5422f80, 0xc000567b00})
    pkg/registry/networking/rest/storage_settings.go:48 +0x1b2
k8s.io/kubernetes/pkg/controlplane.(*Instance).InstallAPIs(0xc0001be850, {0x542f100, 0xc001106150}, {0x5422f80, 0xc000567b00}, {0xc0005cef38, 0x12, 0x1?})
    pkg/controlplane/instance.go:566 +0x275
k8s.io/kubernetes/pkg/controlplane.completedConfig.New({{0xc00041d170}, 0xc0008bc608}, {0x5456b20, 0xc0012fcb00})
    pkg/controlplane/instance.go:416 +0x87e
k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServer(0x48a84a0?, {0x5456b20, 0xc0012fcb00})
    cmd/kube-apiserver/app/server.go:217 +0x3e
k8s.io/kubernetes/cmd/kube-apiserver/app.CreateServerChain({0xc0006f0060?})
    cmd/kube-apiserver/app/server.go:196 +0x2a8
k8s.io/kubernetes/cmd/kube-apiserver/app.Run({0xc0012fc000?}, 0x7f4e7fb6faa8?)
    cmd/kube-apiserver/app/server.go:163 +0x2f2
k8s.io/kubernetes/cmd/kube-apiserver/app.NewAPIServerCommand.func2(0xc001334000?, {0xc0000fcd00?, 0x0?, 0x10?})
    cmd/kube-apiserver/app/server.go:129 +0xc5
github.com/spf13/cobra.(*Command).execute(0xc001334000, {0xc00004c130, 0x10, 0x11})
    vendor/github.com/spf13/cobra/command.go:856 +0x67c
github.com/spf13/cobra.(*Command).ExecuteC(0xc001334000)
    vendor/github.com/spf13/cobra/command.go:974 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
    vendor/github.com/spf13/cobra/command.go:902
k8s.io/component-base/cli.run(0xc001334000)
    vendor/k8s.io/component-base/cli/run.go:146 +0x305
k8s.io/component-base/cli.Run(0xc0000021a0?)
    vendor/k8s.io/component-base/cli/run.go:46 +0x1d
main.main()
    cmd/kube-apiserver/apiserver.go:34 +0x1e

I can't understand correctly how the to pass the right codec when create the RESTStorage, so it can differentiate between the old and new types and versions.

Doing a bit of archeology I've seen some related old issues, but I wanted to check with all of you if this comment from Clayton applies here

https://github.com/kubernetes/kubernetes/issues/35792#issuecomment-256949539

>  ... is not supported in any way (versioning codec inside another).

[Jordan Liggitt]

Every API group has a preferred version order the etcd encoder picks to encode in, controlled by the order versions are passed to scheme.SetVersionPriority when registering versions.

For resources which do not exist in the most preferred versions of the API group, an override is required to tell the etcd encoder to ignore the preferred version for that specific resource.

https://github.com/kubernetes/kubernetes/blob/master/pkg/kubeapiserver/default_storage_factory_builder.go#L60-L67 is where to specify that override.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/03005270-bea8-4b22-a9e0-dd0ca3569863n%40googlegroups.com.