[ANNOUNCEMENT] Kubernetes Emulation Version Update
Siyuan Zhang
Dear Kubernetes Community,
We are excited to share a few significant updates regarding the Kubernetes open source (k/k) development process, particularly the changes brought by the new Emulation Version. This announcement aims to inform you about the key changes and how they will impact your contributions and projects.
Key Changes Introduced by the Emulation Version
The Emulation Version (KEP-4330) was introduced in release 1.31 to Kubernetes control plane components to make upgrades safer by increasing the granularity of steps available to cluster administrators.
One key mechanism behind this enhancement is the option to emulate the capabilities (APIs, features, ...) of a prior Kubernetes release version by setting the `--emulated-version` flag.
The KEP promises to support emulation versions for up to N-3 minor versions, which means we need to preserve the component behavior from up to 3 minor versions ago in the code base. That would bring a few major changes to the general development and code review workflow.
Impact on Contributor Development Workflow
Normal Feature Stage Progression
Emulation version would NOT change the policies regarding feature progression, but require some accounting adjustments.
When the stage or default value of a feature changes, the version when the change happens needs to be tracked in the append-only featuregate.VersionedSpecs list, in the format like
Also, make sure to run git fetch $upstream-remote-name --tags && make update to update the versioned_feature_list.yaml file before submitting your PR, otherwise make verify would fail.
Feature Removal
Alpha features can be removed anytime as before because Emulation Version does not officially support emulating alpha features.
For Beta+ features, there is no change to when a feature can be deprecated, but there are new requirements of when the feature code can be removed:
First, set the feature gate to one of the two options: {LockToDefault: true, PreRelease: featuregate.GA} or {LockToDefault: true, PreRelease: featuregate.Deprecated}, then it can only be removed at least 3 versions later.
For feature usage: as long as a feature gate exists in the code base, functions and tests related to that feature should assume the feature could be enabled and disabled, even after feature graduation/deprecation. (we have a plan to police the feature cleanup to ensure code health)
For testing: disablement/enablement tests are still needed after a feature gate is locked to true/false. At the beginning of the tests, you would need to set the emulation version to the version before it is locked (more instructions), like
Feature Behavior Update or Ungated Code Change
Alpha features can be updated without restrictions because Emulation Version does not officially support emulating alpha features.
For changes to gated Beta+ features or ungated code, developers and reviewers can use the following process to decide if it is necessary to preserve the old behavior in the code base.
Here are some common change scenarios and whether the change needs to be preserved or not:
API changes [Not required]
Change of supported systems like adding Windows support [Yes]
Cluster admin facing changes like changing existing cmd flag [Yes]
Adding new cluster admin facing config options [Not required]
Bug fix [Not required]
Performance optimizations [Not required]
Unstable metrics change [Not required]
Adding new metrics [Not required]
Code refactoring [No]
All in all, this is a judgmental call of the developers and reviewers, and best effort to emulate old versions. We would not place any enforcement around this.
How to Preserve the Change
Our recommended way to preserve the behavior changes across version boundaries is to:
tie the behavior change to a new feature gate with version history.
the feature can start from Beta, on by default.
KEP is not required, and adding a new feature does not have to be heavy weighted, as stated in the feature gate lifecycle guidelines.
Here are some examples of lightweight feature gates to gate bug fixes/performance tuning:
* In very RARE cases, for example if you are changing supported custom resource formats (PR/130783), it is probably too cumbersome to add a feature gate for each change of the options, we can discuss whether or not to preserve the old behavior, and the best way to do it case by case.
API Changes
API changes still follow the same API change guidelines, which already considers backward compatibility.
For API promotion, controllers can just start using the newer version API without having to switch to the older version for emulation (refer to Impact on Users for the reason why behavior preservation is not required).
API Removal
All API resource type definitions should have the +k8s:prerelease-lifecycle-gen: tags.
Alpha APIs can be removed anytime because Emulation Version does not officially support emulating alpha APIs.
For Beta APIs, they would not be served if the emulated version is greater or equal to the version tagged with k8s:prerelease-lifecycle-gen:removed. The testing code should stay in the code base for 3 versions after the version in k8s:prerelease-lifecycle-gen:removed. with featuregatetesting.SetFeatureGateEmulationVersionDuringTest(t, utilfeature.DefaultFeatureGate, version.MustParse("<version before removed>")) before the test.
Impact on Users
For the control plane components (kube-apiserver, scheduler, controller manager), users can set `--emulated-version` flag to a version ranging from 1.binaryMinor-3 to 1.binaryMinor-1, the component will emulate the behavior (features & APIs served) of this version instead of the underlying binary version.
One exception is for using Beta controllers dependent on default-off Beta APIs in emulation mode: if the API has been promoted to a higher version after the emulated version, the controller would not work unless the user sets `--emulation-forward-compatible=true` (refer to KEP for how it works).
Next Steps
We are currently working on adding more guardrails of enforcing some of the new practices:
We encourage all contributors to review this documentation and familiarize themselves with the new guidelines and follow the new practice in writing and reviewing PRs.
Reach out to the emulation version team (@siyuanfoundation, @Jefftree, @yongruilin, @aaron-prindle, @jpbetz, @BenTheElder) on slack or email for feedback and ideas about improving the developer experience.
Resources
Here are some helpful resources for more information:
Emulation Version
KubeCon Talk
We appreciate your continued contributions to the Kubernetes project and look forward to seeing the positive impact of these changes.
Thank you,
The K8s Emulation Version Team
FAQs
Do I need to Follow the same guidelines for data plane components?
By release 1.33, the `--emulated-version` flag has only been added to the control plane components (kube-apiserver, scheduler, controller manager). The decision to add it to kubelet and kube-proxy is still pending.
While it is not a hard requirement to follow the same practice for data plane components, it would be easier for consistency across the whole k/k code base. Because
a feature may not be exclusively used in just kubelet/kube-proxy, and if it is used anywhere in the control plane, it would need to follow the emulation compatible approach.
we will add a few automated verifications to put guardrails around feature and API changes. It would be hard to identify features/APIs that are only used for the data plane and make exceptions.
We strongly recommend following the same guidelines for at least the feature gates and APIs in data plane components.
How do I do code deprecation under the new guidelines?
Emulation Version assumes a feature gate is off when emulating a version before the version it was first introduced. You can override this by adding an additional entry with the desired default value with version “1.0”. For example, to deprecate the code of LegacySidecarContainers (PR/130058), you can do
Then at the version when you used to remove the feature gate after confirming the deprecation would not cause any problem,
Can I rename an existing feature?
Alpha features can be renamed without restrictions because Emulation Version does not officially support emulating alpha features.
For Beta+ features, renaming is generally not a good idea because it might break existing users’ clusters if they do not pay close attention to the release notes.
We do not plan to support renaming features for emulation mode. If you absolutely have to rename an existing feature, you have to accept the fact it would break the users’ configuration if the old feature is explicitly enabled under emulation mode.