Kubernetes v1.29.0 has been built and pushed using Golang version 1.21.5.
The release notes have been updated in CHANGELOG-1.29.md, with a pointer to them on GitHub:
filename | sha512 hash |
---|---|
kubernetes.tar.gz | f07879916d7c4c7f8059ff9fd3c0006ce9bceb540874e183268a2bf2936df2632c4a3878a613cf2d695a80796e6c3eb52de5e3d83a73c91cb9a0bb5627091bae |
kubernetes-src.tar.gz | a37a7927224785625e9863c1e2dcbc88943593d003b8d126fee63770e6b8eff122004d0f80e1301de34e8a2d6ce208ec6fa55cad3bbe8631b92e5469f45bd00d |
filename | sha512 hash |
---|---|
kubernetes-client-darwin-amd64.tar.gz | 22da1d2a217a8de91c1a8c393d17eb5ca81e243a1a3e509f3a40fb91d623670ace4ee87a09218a184aaa2eec4ca9c5478b992b8c6f136c568767d6e9dea493bf |
kubernetes-client-darwin-arm64.tar.gz | cbc0cafecae18a50f98aaa8b508b1808a50b7a477638dc8699830a9dae7ffa83641f9fdb9f53616b32ebc8df84835fc847ea252c5ebe647c7d3462029a63b7a0 |
kubernetes-client-linux-386.tar.gz | f7ace756a3b6c56f2620d0ea6236fb94328c0a928094e4be7fbb78990a5771e8628bd93eac34017f3c33505c0248e8a64f933724a5fec6b322cf54dc30901985 |
kubernetes-client-linux-amd64.tar.gz | 6ff15bed6030c47e2ce90723500f08fa9968413f5b858456d4395bc67ab529b0b523ad0521e03be37664965e2fa588680aa0a5180054bc5cb3bafeef1497029b |
kubernetes-client-linux-arm.tar.gz | bafe1ca945c41ae671029d5398e564bac0753400ee3a50dc0b4979284c0a905e8c77575d8b64b303e9c776d09c919d27f1f99847390d4e2e1c43be826a8dc1a4 |
kubernetes-client-linux-arm64.tar.gz | f3bca520625eaf6e6dd9af4cc709ff20bfce4da298a03e0be8835013a95fe0d6a25693d7702a4739c9477f9d49d2492d739718245ff91716fff90f60279ff376 |
kubernetes-client-linux-ppc64le.tar.gz | e6ea574272cefe9fd6e8eea2bddd89e1d67d0cb560089813e7429f3fb6d98be0c6601f33c8a0b2364d3becfb93c0904c171096ed6cafc4071e08851566d70d82 |
kubernetes-client-linux-s390x.tar.gz | b67dd572d84382e3f713d56bfb371de379807dca52cc4a1e082d6f4720a12770354ef2c9eac93bfc73bc0ea5f4be293db3b6c03328b94a797c2da17b9c40d9f3 |
kubernetes-client-windows-386.tar.gz | 0cf4b665f46e36616452916d744367b0ae2238098705b32de79559d06ea551173ab95190a26e87bebc03e67a75dc6a65699be3ef3db12aef82f32b66fd5afb0e |
kubernetes-client-windows-amd64.tar.gz | 69cbe2b3942ba7d9c66e99f819adca94a9c7b420ad72cfd74407954c23ad70a4e7e76296824c4899f88232cabffe08d364c96af83bdaa538f29fa1303bcda2fa |
kubernetes-client-windows-arm64.tar.gz | 44b0d1a7904bc2bf754abecb9b43a9efdc7cf700ab18f2564d95d98b4e38fe6d91f066943db7105baea964f86d77ade3b1acd57c7aaf1cdf689660f0d4422960 |
filename | sha512 hash |
---|---|
kubernetes-server-linux-amd64.tar.gz | 651a8bf34acb6d61c39cc67ae23d9ef18204f95b309561d31f49da26c0c6a1b7585e7d7c2ac2f1522b2c326470a4e1ec9aa0dcf3bb1f66e1a41e6a2286e0aa5f |
kubernetes-server-linux-arm64.tar.gz | 7f1f58b05c923d860f2daa6d31906faf834584b1560f4eda01ba5499338d07a7f183030ab625557b1f5df50a5f0ea30d97d487e2571c85260e5b88fc3519cd43 |
kubernetes-server-linux-ppc64le.tar.gz | 3ca2af4a7d68c0d84ef65e69190daeb2392946c87c6b8e84ff8d5cf917c979f0778fc00040d4b471e71b8474ca57ac8fdf786f006260d4403b53f59a203a48f1 |
kubernetes-server-linux-s390x.tar.gz | dfa172456f98210e614a9a538b9027ba211cc19f6eec22a42d5e89ce12d7f5e7e58dfd3229bb974ecba31ffafdf1a5361aef18b9610a45614a181918d87500db |
filename | sha512 hash |
---|---|
kubernetes-node-linux-amd64.tar.gz | 8057197e9354e2e0f48aab18c0ce87e4ea39c1682cfd4c491c2bc83f8881787b09cb0c9b9f4d7bef8fbe53cc4056f5381745dbfde7f7474bb76a2358b8b3953e |
kubernetes-node-linux-arm64.tar.gz | 70d086c71f6258b1667bcb1efe60c15810b5b76848fdf26781c5a90efb8a78030e9ffb230bb0fd52d994f02b13c0b558c8e8ad3a42b601a0f9440a71cf91be2d |
kubernetes-node-linux-ppc64le.tar.gz | 2740f6ac0dfeebbe4ba8804b43ec5968997d9137de9a9432861c3e71e614cb84b309da31bde3554f896f829a570c21b833f0af241659ad326fa753a80f185ec4 |
kubernetes-node-linux-s390x.tar.gz | 9877d5a6cc84569efe30256ba5e8095f38bfa0b11c28892499a12b577b467b516880a33022d88f65263c7ffa2a9a3687ef52cb85fa611e95b14ae0c5b7a79c5c |
kubernetes-node-windows-amd64.tar.gz | 66b264de5e810bff31c4cf7cc575c3c57fed491fa4e21de7035dad76127e17d5fc88aff9f65277adf0826b255bf9b983f61c91bff2f8386d950f87509db6ec6b |
All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.
kube-proxy
and kubelet
during kubeadm upgrade plan --config
. This was a legacy behavior that was not well supported for upgrades and could be used only at the plan stage to determine if the configuration for these components stored in the cluster needs manual version migration. In the future, kubeadm
will attempt alternative component config migration approaches. (#120788, @chendave)kubeadm
: a separate "super-admin.conf" file is now deployed. The User in admin.conf
is now bound to a new RBAC Group kubeadm:cluster-admins
that has cluster-admin
ClusterRole
access. The User in super-admin.conf
is now bound to the system:masters
built-in super-powers / break-glass Group that can bypass RBAC. Before this change, the default admin.conf
was bound to system:masters
Group, which was undesired. Executing kubeadm init phase kubeconfig all
or just kubeadm init
will now generate the new super-admin.conf
file. The cluster admin can then decide to keep the file present on a node host or move it to a safe location. kubadm certs renew
will renew the certificate in super-admin.conf
to one year if the file exists; if it does not exist a "MISSING" note will be printed. kubeadm upgrade apply
for this release will migrate this particular node to the two file setup. Subsequent kubeadm releases will continue to optionally renew the certificate in super-admin.conf
if the file exists on disk and if renew on upgrade is not disabled. kubeadm join --control-plane
will now generate only an admin.conf
file that has the less privileged User. (#121305, @neolit123)CronJob
objects containing TZ
or CRON_TZ
in .spec.schedule
, accidentally enabled in v1.22
, is now disallowed. Use the .spec.timeZone
field instead, supported in v1.25+
clusters in default configurations. See https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#unsupported-timezone-specification for more information. (#116252, @soltysh)Removed the networking alpha
API ClusterCIDR
. (#121229, @aojea)
'kube-apiserver
: adds --authentication-config
flag for reading AuthenticationConfiguration
files. --authentication-config
flag is mutually exclusive with the existing --oidc-*
flags.' (#119142, @aramase)
'kube-scheduler
component config (KubeSchedulerConfiguration
) kubescheduler.config.k8s.io/v1beta3
is removed in v1.29
. Migrated kube-scheduler
configuration files to kubescheduler.config.k8s.io/v1
.' (#119994, @SataQiu)
A new sleep action for the PreStop
lifecycle hook was added, allowing containers to pause for a specified duration before termination. (#119026, @AxeZhan)
Added CEL expressions to v1alpha1 AuthenticationConfiguration
. (#121078, @aramase)
Added Windows support for InPlace Pod Vertical Scaling feature. (#112599, @fabi200123) [SIG Autoscaling, Node, Scalability, Scheduling and Windows]
Added ImageMaximumGCAge
field to Kubelet configuration, which allows a user to set the maximum age an image is unused before it's garbage collected. (#121275, @haircommander)
Added UserNamespacesPodSecurityStandards
feature gate to enable user namespace support for Pod Security Standards. Enabling this feature will modify all Pod Security Standard rules to allow setting: spec[.*].securityContext.[runAsNonRoot,runAsUser]
. This feature gate should only be enabled if all nodes in the cluster support the user namespace feature and have it enabled. The feature gate will not graduate or be enabled by default in future Kubernetes releases. (#118760, @saschagrunert) [SIG API Machinery, Auth, Node and Release]
Added optionalOldSelf
to x-kubernetes-validations
to support ratcheting CRD schema constraints. (#121034, @alexzielenski)
Added a new ServiceCIDR
type that allows to dynamically configure the cluster range used to allocate Service ClusterIPs
addresses. (#116516, @aojea)
Added a new ipMode
field to the .status
of Services where type
is set to LoadBalancer
. The new field is behind the LoadBalancerIPMode
feature gate. (#119937, @RyanAoh) [SIG API Machinery, Apps, Cloud Provider, Network and Testing]
Added options for configuring nf_conntrack_udp_timeout
, and nf_conntrack_udp_timeout_stream
variables of netfilter conntrack subsystem. (#120808, @aroradaman)
Added support for CEL expressions to v1alpha1 AuthorizationConfiguration
webhook matchConditions
. (#121223, @ritazh)
Added support for projecting certificates.k8s.io/v1alpha1
ClusterTrustBundle objects into pods. (#113374, @ahmedtd)
Added the DisableNodeKubeProxyVersion
feature gate. If DisableNodeKubeProxyVersion
is enabled, the kubeProxyVersion
field is not set. (#120954, @HirazawaUi)
Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps. The incorrect cost was evident when the result of a function was used in subsequent operations. (#119800, @jpbetz) [SIG API Machinery, Auth and Cloud Provider]
Fixed the API comments for the Job Ready
field in status. (#121765, @mimowo)
Fixed the API comments for the FailIndex
Job pod failure policy action. (#121764, @mimowo)
Go API: the ResourceRequirements
struct was replaced with VolumeResourceRequirements
for use with volumes. (#118653, @pohly)
Graduated Job BackoffLimitPerIndex
feature to beta
. (#121356, @mimowo)
Marked the onPodConditions
field as optional in Job
's pod failure policy. (#120204, @mimowo)
Promoted PodReadyToStartContainers
condition to beta
. (#119659, @kannon92)
The flowcontrol.apiserver.k8s.io/v1beta3
FlowSchema
and PriorityLevelConfiguration
APIs has been promoted to flowcontrol.apiserver.k8s.io/v1
, with the following changes:
PriorityLevelConfiguration
: the .spec.limited.nominalConcurrencyShares
field defaults to 30
only if the field is omitted (v1beta3 also defaulted an explicit 0
value to 30
). Specifying an explicit 0
value is not allowed in the v1
version in v1.29 to ensure compatibility with v1.28
API servers. In v1.30
, explicit 0
values will be allowed in this field in the v1
API. The flowcontrol.apiserver.k8s.io/v1beta3
APIs are deprecated and will no longer be served in v1.32. All existing objects are available via the v1
APIs. Transition clients and manifests to use the v1
APIs before upgrading to v1.32
. (#121089, @tkashem)The kube-proxy
command-line documentation was updated to clarify that --bind-address
does not actually have anything to do with binding to an address, and you probably don't actually want to be using it. (#120274, @danwinship)
The kube-scheduler
selectorSpread
plugin has been removed, please use the podTopologySpread
plugin instead. (#117720, @kerthcet)
The matchLabelKeys/mismatchLabelKeys
feature is introduced to the hard/soft PodAffinity/PodAntiAffinity
. (#116065, @sanposhiho)
When updating a CRD, per-expression cost limit check are now skipped for x-kubernetes-validations
rules of versions that are not mutated. (#121460, @jiahuif)
CSINodeExpandSecret
feature has been promoted to GA
in this release and is enabled by default. The CSI drivers can make use of the secretRef
values passed in NodeExpansion
request optionally sent by the CSI Client from this release onwards. (#121303, @humblec)
NodeStageVolume
calls will now be retried if the CSI node driver is not running. (#120330, @rohitssingh)
PersistentVolumeLastPhaseTransitionTime
is now beta and enabled by default. (#120627, @RomanBednar)
ValidatingAdmissionPolicy
type checking now supports CRDs and API extensions types. (#119109, @jiahuif)
kube-apiserver
: added --authorization-config
flag for reading a configuration file containing an apiserver.config.k8s.io/v1alpha1 AuthorizationConfiguration
object. The --authorization-config
flag is mutually exclusive with --authorization-modes
and --authorization-webhook-*
flags. The alpha
StructuredAuthorizationConfiguration
feature flag must be enabled for --authorization-config
to be specified. (#120154, @palnabarun)
kube-proxy
now has a new nftables-based mode, available by running
`kube-proxy --feature-gates NFTablesProxyMode=true --proxy-mode nftables`
This is currently an alpha-level feature and while it probably will not eat your data, it may nibble at it a bit. (It passes e2e testing but has not yet seen real-world use.)
At this point it should be functionally mostly identical to the iptables mode, except that it does not (and will not) support Service NodePorts on 127.0.0.1. (Also note that there are currently no command-line arguments for the nftables-specific config; you will need to use a config file if you want to set the equivalent of any of the --iptables-xxx
options.)
As this code is still very new, it has not been heavily optimized yet; while it is expected to eventually have better performance than the iptables backend, very little performance testing has been done so far. (#121046, @danwinship)
kube-proxy
: Added an option/flag for configuring the nf_conntrack_tcp_be_liberal
sysctl (in the kernel's netfilter conntrack subsystem). When enabled, kube-proxy
will not install the DROP
rule for invalid conntrack states, which currently breaks users of asymmetric routing. (#120354, @aroradaman)
A customizable OrderedScoreFuncs()
function was introduced. Out-of-tree plugins that used the scheduler's preemption interface could implement this function for custom preemption preferences or return nil to keep the current behavior. (#121867, @lianghao208)
Added apiextensions_apiserver_update_ratcheting_time
metric for tracking time taken during requests by feature CRDValidationRatcheting
. (#121462, @alexzielenski)
Added apiserver_envelope_encryption_dek_cache_filled
to measure number of records in data encryption key (DEK) cache. (#119878, @ritazh)
Added apiserver_watch_list_duration_seconds
metrics which will measure response latency distribution in seconds for watchlist requests broken by group, version, resource and scope. (#120490, @p0lyn0mial)
Added job_pods_creation_total
metrics for tracking Pods created by the Job controller labeled by events which triggered the Pod creation. (#121481, @dejanzele)
Added kubectl node drain
helper callbacks OnPodDeletionOrEvictionStarted
and OnPodDeletionOrEvictionFailed
; people extending kubectl
can use these new callbacks for more granularity. Deprecated the OnPodDeletedOrEvicted
node drain helper callback. (#117502, @adilGhaffarDev)
Added a new --init-only
command line flag to kube-proxy
. Setting the flag makes kube-proxy
perform its initial configuration that requires privileged mode, and then exit. The --init-only
mode is intended to be executed in a privileged init container, so that the main container may run with a stricter securityContext
. (#120864, @uablrek) [SIG Network and Scalability]
Added a new scheduler metric, pod_scheduling_sli_duration_seconds
, and started the deprecation for pod_scheduling_duration_seconds
. (#119049, @helayoty)
Added a return value to QueueingHint
to indicate an error. If QueueingHint
returns an error, the scheduler logs it and treats the event as a QueueAfterBackoff
so that the Pod won't be stuck in the unschedulable pod pool. (#119290, @carlory)
Added apiserver identity to the following metrics: apiserver_envelope_encryption_key_id_hash_total
, apiserver_envelope_encryption_key_id_hash_last_timestamp_seconds
, apiserver_envelope_encryption_key_id_hash_status_last_timestamp_seconds
, apiserver_encryption_config_controller_automatic_reload_failures_total
, apiserver_encryption_config_controller_automatic_reload_success_total
, apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds
.
Fixed bug to surface events for the following metrics: apiserver_encryption_config_controller_automatic_reload_failures_total
, apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds
, apiserver_encryption_config_controller_automatic_reload_success_total
. (#120438, @ritazh)
Added container filesystem to the ImageFsInfoResponse
. (#120914, @kannon92)
Added multiplication functionality to Quantity
. (#117411, @tenzen-y)
Added new feature gate called RuntimeClassInImageCriApi
to address kubelet
changes needed for KEP 4216. Noteable changes:
Added support for split image filesystem in kubelet. (#120616, @kannon92)
Bumped cel-go
to v0.17.7
and introduced set ext
library with new options. (#121577, @cici37)
Bumped distroless-iptables
to 0.3.2
based on Go 1.21.1
. (#120527, @cpanato)
Bumped distroless-iptables
to 0.3.3
based on Go 1.21.2
. (#121073, @cpanato)
Bumped distroless-iptables
to 0.4.1
based on Go 1.21.3
. (#121216, @cpanato)
Bumped distroless-iptables to 0.4.1 based on Go 1.21.3
. (#121871, @cpanato)
CEL can now correctly handle a CRD openAPIV3Schema
that has neither Properties
nor AdditionalProperties
. (#121459, @jiahuif)
CEL cost estimator no longer treats enums as unbounded strings when determining its length. Instead, the length is set to the longest possible enum value. (#121085, @jiahuif) [SIG API Machinery]
CRI: image pull per runtime class is now supported. (#121121, @kiashok)
Certain requestBody
parameters in the OpenAPI v3
are now correctly marked as required. (#120735, @Jefftree)
Changed kubectl help
to display basic details for subcommands from plugins. (#116752, @xvzf)
Changed the KMSv2KDF
feature gate to be enabled by default. (#120433, @enj) [SIG API Machinery, Auth and Testing]
Client-side apply will now use OpenAPI v3
by default. (#120707, @Jefftree)
Decoding etcd's response now respects the timeout context. (#121614, @HirazawaUi)
Decoupled TaintManager
from NodeLifeCycleController
(KEP-3902). (#119208, @atosatto)
Enabled traces for KMSv2 encrypt/decrypt operations. (#121095, @aramase)
Fixed kube-proxy
panicking on exit when the Node
object changed its PodCIDR
. (#120375, @pegasas)
Fixed bugs in handling of server-side apply, create, and update API requests for objects containing duplicate items in keyed lists.
create
or update
API request with duplicate items in a keyed list no longer wipes out managedFields. Examples include env var entries with the same name, or port entries with the same containerPort in a pod spec.Fixed overriding default KubeletConfig
fields in drop-in configs if not set. (#121193, @sohankunkerkar)
Graduated API List chunking (aka pagination) feature to stable
. (#119503, @wojtek-t)
Graduated the ReadWriteOncePod
feature gate to GA
. (#121077, @chrishenzie)
Graduated the following kubelet resource metrics to general availability:
container_cpu_usage_seconds_total
container_memory_working_set_bytes
container_start_time_seconds
node_cpu_usage_seconds_total
node_memory_working_set_bytes
pod_cpu_usage_seconds_total
pod_memory_working_set_bytes
resource_scrape_error
Deprecated (renamed) scrape_error
in favor of resource_scrape_error
(#116897, @Richabanker) [SIG Architecture, Instrumentation, Node and Testing]
Implemented API for streaming for the etcd
store implementation. When sendInitialEvents ListOption
is set together with watch=true
, it begins the watch stream with synthetic init events followed by a synthetic Bookmark
, after which the server continues streaming events. (#119557, @p0lyn0mial)
Improved memory usage of kube-scheduler
by dropping the .metadata.managedFields
field that kube-scheduler
doesn't require. (#119556, @linxiulei)
In a scheduler with Permit
plugins, when a Pod is rejected during WaitOnPermit
, the scheduler records the plugin. The scheduler will use the record to honor cluster events and queueing hints registered
for the plugin, to inform whether to retry the pod. (#119785, @sanposhiho)
In-tree cloud providers are now switched off by default. Please use DisableCloudProviders
and DisableKubeletCloudCredentialProvider
feature flags if you still need this functionality. (#117503, @dims)
Introduced new apiserver metric apiserver_flowcontrol_current_inqueue_seats
. This metric is analogous to apiserver_flowcontrol_current_inqueue_requests
, but tracks the total number of seats, as each request can take more than one seat. (#119385, @andrewsykim)
Introduced the job_finished_indexes_total
metric for the BackoffLimitPerIndex
feature. (#121292, @mimowo)
Kubeadm: supported updating certificate organization during kubeadm certs renew
operation. (#121841, @SataQiu)
Kubernetes is now built with Go v1.21.3
. (#121149, @cpanato)
List of metric labels can now be configured by supplying a manifest using the --allow-metric-labels-manifest
flag. (#118299, @rexagod)
Listed the pods using <PVC>
as an ephemeral storage volume in "Used by:" part of the output of kubectl describe pvc <PVC>
command. (#120427, @MaGaroo)
Migrated the nodevolumelimits
scheduler plugin to use contextual logging. (#116884, @mengjiao-liu)
Migrated the volumebinding scheduler plugins
to use contextual logging. (#116803, @mengjiao-liu)
Priority and Fairness feature is now stable
, the feature gate will be removed in v1.31
. (#121638, @tkashem)
Promoted PodHostIPs
condition to beta
. (#120257, @wzshiming)
Promoted PodHostIPs
condition to beta
. (#121477, @wzshiming)
Promoted PodReplacementPolicy
to beta
. (#121491, @dejanzele)
Promoted ServiceNodePortStaticSubrange
to stable and lock to default. (#120233, @xuzhenglun)
Promoted plugin subcommand resolution feature to beta
. (#120663, @ardaguclu)
Removed /livez
livezchecks for KMS v1 and v2 to ensure KMS health does not cause kube-apiserver
restart. KMS health checks are still in place as a healthz and readiness checks. (#120583, @ritazh)
Restartable init containers resource in pod autoscaler are now calculated. (#120001, @qingwave)
Sidecar termination is now serialized and each sidecar container will receive a SIGTERM
after all main containers and later starting sidecar containers have terminated. (#120620, @tzneal)
The CRD validation rule with feature gate CustomResourceValidationExpressions
was promoted to GA
. (#121373, @cici37)
The KMSv2 features with feature gates KMSv2
and KMSv2KDF
are promoted to GA
. The KMSv1
feature gate is now disabled by default. (#121485, @ritazh)
The --interactive
flag in kubectl delete
is now visible to all users by default. (#120416, @ardaguclu)
The CloudDualStackNodeIPs
feature is now beta
, meaning that when using an external cloud provider that has been updated to support the feature, you can pass comma-separated dual-stack --node-ips
to kubelet
and have the cloud provider take both IPs into account. (#120275, @danwinship)
The Dockerfile
for the kubectl image has been updated with the addition of a specific base image and essential utilities (bash and jq). (#119592, @rayandas)
The SidecarContainers
feature has graduated to beta
and is enabled by default. (#121579, @gjkim42)
The kube-apiserver
will now expose four new metrics to inform about errors on the clusterIP and nodePort allocation logic. (#120843, @aojea)
The volume_zone
plugin will consider beta
labels as GA
labels during the scheduling process. Therefore, if the values of the labels are the same, PVs with beta
labels can also be scheduled to nodes with GA
labels. (#118923, @AxeZhan)
Updated the generic apiserver library to produce an error if a new API server is configured with support for a data format other than JSON, YAML, or Protobuf. (#121325, @benluddy) [SIG API Machinery]
Use of secret-based service account tokens now adds an authentication.k8s.io/legacy-token-autogenerated-secret
or authentication.k8s.io/legacy-token-manual-secret
audit annotation containing the name of the secret used. (#118598, @yuanchen8911) [SIG Auth, Instrumentation and Testing]
--sync-frequency
will not affect the update interval of volumes that use ConfigMaps
or Secrets
when the configMapAndSecretChangeDetectionStrategy
is set to Cache
. The update interval is only affected by node.alpha.kubernetes.io/ttl
node annotation." (#120255, @likakuli)
CRDValidationRatcheting
: added support for ratcheting x-kubernetes-validations
in schema. (#121016, @alexzielenski)
DevicePluginCDIDevices
feature has been graduated to beta
and enabled by default in the kubelet. (#121254, @bart0sh)
ValidatingAdmissionPolicy
now preserves types of composition variables, and raises type-related errors early. (#121001, @jiahuif)
cluster/gce
: added webhook to replace PersistentVolumeLabel
admission controller. (#121628, @andrewsykim)
dra
: the scheduler plugin now avoids additional scheduling attempts in some cases by falling back to SSA after a conflict. (#120534, @pohly)
kube-apiserver
added:
alpha
support (guarded by the ServiceAccountTokenJTI
feature gate) for adding a jti
(JWT ID) claim to service account tokens it issues, adding an authentication.kubernetes.io/credential-id
audit annotation in audit logs when the tokens are issued, and authentication.kubernetes.io/credential-id
entry in the extra user info when the token is used to authenticate.alpha
support (guarded by the ServiceAccountTokenPodNodeInfo
feature gate) for including the node name (and uid, if the node exists) as additional claims in service account tokens it issues which are bound to pods, and authentication.kubernetes.io/node-name
and authentication.kubernetes.io/node-uid
extra user info when the token is used to authenticate.alpha
support (guarded by the ServiceAccountTokenNodeBinding
feature gate) for allowing TokenRequests
that bind tokens directly to nodes, and (guarded by the ServiceAccountTokenNodeBindingValidation feature gate) for validating the node name and uid still exist when the token is used. (#120780, @munnerz)kube-controller-manager
: The LegacyServiceAccountTokenCleanUp
feature gate is now beta
and enabled by default. When enabled, legacy auto-generated service account token secrets are auto-labeled with a kubernetes.io/legacy-token-invalid-since
label if the credentials have not been used in the time specified by --legacy-service-account-token-clean-up-period
(defaulting to one year), and are referenced from the .secrets
list of a ServiceAccount object, and are not referenced from pods. This label causes the authentication layer to reject use of the credentials. After being labeled as invalid, if the time specified by --legacy-service-account-token-clean-up-period
(defaulting to one year) passes without the credential being used, the secret is automatically deleted. Secrets labeled as invalid which have not been auto-deleted yet can be re-activated by removing the kubernetes.io/legacy-token-invalid-since
label. (#120682, @yt2985)
kube-proxy
will only install the DROP
rules for invalid conntrack
states if the nf_conntrack_tcp_be_liberal
is not set. (#120412, @aojea)
kube-scheduler
implemented scheduling hints for the NodeUnschedulable
plugin. The scheduling hints allow the scheduler to only retry scheduling a Pod
that was previously rejected by the NodeSchedulable
plugin if a new Node
or a Node
update sets .spec.unschedulable
to false. (#119396, @wackxu)
kube-scheduler
implements scheduling hints for the NodeAffinity
plugin. The scheduling hints allow the scheduler to only retry scheduling a Pod
that was previously rejected by the NodeAffinity
plugin if a new Node
or a Node
update matches the Pod
's node affinity. (#119155, @carlory)
kubeadm
: promoted feature gate EtcdLearnerMode
to beta
. Learner mode for joining etcd
members is now enabled by default. (#120228, @pacoxu)
kubeadm
: turned on feature gate MergeCLIArgumentsWithConfig
to merge the config from flag and config file, otherwise, if the flag --ignore-preflight-errors
is set from the CLI, then the value from config file will be ignored. (#119946, @chendave)
kubeadm
: will now allow deploying a kubelet that is 3 versions older than the version of kubeadm
(N-3). This aligns with the recent change made by SIG Architecture that extends the support skew between the control plane and kubelets. Tolerate this new kubelet skew for the commands init
, join
and upgrade
. Note that if the kubeadm
user applies a control plane version that is older than the kubeadm
version (N-1 maximum) then the skew between the kubelet and control plane would become a maximum of N-2. (#120825, @pacoxu)
kubelet
, when using --cloud-provider=external
, will now initialize the node addresses with the value of --node-ip
, if it exists, or waits for the cloud provider to assign the addresses. (#121028, @aojea)
kubelet
allows pods to use the net.ipv4.tcp_fin_timeout
, “net.ipv4.tcp_keepalive_intvl” and “net.ipv4.tcp_keepalive_probes“ sysctl by default; Pod Security Admission allows this sysctl in v1.29+
versions of the baseline and restricted policies. (#121240, @HirazawaUi)
kubelet
now allows pods to use the net.ipv4.tcp_keepalive_time
sysctl by default and the minimal kernel version is 4.5; Pod Security Admission allows this sysctl in v1.29+
versions of the baseline and restricted policies. (#118846, @cyclinder)
kubelet
now emits a metric for end-to-end pod startup latency, including image pull. (#121041, @ruiwen-zhao)
kubelet
now exposes latency metrics of different stages of the node startup. (#118568, @qiutongs)
kubectl rollout restart
without specifying a particular deployment. (#120118, @Ithrael)not enough cpus available to satisfy request
to not enough cpus available to satisfy request: <num_requested> requested, only <num_available> available
. (#121059, @matte21)kubeproxy
. (#120105, @princepereira)apiserver
failures was fixed in WaitForPodsResponding
(#120559, @pohly)/cluster
script, without affecting CCM. New variable name: KUBE_CONTROLLER_MANAGER_TEST_ARGS
. (#120524, @jprzychodzen) [SIG Cloud Provider]k8s.io/dynamic-resource-allocation
: DRA drivers updating to this release are compatible with Kubernetes v1.27
and v1.28
. (#120868, @pohly)kubeadm
: printing the default component configs for reset
and join
is now unsupported.' (#119346, @chendave)kubeadm
: removed system:masters
organization from etcd/healthcheck-client
certificate.' (#119859, @SataQiu)CAP_NET_RAW
to netadmin debug profile and removed privileges when debugging nodes. (#118647, @mochizuki875)kubelet
without specifying a name. They will now get a visible validation error. (#119522, @YTGhost)kube-apiserver
). (#119105, @sanposhiho)ExternalTrafficPolicy
for Services
with ExternalIPs
. (#119150, @tnqn)error
to info
for uncached partitions when using CRI stats provider. (#100448, @saschagrunert)replace()
to handle a zero length replacement string correctly. Previously this would cause the estimated cost to be higher than it should be. (#120097, @jpbetz) [SIG API Machinery]APIServices
. (#120108, @tnqn)externalTrafficPolicy: Local
services. (#121116, @alexanderConstantinescu)kubectl events
not filtering events by GroupVersion
for resources with a full name. (#120119, @Ithrael)systemLogQuery
service name matching. (#120678, @rothgar)1.27
scheduling regression that PostFilter
plugin may not function if previous PreFilter
plugins return Skip
. (#119769, @Huang-Wei)v1.26
regression scheduling bug by ensuring that preemption is skipped when a PreFilter
plugin returns UnschedulableAndUnresolvable
. (#119778, @sanposhiho)v1.28.0
regression where kube-controller-manager
can crash when StatefulSet
with Parallel
policy and PVC labels are scaled up. (#121142, @aleksandra-malinowska)v1.28
regression around restarting init containers in the right order relative to normal containers. (#120281, @gjkim42)v1.28
regression handling negative index json patches. (#120327, @liggitt)v1.28
regression in scheduler: a pod with concurrent events could incorrectly get moved to the unschedulable queue where it could get stuck until the next periodic purging after 5 minutes, if there was no other event for it. (#120413, @pohly)SidecarContainers
feature enabled. (#120269, @gjkim42)Services
using finalizers may hold onto ClusterIP
and/or NodePort
allocated resources for longer than expected if the finalizer is removed using the status subresource. (#120623, @aojea)cgroupv2
systems where swap
is disabled. (#120784, @elezar)Always
, were erroneously reused by a regular container. (#119447, @gjkim42) [SIG Node and Testing]containerRestartPolicy
of Always
, were erroneously reused by a regular container. (#120461, @gjkim42)Always
, were erroneously reused by a regular container. (#120715, @gjkim42) [SIG Node]TopologyCache
's HasPopulatedHints
method. (#118189, @Miciah)CLIENTSET_PKG: unbound variable
) when invoking deprecated generate-groups.sh
script. (#120877, @soltysh)kube-proxy
where it might refuse to start if given single-stack IPv6
configuration options on a node that has both IPv4
and IPv6
IPs. (#121008, @danwinship)PodDisruptionConditions
by default, that prevented the control plane's pod garbage collector from deleting pods that contained duplicated field keys (environmental variables with repeated keys or container ports). (#121103, @mimowo)v1.27
configurations in kube-apiserver
: fixed the AggregatedDiscoveryEndpoint
feature (beta
in v1.27+
) to successfully fetch discovery information from aggregated API servers that do not check Accept
headers when serving the /apis
endpoint. (#119870, @Jefftree)EventedPLEG
feature gate is enabled. (#120942, @sairameshv)v1.27.0
in the scheduler framework when running score plugins. The skippedScorePlugins
number might be greater than enabledScorePlugins
, so when initializing a slice the cap(len(skippedScorePlugins) - len(enabledScorePlugins))
is negative, which is not allowed. (#121632, @kerthcet)unschedulable
queue instead of the backoff
queue. This happened when some plugin previously declared the pod as unschedulable
and then in a later attempt encounters some other error. Scheduling of that pod then got delayed by up to five minutes, after which periodic flushing moved the pod back into the active
queue. (#120334, @pohly)StatefulSet
might not restart a pod after eviction or node failure. (#120398, @aleksandra-malinowska)CronJob
could fail to clean up Jobs when the ResourceQuota
for Jobs
had been reached. (#119776, @ASverdlov)StatefulSet
might not restart a pod after eviction or node failure. (#121389, @aleksandra-malinowska)garbagecollection
controller registering duplicate event handlers if discovery requests failed. (#117992, @liggitt)container_start_time_seconds
had timestamp equal to container start time. (#120518, @saschagrunert) [SIG Instrumentation, Node and Testing]ImageLocality
plugin. (#116938, @olderTaoist)loadbalancer
and endpoint
in kubeproxy
mock test framework. (#120723, @princepereira)podRecreationPolicy: Failed
is used, and the number of terminating pods exceeds parallelism. (#121147, @kannon92)APIservices
panicking and affected health check introduced in release v1.28.0
. (#120814, @Jefftree)kubelet
. (#119986, @ruiwen-zhao)kubectl logs POD_NAME -f
is running. (#115702, @xyz-li)progressNotify
option set is to be created, and the registry hasn't provided a newFunc
, return an error. (#120212, @p0lyn0mial) [SIG API Machinery]kubectl wait --for
. It is now possible to use simple filter expressions which match on a field's content. (#118748, @andreaskaris)wait.PollUntilContextTimeout
function, if immediate
is true, the condition will now be invoked before waiting, guaranteeing that the condition is invoked at least once and then wait a interval before executing again. (#119762, @AxeZhan)pod_start_duration_seconds
were changed to {0.5, 1, 2, 3, 4, 5, 6, 8, 10, 20, 30, 45, 60, 120, 180, 240, 300, 360, 480, 600, 900, 1200, 1800, 2700, 3600}
. (#120680, @ruiwen-zhao)CVE-2023-44487
and CVE-2023-39325
for the API server when the client is unauthenticated. The mitigation may be disabled by setting the UnauthenticatedHTTP2DOSMitigation
feature gate to false
(it is enabled by default). An API server fronted by an L7 load balancer that already mitigates these http/2 attacks may choose to disable the kube-apiserver mitigation to avoid disrupting load balancer -> kube-apiserver connections if http/2 requests from multiple clients share the same backend connection. An API server on a private network may opt to disable the kube-apiserver mitigation to prevent performance regressions for unauthenticated clients. Authenticated requests rely on the fix in golang.org/x/net v0.17.0
alone. https://issue.k8s.io/121197 tracks further mitigation of http/2 attacks by authenticated clients. (#121120, @enj)ClusterTrustBundleAttest
plugin is enabled. (#120779, @enj)apiserver_request_body_size_bytes
to track the size distribution of requests by resource
and verb
. (#120474, @YaoC) [SIG API Machinery and Instrumentation]DaemonSet
rolling update to exclude nodes if scheduling constraints are not met. This eliminates the problem of rolling updates to a DaemonSet
getting stuck around tolerations. (#119317, @mochizuki875)ProviderID
is updated. (#120492, @cezarygerard)status.loadBalancer
of a Service whose spec.type
is not LoadBalancer
was previously allowed, but any update to the metadata
or spec
would wipe that field. Setting this field is no longer permitted unless spec.type
is LoadBalancer
. In the very unlikely event that this has unexpected impact, you can enable the AllowServiceLBStatusOnNonLB
feature gate, which will restore the previous behavior. If you do need to set this, please file an issue with the Kubernetes project to help contributors understand why you need it. (#119789, @thockin)--bind-address
parameter in kube-proxy is misleading, no port is opened with this address. Instead it is translated internally to "nodeIP". The nodeIPs for both families are now taken from the Node object if --bind-address
is unspecified or set to the "any" address (0.0.0.0 or ::). It is recommended to leave --bind-address
unspecified, and in particular avoid to set it to localhost (127.0.0.1 or ::1) (#119525, @uablrek) [SIG Network and Scalability]kube-openapi
to remove invalid defaults: OpenAPI spec no longer includes default of {}
for certain fields where it did not make sense. (#120757, @alexzielenski)/run/crio/crio.sock
don't see strange behaviour from CRI stats provider. (#118704, @dgl)kubelet
if target path directory already exists on the node. (#119735, @akankshapanse)cluster-bootstrap
: improved the security of the functions responsible for generation and validation of bootstrap tokens. (#120400, @neolit123)etcd
: updated to v3.5.10
. (#121566, @mzaian)k8s.io/dynamic-resource-allocation/controller:
UnsuitableNodes
can now handle a mix of allocated and unallocated claims correctly. (#120338, @pohly)k8s.io/dynamic-resource-allocation/controller
: ResourceClaimParameters
and ResourceClassParameters
validation errors are now visible on ResourceClaim
, ResourceClass
and Pod
. (#121065, @byako)k8s.io/dynamic-resource-allocation
: can now handle a selected
node which isn't listed as potential
node. (#120871, @pohly)kube-proxy
now reports its health more accurately in dual-stack clusters when there are problems with only one IP family. (#118146, @aroradaman)kubeadm
: Fixed the bug where it always did CRI detection when --config
was passed, even if it is not required by the subcommand. (#120828, @SataQiu)kubeadm
: fixed nil
pointer when etcd
member is already removed. (#119753, @pacoxu)kubeadm
: fixed the bug where --image-repository
flag is missing for some init phase sub-commands. (#120072, @SataQiu)kubeadm
: improved the logic that checks whether a systemd
service exists. (#120514, @fengxsong)kubeadm
: will now use universal deserializer to decode static pod. (#120549, @pacoxu)kubectl prune v2
: Switched annotation from contains-group-resources
to contains-group-kinds
, because this is what we defined in the KEP and is clearer to end-users. Although the functionality is in alpha
, we will recognize the prior annotation. This migration support will be removed in beta
/GA
. (#118942, @justinsb)kubectl
will not print events if --show-events=false
argument is passed to describe PVC subcommand. (#120380, @MaGaroo)scheduler
: Fixed missing field apiVersion
from events reported by the taint manager. (#114095, @aimuz)local-up-cluster.sh
, facilitating local debugging. (#120312, @HirazawaUi)caches populated
log messages. (#119796, @sttts)kube-proxy
by allowing to set sysctl
values lower than the existing one. (#120448, @aroradaman)kube-apiserver
HTTP logs for impersonated requests. (#119795, @sttts)--cloud-provider
and --cloud-config
CLI parameters in kube-apiserver. These parameters will be removed in a future release. (#120903, @dims) [SIG API Machinery][Slow]
via the DriverInfo.FeatureTag
field is no longer supported. (#121391, @pohly)vsphere
cloud provider would not trust a certificate if:x509.UnknownAuthorityError
)x509.HostnameError
)Adding GroupVersion
log line was constantly repeated without any group version changes. (#119825, @Jefftree)ResourceClaim
names are now more readable because of an additional hyphen before the random suffix (<pod name>-<claim name>-<random suffix>
). (#120336, @pohly)JobReadyPods
to stable
. The feature gate can no longer be disabled. (#121302, @stuton)kube-controller-manager
by dropping the .metadata.managedFields
field that kube-controller-manager
doesn't require. (#118455, @linxiulei)GOTOOLCHAIN
and otherwise ensure ./.go-version
is used. (#120279, @BenTheElder)NodeUnschedulable
Filter to avoid unnecessary calculations. (#119399, @wackxu)PersistentDisk
volumes were using them in read-only mode. This validation provided very little value at relatively host implementation cost, and will no longer be validated. If this is a problem for a specific use-case, please set the SkipReadOnlyValidationGCE
gate to false to re-enable the validation, and file a Kubernetes bug with details. (#121083, @thockin)Services
only forward traffic on the port and protocol specified. (#120069, @aojea)GA
feature gate about CSIMigrationvSphere
. (#121291, @bzsuni)GA
feature gate about ProbeTerminationGracePeriod
. (#121257, @bzsuni)GA
feature gate for JobTrackingWithFinalizers
in v1.28
. (#119100, @bzsuni)GA
ed feature gate TopologyManager
. (#121252, @tukwila)GA
ed feature gates OpenAPIV3
. (#121255, @tukwila)GA
ed feature gates SeccompDefault
. (#121246, @tukwila)1.22
. (#119537, @ardaguclu)CronJobTimeZone
feature gate (the feature is stable and always enabled)DownwardAPIHugePages
feature gate (the feature is stable and always enabled) (#120249, @pacoxu) [SIG Apps and Node]GRPCContainerProbe
feature gate (the feature is stable and always enabled). (#120248, @pacoxu)apiserver_request_body_sizes
metric to apiserver_request_body_size_bytes
. (#120503, @dgrisonnet)job_controller_job_sync_duration_seconds
metric from 4ms
to 1min
. (#120577, @alculquicondor)horizontalpodautoscaling
and clusterrole-aggregation
controllers now assume the autoscaling/v1
and rbac.authorization.k8s.io/v1
APIs are available. If you disable those APIs and do not want to run those controllers, exclude them by passing --controllers=-horizontalpodautoscaling
or --controllers=-clusterrole-aggregation
to kube-controller-manager
. (#117977, @liggitt) [SIG API Machinery and Cloud Provider]ComponentSLIs
feature-gate and served at /metrics/slis
are now GA and unconditionally enabled. The feature-gate will be removed in v1.31
. (#120574, @logicalhan)v1.3.0
. (#119969, @saschagrunert)cri-tools
to v1.28.0
. (#119933, @saschagrunert)distroless-iptables
to use registry.k8s.io/build-image/distroless-iptables:v0.3.1
. (#120352, @saschagrunert)1.1.10
. (#121739, @ty-dc)coredns
to v1.11.1
. (#120116, @tukwila)EnqueueExtensions
from plugins other than PreEnqueue
, PreFilter
, Filter
, Reserve
and Permit
are now ignored. It reduces the number of kinds of cluster events the scheduler needs to subscribe/handle. (#121571, @sanposhiho)GetPodQOS(pod *core.Pod)
function now returns the stored value from PodStatus.QOSClass
, if set. To compute/evaluate the value of QOSClass
from scratch, ComputePodQOS(pod*core.Pod)
must be used. (#119665, @vinaykul)RetroactiveDefaultStorageClass
feature gate that graduated to GA in v1.28
and was unconditionally enabled has been removed in v1.29
. (#120861, @RomanBednar)Statefulset
now waits for new replicas in tests when removing .start.ordinal
. (#119761, @soltysh)ValidatingAdmissionPolicy
and ValidatingAdmissionPolicyBinding
objects are persisted in etcd
using the v1beta1
version. Either remove alpha objects, or disable the alpha ValidatingAdmissionPolicy
feature in a v1.27
server before upgrading to a v1.28
server with the beta feature and API enabled. (#120018, @liggitt)client-go
: k8s.io/client-go/tools
events and record packages now have new APIs for specifying a context and logger. (#120729, @pohly)kube-controller-manager
help now includes controllers behind a feature gate in --controllers
flag. (#120371, @atiratree)kubeadm
: removed system:masters
organization from apiserver-etcd-client
certificate. (#120521, @SataQiu)kubeadm
: removed leftover disclaimer that could be seen in the kubeadm init phase certs
command help screen, since the "certs" phase of "init" is no longer alpha. (#121172, @SataQiu)kubeadm
: updated warning message when swap space is detected. When swap is active on Linux, kubeadm
explains that swap is supported for cgroup v2 only and is beta but disabled by default. (#120198, @pacoxu)kubectl
will not support the /swagger-2.0.0.pb-v1
endpoint that has been long deprecated. (#119410, @Jefftree)scheduler
: handling of unschedulable pods because a ResourceClass
is missing is a bit more efficient and no longer relies on periodic retries. (#120213, @pohly)
Contributors, the CHANGELOG-1.29.md has been bootstrapped with v1.29.0 release notes and you may edit now as needed.
Published by your Kubernetes Release Managers.