Kubernetes v1.32.0 has been built and pushed using Golang version 1.23.3.
The release notes have been updated in CHANGELOG-1.32.md, with a pointer to them on GitHub:
filename | sha512 hash |
---|---|
kubernetes.tar.gz | 6ff36174fd78b83b7cf2a05ff991725efcd3529f2c8c9924586258d359af5049062c1f4aff6d8e9044981781c80de6cc738365b85e47fd2e2971cd53a36882c2 |
kubernetes-src.tar.gz | 3c401843abef2e74c2e20557f1a7165623dc98c1e290cd629035ac323a491125c666966c638e8baf9f1cb039f330e1b80a4795551145dc04c323c487c25ced22 |
filename | sha512 hash |
---|---|
kubernetes-client-darwin-amd64.tar.gz | adab0d3f2947323dc8690aebe8bf9aca0179a460ee43dd4144677d293d9d75cfe8c363d1f377d03533758aef891bba3fe4c884ec16e94b84dad83c5de1570a98 |
kubernetes-client-darwin-arm64.tar.gz | 155376003480f5689a503bd3857606813882bc45bdf7d3b07a002d282cbb74fc585844ccffd00ca5f49ed3e65721c9f63d25d67a19f09a9f3257416017e83e83 |
kubernetes-client-linux-386.tar.gz | 96716dcadf056057f9e9e7cda99935a95381333b8dfa101c3c168903c7dbdef2994d59585e8ee2d362c552f04038c3a0b47077ab7506a2a98ccbd1c1d91f183c |
kubernetes-client-linux-amd64.tar.gz | 302e02599f0bdd3665aadb9e16a2f1f50712bf875f7525a0184450c0dcd59cefbfa67c3211aaa4d4eca197bd9fb49e1de35ffb9d579527ed4830d04400b09ef7 |
kubernetes-client-linux-arm.tar.gz | b104c1fcea77ee2c614ee9089e94accc8aa5f915315711a974a51f0f5e0899e4741dcd6a046fea69264cb6933ce5c84ccaa9f7c9c1849def7da098ad5d2cc845 |
kubernetes-client-linux-arm64.tar.gz | 378face3b06a2d062aa734ba0b9fd13f20f877bd611556c352be6246fa70067e60ee44fe55c4c0f064b5715b311075b4db540c7cc52d1a2af4b96a563625f4f1 |
kubernetes-client-linux-ppc64le.tar.gz | 6172956799cdf4a65fa5450f26ed4e491a935473418daacb51686d93745445747b893eea701af9d8c508ac8cbd3f4cbabef6cb17b94448e5c2732dc13d35e046 |
kubernetes-client-linux-s390x.tar.gz | a3ebc9175317aa93acc26edee8b7f5502a0d9405c1b04b39d907bbebe022e23c9fed058f5cede7045e388d1706c2657b0282d14862e01a7b34002e88e7224d8f |
kubernetes-client-windows-386.tar.gz | affcae9e4065cdfc130c6bb690539a631c1be0d992e9b02efbd49e0d519275d23e77c2ba5aa563ed5b89498e8bb26ce73019575bd557152b0d554578a96bf945 |
kubernetes-client-windows-amd64.tar.gz | 9619c05daa723c7853daae3432f771a31a6fe57887c32e5e592eb3bf619a636a8c3c2dec0eba2ec60382dc3d2ea8b0dc58e5b5f15fa43cdb7371a3ec0a7e4f55 |
kubernetes-client-windows-arm64.tar.gz | e4c8c0d70d5c825dccd3dfa4517baf07f863deac440560c176206b626b0ebd585f0c0601e8956e4a73eda41d31d963895fe337a700a9c1853b7ee3ff2bd568e3 |
filename | sha512 hash |
---|---|
kubernetes-server-linux-amd64.tar.gz | 09ffc69de339bb507a9f8fdd2206dcc1e77f58184bfa1f771c715edc200861131e5028ae38ec1f5a1112d3303159fb2b9246266114ce0a502776b2c28354dfba |
kubernetes-server-linux-arm64.tar.gz | 56b04497a022b3cd4efac6d1771ead89aef9e6e33639209bb2c1eaa95f4c01cf6ac5f3fa6e66b5edcbd0cab1c164522ad0585daedf271b27b53a8e2d573f6a82 |
kubernetes-server-linux-ppc64le.tar.gz | 75d09f92b6756f1ef96868dd3b83241154729033015544de5e4a881f0ea8bb62bebc326df8c199ab98cb29b7171ff2fce4d4ee15f26d8d68e4545067bbdfa5bb |
kubernetes-server-linux-s390x.tar.gz | 562b42b297a161eded117b5fb0f346c9531e959d4d798e623521703960dccf8841aa261b2678b40d1efc11123af85be1b769ac197a3f89246479486efef85d5b |
filename | sha512 hash |
---|---|
kubernetes-node-linux-amd64.tar.gz | 37b1c6da21d0b915a8dd372caa2c48715dcc9071191f753b2ebdc812643265b646777ecf781c4d269d5490066968648c3321ce0d56b3ac8d3c528c6357de2e67 |
kubernetes-node-linux-arm64.tar.gz | d6708bf5e5c9e70242af57b20bf64396d419fc6654c090741c508d4c265717b0a1d6e8948de5d6927dd356f22c2085607f7b9549bb0f4ee7aafcb3b2f4b862b3 |
kubernetes-node-linux-ppc64le.tar.gz | c26df8571204a0ae5b18a126c21cd8985b6fd0a8df50c8da4cfd86006b3974fa452ff30de0c4f6ed5cd54e59705a2f639a8ee4201fd681048968cbea416e7e40 |
kubernetes-node-linux-s390x.tar.gz | d5a13e1d13a6d9ff081f691b06ca66b8e9bff7cd12591b1281e7c05382aeeee4cd3ec83a23176e07d21c018ca29795b3944cbff7af5f62700046bf2062912959 |
kubernetes-node-windows-amd64.tar.gz | 57f4b842d1637a67ae59e400d237c8d63aea9a7dc018384e3fca9804d457b9125f46bb5776d36f2150642bb70f6f2e8781b4e62e8de84627c076004d1244212a |
All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.
There are no urgent upgrade notes for the v1.32 release.
DisableNodeKubeProxyVersion
feature gate to default-off to give a full year from deprecation announcement in 1.29 to clearing the field by default, per the Kubernetes deprecation policy. (#126720, @liggitt) [SIG Architecture and Node]ACTION REQUIRED for custom scheduler plugin developers: PodEligibleToPreemptOthers
in the preemption
interface now includes ctx
in the parameters. Please update your plugins' implementation accordingly. (#126465, @googs1025) [SIG Scheduling]
Changed NodeToStatusMap from a map to a struct and exposed methods to access the entries. Added absentNodesStatus, which informs the status of nodes that are absent in the map. For developers of out-of-tree PostFilter plugins, ensure to update the usage of NodeToStatusMap. Additionally, NodeToStatusMap should eventually be renamed to NodeToStatusReader. (#126022, @macsko) [SIG Node, Scheduling, and Testing]
A new /resize subresource was added to request pod resource resizing. Update your k8s client code to utilize the /resize subresource for Pod resizing operations. (#128266, @AnishShah) [SIG API Machinery, Apps, Node and Testing]
A new feature that allows unsafe deletion of corrupt resources has been added, it is disabled by default, and it can be enabled by setting the option --feature-gates=AllowUnsafeMalformedObjectDeletion=true
. It comes with an API change, a new delete option ignoreStoreReadErrorWithClusterBreakingPotential
has been introduced, it is not set by default, this maintains backward compatibility. In order to perform an unsafe deletion of a corrupt resource, the user must enable the option for the delete request. A resource is considered corrupt if it can not be successfully retrieved from the storage due to a) transformation error e.g. decryption failure, or b) the object failed to decode. Normal deletion flow is attempted first, and if it fails with a corrupt resource error then it triggers unsafe delete. In addition, when this feature is enabled, the 'details' field of 'Status' from the LIST response includes information that identifies the corrupt object(s). NOTE: unsafe deletion ignores finalizer constraints, and skips precondition checks. WARNING: this may break the workload associated with the resource being unsafe-deleted, if it relies on the normal deletion flow, so cluster breaking consequences apply. (#127513, @tkashem) [SIG API Machinery, Etcd, Node and Testing]
Added singleProcessOOMKill
flag to the kubelet configuration. Setting that to true enable single process OOM killing in cgroups v2. In this mode, if a single process is OOM killed within a container, the remaining processes will not be OOM killed. (#126096, @utam0k) [SIG API Machinery, Node, Testing and Windows]
Added a /flagz
endpoint for kube-apiserver endpoint. (#127581, @richabanker) [SIG API Machinery, Architecture, Auth and Instrumentation]
Added a Stream
field to PodLogOptions
, which allows clients to request certain log stream (stdout or stderr) of the container. Please also note that the combination of a specific Stream
and TailLines
is not supported. (#127360, @knight42) [SIG API Machinery, Apps, Architecture, Node, Release and Testing]
Added alpha support for asynchronous Pod preemption. When the SchedulerAsyncPreemption
feature gate is enabled, the scheduler now runs API calls to trigger preemptions asynchronously for better performance. (#128170, @sanposhiho) [SIG Scheduling and Testing]
Added driver-owned fields in ResourceClaim.Status
to report device status data for each allocated device. (#128240, @LionelJouin) [SIG API Machinery, Network, Node and Testing]
Added enforcement of an upper cost bound for DRA evaluations of CEL. The API server and scheduler now enforce an upper bound on the cost and runtime steps required for evaluating a CEL expression. (#128101, @pohly) [SIG API Machinery and Node]
Added the ability to change the maximum backoff delay accrued between container restarts for a node for containers in CrashLoopBackOff
. To set this for a node, turn on the feature gate KubeletCrashLoopBackoffMax
and set the CrashLoopBackOff.MaxContainerRestartPeriod
field between "1s"
and "300s"
in your kubelet config file. (#128374, @lauralorenz) [SIG API Machinery and Node]
Allow for Pod search domains to be a single dot .
or contain an underscore _
(#127167, @adrianmoisey) [SIG Apps, Network and Testing]
Annotation batch.kubernetes.io/cronjob-scheduled-timestamp
added to Job objects scheduled from CronJobs is promoted to stable. (#128336, @soltysh)
Apply fsGroup policy for ReadWriteOncePod volumes. (#128244, @gnufied) [SIG Storage and Testing]
Changed the Pod API to support resources
at spec
level for pod-level resources. (#128407, @ndixita) [SIG API Machinery, Apps, CLI, Cluster Lifecycle, Node, Release, Scheduling and Testing]
ContainerStatus.AllocatedResources is now guarded by a separate feature gate, InPlacePodVerticalSaclingAllocatedStatus (#128377, @tallclair) [SIG API Machinery, CLI, Node, Scheduling and Testing]
Coordination.v1alpha1 API is dropped and replaced with coordination.v1alpha2. Old coordination.v1alpha1 types must be deleted before upgrade (#127857, @Jefftree) [SIG API Machinery, Etcd, Scheduling and Testing]
DRA: Restricted the length of opaque device configuration parameters. At admission time, Kubernetes enforces a 10KiB size limit. (#128601, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]
DRA: scheduling pods is up to 16x faster, depending on the scenario. Scheduling throughput depends a lot on cluster utilization. It is higher for lightly loaded clusters with free resources and gets lower when the cluster utilization increases. (#127277, @pohly) [SIG API Machinery, Apps, Architecture, Auth, Etcd, Instrumentation, Node, Scheduling and Testing]
DRA: the DeviceRequestAllocationResult
struct now has an "AdminAccess" field which should be used instead of the corresponding field in the DeviceRequest
field when dealing with an allocation. If a device is only allocated for admin access, allocating it again for normal usage is now supported, as originally intended. To allow admin access, starting with 1.32 the DRAAdminAccess
feature gate must be enabled. (#127266, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Network, Node, Scheduling and Testing]
Disallow k8s.io
and kubernetes.io
namespaced extra key in structured authentication configuration. (#126553, @aramase) [SIG Auth]
Fixed a bug in the NestedNumberAsFloat64
Unstructured field accessor that could have caused it to return rounded float64 values instead of errors when accessing very large int64 values. (#128099, @benluddy)
Fixed the bug where spec.terminationGracePeriodSeconds
of the pod will always be overwritten by the MaxPodGracePeriodSeconds of the soft eviction, you can enable the AllowOverwriteTerminationGracePeriodSeconds
feature gate, which will restore the previous behavior. If you do need to set this, please file an issue with the Kubernetes project to help contributors understand why you needed it. (#122890, @HirazawaUi) [SIG API Machinery, Architecture, Node and Testing]
Graduated Job's ManagedBy
field to beta. (#127402, @mimowo) [SIG API Machinery, Apps and Testing]
Implemented a new, alpha seLinuxChangePolicy
field within a Pod-level securityContext
, under SELinuxChangePolicy feature gate. This field allows for opting out from mounting Pod volumes with SELinux label when SELinuxMount feature is enabled (it is alpha and disabled by default now). Please see the KEP how we expect to warn users before any SELinux behavior changes and how they can opt-out before. Note that this field and feature gate is useful only with clusters that run with SELinux enabled. No action is required on clusters without SELinux. (#127981, @jsafrane) [SIG API Machinery, Apps, Architecture, Node, Storage and Testing]
Introduced v1alpha1
API for mutating admission policies, enabling extensible # admission control via CEL expressions (KEP 3962: Mutating Admission Policies). # To use, enable the MutatingAdmissionPolicy
feature gate and the admissionregistration.k8s.io/v1alpha1
# API via --runtime-config
. (#127134, @jpbetz) [SIG API Machinery, Auth, Etcd and Testing]
Introduced compressible resource setting on system reserved and kube reserved slices. (#125982, @harche)
kube-apiserver: Promoted the StructuredAuthorizationConfiguration
feature gate to GA. The --authorization-config
flag now accepts AuthorizationConfiguration
in version apiserver.config.k8s.io/v1
(with no changes from apiserver.config.k8s.io/v1beta1
). (#128172, @liggitt) [SIG API Machinery, Auth and Testing]
kube-proxy now reconciles Service/Endpoint changes with conntrack table and cleans up only stale UDP flow entries (#127318, @aroradaman) [SIG Network and Windows]
kube-scheduler removed AzureDiskLimits
,CinderLimits
EBSLimits
and GCEPDLimits
plugin. Given the corresponding CSI driver reports how many volumes a node can handle in NodeGetInfoResponse, the kubelet stores this limit in CSINode and the scheduler then knows the limit of the driver on the node. Removed plugins AzureDiskLimits, CinderLimits, EBSLimits and GCEPDLimits if you explicitly enabled them in the scheduler config. (#124003, @carlory) [SIG Scheduling, Storage and Testing]
kubelet: the --image-credential-provider-config
file was loaded with strict deserialization, which failed if the config file contained duplicate or unknown fields. This protected against accidentally running with malformed config files, unindented files, or typos in field names, and it prevented unexpected behavior. (#128062, @aramase) [SIG Auth and Node]
NodeRestriction admission now validates the audience value that kubelet is requesting a service account token for is part of the pod spec volume. This change is introduced with a new kube-apiserver featuregate ServiceAccountNodeAudienceRestriction
that's enabled by default. (#128077, @aramase) [SIG Auth, Storage and Testing]
Promoted CustomResourceFieldSelectors
to stable; the feature was enabled by default. The --feature-gates=CustomResourceFieldSelectors=true
flag was no longer needed on kube-apiserver binaries and would be removed in a future release. (#127673, @jpbetz) [SIG API Machinery and Testing]
Promoted feature gate StatefulSetAutoDeletePVC
from beta to stable. (#128247, @mattcary) [SIG API Machinery, Apps, Auth and Testing]
Removed all support for classic dynamic resource allocation (DRA). The DRAControlPlaneController
feature gate, formerly alpha, is no longer available. Kubernetes now only uses the structured parameters model (also alpha) for allocating dynamic resources to Pods.
if and only if classic DRA was enabled in a cluster, remove all workloads (pods, app deployments, etc. ) which depend on classic DRA and make sure that all PodSchedulingContext resources are gone before upgrading. PodSchedulingContext resources cannot be removed through the apiserver after an upgrade and workloads would not work properly. (#128003, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]
Removed generally available feature gate HPAContainerMetrics
(#126862, @carlory) [SIG API Machinery, Apps and Autoscaling]
Removed restrictions on subresource flag in kubectl commands (#128296, @AnishShah) [SIG CLI]
Revised the kubelet API Authorization with new subresources, that allow finer-grained authorization checks and access control for kubelet endpoints. Provided you enable the KubeletFineGrainedAuthz
feature gate, you can access kubelet's /healthz
endpoint by granting the caller nodes/helathz
permission in RBAC. Similarly you can also access kubelet's /pods
endpoint to fetch a list of Pods bound to that node by granting the caller nodes/pods
permission in RBAC. Similarly you can also access kubelet's /configz
endpoint to fetch kubelet's configuration by granting the caller nodes/configz
permission in RBAC. You can still access kubelet's /healthz
, /pods
and /configz
by granting the caller nodes/proxy
permission in RBAC but that also grants the caller permissions to exec, run and attach to containers on the nodes and doing so does not follow the least privilege principle. Granting callers more permissions than they need can give attackers an opportunity to escalate privileges. (#126347, @vinayakankugoyal) [SIG API Machinery, Auth, Cluster Lifecycle and Node]
The core functionality of Dynamic Resource Allocation (DRA) got promoted to beta. No action is required when upgrading, the previous v1alpha3 API is still supported, so existing deployments and DRA drivers based on v1alpha3 continue to work. Downgrading from 1.32 to 1.31 with DRA resources in the cluster (resourceclaims, resourceclaimtemplates, deviceclasses, resourceslices) is not supported because the new v1beta1 is used as storage version and not readable by 1.31. (#127511, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]
The default value for node-monitor-grace-period has been increased to 50s (earlier 40s) (Ref - https://github.com/kubernetes/kubernetes/issues/121793) (#126287, @devppratik) [SIG API Machinery, Apps and Node]
The resource/v1alpha3.ResourceSliceList filed which should have been named "metadata" but was instead named "listMeta" is now properly "metadata". (#126749, @thockin) [SIG API Machinery]
The synthetic "Bookmark" event for the watch stream requests will now include a new annotation: kubernetes.io/initial-events-list-blueprint
. THe annotation contains an empty, versioned list that is encoded in the requested format (such as protobuf, JSON, or CBOR), then base64-encoded and stored as a string. (#127587, @p0lyn0mial) [SIG API Machinery]
To enhance usability and developer experience, CRD validation rules now support direct use of (CEL) reserved keywords as field names in object validation expressions. Name format CEL library is supported in new expressions. (#126977, @aaron-prindle) [SIG API Machinery, Architecture, Auth, Etcd, Instrumentation, Release, Scheduling and Testing]
Updated incorrect description of persistentVolumeClaimRetentionPolicy (#126545, @yangjunmyfm192085) [SIG API Machinery, Apps and CLI]
X.509 client certificate authentication to the kube-apiserver now produces credential IDs (derived from the certificate's signature) , for use in audit logging. (#125634, @ahmedtd) [SIG API Machinery, Auth and Testing]
Added Windows support for the node memory manager. (#128560, @marosset) [SIG Node and Windows]
Added --concurrent-daemonset-syncs
command line flag to kube-controller-manager. This value sets the number of workers for the daemonset controller. (#128444, @tosi3k)
Added a /statusz
endpoint for the kube-apiserver endpoint. (#125577, @richabanker) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Instrumentation, Network, Node and Testing]
Added a health check for the device plugin gRPC registration server. When the registration server is down, kubelet is marked as unhealthy. If systemd watchdog is configured, this will result in a kubelet restart. (#128432, @zhifei92) [SIG Node]
Added a kubelet metric container_aligned_compute_resources_count
to report the count of containers getting aligned compute resources. (#127155, @ffromani) [SIG Node and Testing]
Added a kubelet metrics to report informations about the cpu pools managed by cpumanager when the static policy is in use. (#127506, @ffromani) [SIG Node and Testing]
Added a new controller, volumeattributesclass-protection-controller, into the kube-controller-manager. The new controller manages a protective finalizer on VolumeAttributesClass objects. (#123549, @carlory) [SIG API Machinery, Apps, Auth and Storage]
Added a new option strict-cpu-reservation
for CPU Manager static policy. When this option is enabled, CPU cores in reservedSystemCPUs
will be strictly used for system daemons and interrupt processing no longer available for any workload. (#127483, @jingczhang) [SIG Node]
Added a one-time random duration of up to 50% of kubelet's nodeStatusReportFrequency
to help spread the node status update load evenly over time. (#128640, @mengqiy)
Added an option to enable leader election in local-up-cluster.sh via the LEADER_ELECT CLI flag. (#127786, @Jefftree)
Added kubelet support for systemd watchdog integration. With this enabled, systemd can automatically recover a hung kubelet. (#127566, @zhifei92) [SIG Cloud Provider, Node and Testing]
Added metrics to measure the latency of DRA Node operations and DRA GRPC calls (#127146, @bart0sh) [SIG Instrumentation, Network, Node, and Testing]
Added new functionality to the Go client code (client-go
) library. The List()
method for the metadata client allows enabling API streaming when fetching collections; this improves performance when listing many objects. To request this behavior, your client software must enable the WatchListClient
client-go feature gate. Additionally, streaming is only available if supported by the cluster; the API server that you connect to must also support streaming. If the API server does not support or allow streaming, then client-go
falls back to fetching the collection using the list API verb. (#127388, @p0lyn0mial) [SIG API Machinery and Testing]
Added preemptionPolicy field when using kubectl get PriorityClass -owide
(#126529, @googs1025) [SIG CLI]
Added status for extended Pod resources within the status.containerStatuses[].resources
field. (#124227, @iholder101) [SIG Node and Testing]
Added support to the kube-apiserver for an alpha feature enabling external signing of service account tokens and fetching of public verifying keys, by enabling the Alpha ExternalServiceAccountTokenSigner
feature gate and specifying --service-account-signing-endpoint
. The flag value can either be the location of a Unix domain socket on a filesystem, or be prefixed with an @ symbol and name a Unix domain socket in the abstract socket namespace. (#128190, @HarshalNeelkamal) [SIG API Machinery, Apps, Auth, Etcd, Instrumentation, Node, Release and Testing]
Added the feature gate CBORServingAndStorage to allow CBOR as the encoding for API request and response bodies, and as the storage encoding for custom resources. Clients must opt in; programs built with client-go can do this using the client-go feature gates ClientsAllowCBOR and ClientsPreferCBOR. (#128539, @benluddy) [SIG API Machinery, Etcd and Testing]
Adopted a new implementation of watch caches for list verbs, using a btree data structure. The new implementation is active by default; you can opt out by disabling the BtreeWatchCache
feature gate. (#128415, @serathius) [SIG API Machinery, Auth and Cloud Provider]
Allows PreStop lifecycle handler's sleep action to have a zero value (#127094, @sreeram-venkitesh) [SIG Apps, Node and Testing]
CRI: Added a field to support CPU affinity on Windows. (#124285, @kiashok) [SIG Node and Windows]
Changed OOM score adjustment calculation for sidecar containers: the OOM adjustment for these containers will match or fall below the OOM score adjustment of regular containers in the Pod. (#128029, @bouaouda-achraf)
Client-go/rest: contextual logging of request/response with accurate source code location of the caller (#126999, @pohly) [SIG API Machinery and Instrumentation]
DRA: The resource claim controller now maintains metrics about the total number of ResourceClaims
and the number of allocated ResourceClaims
. (#127661, @pohly) [SIG Apps, Instrumentation and Node]
Enabled graceful shutdown feature for Windows node (#127404, @zylxjtu) [SIG Node, Testing and Windows]
Enabled kube-controller-manager '--concurrent-job-syncs' flag works on orphan Pod processors (#126567, @fusida) [SIG Apps]
Ensured resizing for Guaranteed pods with integer CPU requests on nodes with static CPU & Memory policy configured is not allowed for the beta release of in-place resize. The feature gate InPlacePodVerticalScalingExclusiveCPUs
defaults to false
, but can be enabled to unblock development on (#127262, @tallclair) [SIG Node]. (#128287, @esotsal) [SIG Node, Release and Testing]
Extend discovery GroupManager with Group lister interface (#127524, @mjudeikis) [SIG API Machinery]
Fixed: Avoid overwriting in-pod vertical scaling updates on systemd daemon reloads when using systemd (#124216, @iholder101) [SIG Node]
Fixed an issue where kubectl doesn't print image volume when kubectl describe a pod with that volume. (#126706, @carlory)
Graduated the AnonymousAuthConfigurableEndpoints feature gate to beta and enable by default to allow configurable endpoints for anonymous authentication. (#127009, @vinayakankugoyal) [SIG Auth]
Graduated the kubelet memory manager to generally available (GA). (#128517, @Tal-or)
Graduated SchedulerQueueingHints
to beta; the feature gate is now enabled by default. (#128472, @sanposhiho) [SIG Scheduling]
Graduated the WatchList
feature gate to Beta for kube-apiserver and enabled WatchListClient
for KCM. (#128053, @p0lyn0mial) [SIG API Machinery and Testing]
Implemented a queueing hint for PersistentVolumeClaim/Add event in the CSILimit
plugin. (#124703, @utam0k) [SIG Scheduling and Storage]
Implemented new cluster events UpdatePodSchedulingGatesEliminated
and UpdatePodTolerations
for scheduler plugins. (#127083, @sanposhiho)
Improved Node's QueueingHint in the NodeAffinity
plugin by ignoring unrelated changes that keep pods unschedulable. (#127444, @dom4ha) [SIG Scheduling and Testing]
Improved Node's QueueingHint in the NodeResourceFit
plugin by ignoring unrelated changes that keep pods unschedulable. (#127473, @dom4ha) [SIG Scheduling and Testing]
Improved performance of the job controller when handling job delete events. (#127378, @hakuna-matatah)
Improved performance of the job controller when handling job update events. (#127228, @hakuna-matatah)
Included an additional resource labeltransformation in on_operations_total metric which could be used for resource specific validations for example handling of encryption config by the apiserver. (#126512, @kmala) [SIG API Machinery, Auth, Etcd and Testing]
Introduced a new metric kubelet_admission_rejections_total
to track the number of pods rejected during admission. (#128556, @AnishShah)
JWT authenticators now set the jti
claim (if present and is a string value) as credential id for use by audit logging. (#127010, @aramase) [SIG API Machinery, Auth and Testing]
kube-apiserver: Promoted AuthorizeWithSelectors
feature to beta, which includes field and label selector information from requests in webhook authorization calls. Promoted AuthorizeNodeWithSelectors
feature to beta, which changes node authorizer behavior to limit requests from node API clients, so that each Node can only get / list / watch its own Node API object, and can also only get / list / watch Pod API objects bound to that node. Clients using kubelet credentials to read other nodes or unrelated pods must change their authentication credentials (recommended), adjust their usage, or obtain broader read access independent of the node authorizer. (#128168, @liggitt) [SIG API Machinery, Auth and Testing]
kube-apiserver: a new --requestheader-uid-headers
flag allows configuring request header authentication to obtain the authenticating user's UID from the specified headers. The suggested value for the new option is X-Remote-Uid
. When specified, the kube-system/extension-apiserver-authentication
configmap will include the value in its .data[requestheader-uid-headers]
field. (#115834, @stlaz) [SIG API Machinery, Auth, Cloud Provider and Testing]
kube-proxy uses field-selector clusterIP!=None on Services to avoid watching for Headless Services, reducing unnecessary network bandwidth (#126769, @Sakuralbj) [SIG Network]
: kubeadm upgrade apply
now supports phase sub-command, users can use kubeadm upgrade apply phase <phase-name>
to execute the specified phase, or use kubeadm upgrade apply --skip-phases <phase-names>
to skip some phases during cluster upgrade. (#126032, @SataQiu) [SIG Cluster Lifecycle]
kubeadm: kubeadm upgrade node
now supports addon
and post-upgrade
phases. Users can use kubeadm upgrade node phase addon
to execute the addon upgrade, or use kubeadm upgrade node --skip-phases addon
to skip the addon upgrade. Currently, the post-upgrade
phase is no-op, and it is mainly used to handle some release-specific post-upgrade tasks. (#127242, @SataQiu) [SIG Cluster Lifecycle]
kubeadm: added a validation warning when the certificateValidityPeriod is more than the caCertificateValidityPeriod (#126538, @SataQiu) [SIG Cluster Lifecycle]
kubeadm: added the feature gate NodeLocalCRISocket
. When the feature gate is enabled, kubeadm will generate the /var/lib/kubelet/instance-config.yaml
file to customize the containerRuntimeEndpoint
field in the kubelet configuration for each node and will not write the same CRI socket on the Node object as an annotation. (#128031, @HirazawaUi) [SIG Cluster Lifecycle]
kubeadm: allow mixing the flag --config with the special flag --print-manifest of the subphases of 'kubeadm init phase addon'. (#126740, @neolit123) [SIG Cluster Lifecycle]
kubeadm: consider --bind-address or --advertise-address and --secure-port for control plane components when the feature gate WaitForAllControlPlaneComponents is enabled. Use /livez for kube-apiserver and kube-scheduler, but continue using /healthz for kube-controller-manager until it supports /livez. (#128474, @neolit123) [SIG Cluster Lifecycle]
kubeadm: if an unknown command name is passed to any parent command such as 'kubeadm init phase' return an error. If 'kubeadm init phase' or another command that has subcommands is called without subcommand name, print the available commands and also return an error. (#127096, @neolit123) [SIG Cluster Lifecycle]
kubeadm: promoted feature gate EtcdLearnerMode
to GA. Learner mode in etcd deployed by kubeadm is now locked to enabled by default. (#126374, @pacoxu) [SIG Cluster Lifecycle]
kubelet: add log and event for cgroup v2 with kernel older than 5.8. (#126595, @pacoxu) [SIG Node]
Kubernetes is now built with Go 1.23.3. (#128852, @cpanato) [SIG Release and Testing]
Kubernetes is now built with go 1.23.0 (#127076, @cpanato) [SIG Release and Testing]
Kubernetes was built with Go 1.23.1. (#127611, @haitch) [SIG Release and Testing]
Kubernetes was built with Go 1.23.2. (#128110, @haitch) [SIG Release and Testing]
Label apps.kubernetes.io/pod-index
added to Pod from StatefulSets is promoted to stable Label batch.kubernetes.io/job-completion-index
added to Pods from Indexed Jobs is promoted to stable (#128387, @alaypatel07) [SIG Apps]
LoadBalancerIPMode feature was marked as GA. (#127348, @RyanAoh) [SIG Apps, Network and Testing]
Locked the custom profiling feature in kubectl debug
to true. (#127187, @ardaguclu) [SIG CLI and Testing]
Output for the ScalingReplicaSet
event has changed from: Scaled <up|down> replica set to from to: Scaled <up|down> replica set from to . (#125118, @jsoref) [SIG Apps and CLI]
PodLifecycleSleepAction is graduated to GA (#128046, @AxeZhan) [SIG Architecture, Node and Testing]
Pods were allowed to use the net.ipv4.tcp_rmem
and net.ipv4.tcp_wmem
sysctl by default when the kernel version was 4.15 or higher. With the kernel 4.15 the sysctl became namespaced. Pod Security admission allowed these sysctl in v1.32+ versions of the baseline and restricted policies. (#127489, @pacoxu) [SIG Auth, Network and Node]
Prepared Pod validation to handle version skew for InPlacePodVerticalScaling's beta graduation. (#128186, @sreeram-venkitesh)
Promoted RecoverVolumeExpansionFailure
feature gate to beta. (#128342, @gnufied) [SIG Apps and Storage]
Promoted RetryGenerateName
to stable; the feature is enabled by default. --feature-gates=RetryGenerateName=true
not needed on kube-apiserver binaries and will be removed in a future release. (#127093, @jpbetz) [SIG API Machinery]
Promoted SizeMemoryBackedVolumes
to stable. (#126981, @kannon92) [SIG Node, Storage and Testing]
Promoted the RelaxedEnvironmentVariableValidation
feature gate to beta and is enabled by default. (#126897, @HirazawaUi)
Promoted the feature gates StrictCostEnforcementForVAP
and StrictCostEnforcementForWebhooks
. (#127302, @cici37) [SIG API Machinery and Testing]
Promoted the ServiceAccountTokenJTI
feature to GA, which adds a jti
claim to issued service account tokens and embeds the jti
claim as a authentication.kubernetes.io/credential-id=["JTI=..."]
value in user extra info
ServiceAccountTokenPodNodeInfo
feature to GA, which adds the node name and uid as claims into service account tokens mounted into running pods, and embeds that information as authentication.kubernetes.io/node-name
and authentication.kubernetes.io/node-uid
user extra info when the token is usedServiceAccountTokenNodeBindingValidation
feature to GA, which validates service account tokens bound directly to nodes. (#128169, @liggitt) [SIG API Machinery, Auth and Testing]Realigned line breaks from kubectl explain
descriptions. (#126533, @ah8ad3)
Removed attachable volume limits from the capacity of the node for the following volume type when the kubelet was started, affecting the following volume types when the corresponding csi driver was installed:
awsElasticBlockStore
for ebs.csi.aws.com
azureDisk
for disk.csi.azure.com
gcePersistentDisk
for pd.csi.storage.googleapis.com
cinder
for cinder.csi.openstack.org
csi
However it was still enforced using a limit in CSINode objects. (#126924, @carlory)Reverted Go version used to build Kubernetes to 1.23.0. (#127861, @xmudrii) [SIG Release and Testing]
Support inflight_events metric in the scheduler for QueueingHint. (#127052, @sanposhiho) [SIG Scheduling]
Support specifying a custom network parameter when running e2e-node-tests with the remote option. (#127574, @bouaouda-achraf) [SIG Node and Testing]
The Job controller now considers sidecar container restart counts when removing pods. (#124952, @AxeZhan) [SIG Apps and CLI]
The TopologyManagerPolicyOptions
feature-flag is promoted to GA. (#128124, @PiotrProkop)
The scheduler implemented QueueingHint
in VolumeBinding plugin's CSIDriver event, which enhanced the throughput of scheduling. (#125171, @YamasouA) [SIG Scheduling and Storage]
The scheduler retries gated Pods more appropriately, giving them a backoff penalty too. (#126029, @sanposhiho) [SIG Scheduling]
Unallowed label values will show up as "unexpected" in scheduler metrics. (#126762, @richabanker) [SIG Instrumentation and Scheduling]
Updated the control plane's trust anchor publisher to create and manage a new ClusterTrustBundle object, associated with the kubernetes.io/kube-apiserver-serving
X.509 certificate signer. This ClusterTrustBundle contains a PEM bundle in its payload that you can use to verify kube-apiserver serving certificates. (#127326, @stlaz) [SIG API Machinery, Apps, Auth, Cluster Lifecycle and Testing]
Vendor: updated system-validators to v1.9.0. (#128149, @neolit123) [SIG Cluster Lifecycle and Node]
Vendor: updated system-validators to v1.9.1. (#128533, @neolit123)
When SchedulerQueueingHint
is enabled, the scheduler's in-tree plugins now subscribe to specific node events to decide whether to requeue Pods. This allows the scheduler to handle cluster events faster with less memory.
Specific node events include updates to taints, tolerations or allocatable. In-tree plugins now ignore node updates that don't modify any of these fields. (#127220, @sanposhiho) [SIG Node, Scheduling and Storage]
When SchedulerQueueingHints
is enabled, clear events cached in the scheduling queue as soon as possible so that the scheduler consumes less memory. (#120586, @sanposhiho) [SIG Scheduling]
Windows: Support CPU and Topology manager on Windows. (#125296, @jsturtevant) [SIG Node and Windows]
--allocate-node-cidrs
, --cluster-cidr
, and --service-cluster-ip-range
flags to accurately reflect their dependencies and usage conditions. (#126784, @eminwux) [SIG API Machinery, Cloud Provider and Docs]--for=create
option to kubectl wait
. (#127327, @ryanwinter) [SIG CLI]apiserver_admission_webhook_fail_open_count
and apiserver_admission_webhook_request_total
metrics. The type
label can have a value of "admit", not "mutating". (#127898, @modulitos)--cloud-provider=external
can use the --node-ip
flag with one of the unspecified addresses 0.0.0.0 or ::, to create the Node with the IP of the default gateway of the corresponding IP family and then delegating the responsibility to the external cloud provider. This solves the bootstrap problems of out of tree cloud providers that are deployed as Pods within the cluster. (#125337, @aojea) [SIG Cloud Provider, Network, Node and Testing]RemoteRequestHeaderUID
feature gate. (#129081, @stalz) [SIG API SIG API Machinery, cluster lifecycle, testing]image
volume source type, it passes the missing mount attributes to the CRI implementation, including readOnly
, propagation
, and recursiveReadOnly
. When the readOnly field of the containerMount is explicitly set to false, the kubelet will now take the readOnly
as true to the CRI implementation because the image volume plugin requires the mount to be read-only.image
volume source type is used and mounted to /etc/hosts
in the container. (#126806, @carlory) [SIG Node and Storage]allocationMode: all
. (#127565, @pohly)blockOwnerDeletion: true
will not be known to the garbage collector. Use of blockOwnerDeletion
has always been best-effort and racy on startup and object creation. With this fix, it continues to be best-effort for resources that cannot be synced by the garbage collector controller. (#125796, @haorenfsa) [SIG API Machinery, Apps and Testing]nodeAffinity
using the hostname may be scheduled to the wrong node or experience scheduling failures. (#125398, @AxeZhan) [SIG Scheduling and Storage]podCIDR
was released before node was deleted. (#128305, @adrianmoisey) [SIG Apps and Network]failed to initialize top level QOS containers: root container [kubepods] doesn't exist
, due to the cpuset cgroup being deleted on cgroup v2 with systemd cgroup manager. (#125923, @haircommander) [SIG Node and Testing]SidecarContainers
feature enabled, where init containers may fail to start due to a temporary container runtime failure. (#126543, @gjkim42)resolvConf
option in drop-in kubelet configuration files, which validates that drop-in kubelet configuration files are in a supported version. (#127421, @liggitt)NodeUnschedulable
that only happens with QHint enabled, which the scheduler might miss some updates for the Pods rejected by NodeUnschedulable plugin and put the Pods in the queue for a longer time than needed. (#127427, @sanposhiho)ValidatingAdmissionPolicy
decisions and annotations. The apiserver_validating_admission_policy_check_duration metrics will now show elapsed times and no longer be zero. (#128463, @knrc)SerializeObject
and List
). In the past, some children's spans appeared parallel to their parents. (#127551, @carlory) [SIG API Machinery and Instrumentation]application/json;as=Table
content type, the API server now responds with a 406 (Not Acceptable) error. This change helps to ensure that unsupported formats, such as Table
representations are correctly rejected. (#126996, @p0lyn0mial) [SIG API Machinery and Testing]CreateJob
are properly terminated after a timeout. (#127333, @yuyabee) [SIG Cluster Lifecycle]kubeamd join
, ensure that the etcd member addition is performed only if a given member URL does not already exist in the list of members. Similarly, on "kubeadm reset" only remove an etcd member if its ID exists. (#127491, @SataQiu) [SIG Cluster Lifecycle]PodAndContainerStatsFromCRI
feature is enabled (#126488, @haircommander) [SIG Node]ResultChan
and close the RetryWatcher
when the client is forbidden or unauthorized from watching the resource. (#126038, @mprahl) [SIG API Machinery]build-tag
flag is reintroduced to conversion-gen and defaulter-gen which allow users to inject custom build tag during code generation process. (#128259, @dinhxuanvu)StartupProbe
was explicitly stopped when the successThreshold
was reached. This eliminated the problem of executing StartupProbe
more times than the successThreshold
. (#121206, @mochizuki875)kubectl explain
. You could now use -o
as an abbreviation for --output
in commands such as kubectl explain <resource> --output plaintext-openapiv2
. (#127869, @ak20102763)./api/discovery
. Please use v2 (#127008, @Jefftree) [SIG API Machinery]status.containerStatuses[*].state.waiting.message
when in image pull back-off (reason
is ImagePullBackOff
) instead of the generic Back-off pulling image…
message. (#127918, @saschagrunert) [SIG Node and Testing]ImageFsInfo
RPC. (#128052, @saschagrunert)operator
is Exists
and value
is not empty. (#128119, @saschagrunert) [SIG API Machinery and Apps]InPlacePodVerticalScaling
feature in Windows. (#128623, @AnishShah) [SIG Apps and Node]CBORServingAndStorage
feature gate – built-in APIs can now be served in CBOR format for clients that request it. (#128503, @benluddy) [SIG API Machinery, Etcd and Testing]AllowServiceLBStatusOnNonLB
remains deprecated and is now locked to false to support compatibility versions. (#128139, @Jefftree)--validate flag
description in kubectl. (#128081, @soltysh)k8s.io/cloud-provider/service
controller, it may panic when a service is updated because the event recorder was used before it was initialized. All cloud providers should using the v1.31.0
cloud provider service controller must ensure that the controllers is initialized before the informer start to process events or update it to the version 1.32.0. (#128179, @carlory) [SIG API Machinery, Cloud Provider, Network and Testing]PostStartHookContext.StopCh
. (#127341, @mjudeikis)--admission-control-config-file
files are now validated strictly (EnableStrict). Duplicate and unknown fields in the configuration will now cause an error. (#128013, @seans3)--egress-selector-config-file
files were validated strictly (EnableStrict). Duplicate and unknown fields in the configuration will now cause an error. (#128011, @seans3) [SIG API Machinery and Testing]ResourceQuotaConfiguration
admission plugin subsection within --admission-control-config-file
files were validated strictly (EnableStrict). Duplicate and unknown fields in the configuration would cause an error. (#128038, @seans3)--leader-migration-config
files were now validated strictly (EnableStrict). Duplicate and unknown fields in the configuration would cause an error. (#128009, @seans3) [SIG API Machinery and Cloud Provider]crictl
binary does not exist since kubeadm does not rely on crictl
since v1.31. (#126596, @saschagrunert) [SIG Cluster Lifecycle]experimental-cert-rotation
, and use 'enable-client-cert-rotation' instead. (#126913, @pacoxu) [SIG Cluster Lifecycle]socat
and ebtables
from kubeadm preflight checks (#127151, @saschagrunert) [SIG Cluster Lifecycle]--feature-gates
for kubeadm upgrade apply
and --api-server-manifest
, --controller-manager-manifest
, and --scheduler-manifest
for kubeadm upgrade diff
. (#127123, @neolit123) [SIG Cluster Lifecycle]--experimental-output
, please use the flag --output
instead that serves the same purpose. Affected commands are: kubeadm config images list
, kubeadm token list
, kubeadm upgrade plan
, kubeadm certs check-expiration
. (#126914, @carlory) [SIG Cluster Lifecycle]/livez
(for startup and liveness probes) and /readyz
(for the readiness probe). Previously, /healthz
was used for all probes, which is deprecated behavior in the scope of this component. (#126945, @liangyuanpeng) [SIG Cluster Lifecycle]getPodAndContainerForDevice
method. (#126997, @lengrongfu)ValidatingAdmissionPolicy
. (#126645, @cici37) [SIG API Machinery, Auth, and Testing]CloudDualStackNodeIPs
. (#126840, @carlory) [SIG API Machinery and Cloud Provider]LegacyServiceAccountTokenCleanUp
. (#126839, @carlory) [SIG Auth]MinDomainsInPodTopologySpread
. (#126863, @carlory) [SIG Scheduling]NewVolumeManagerReconstruction
. (#126775, @carlory) [SIG Node and Storage]NodeOutOfServiceVolumeDetach
(#127019, @carlory) [SIG Apps and Testing]StableLoadBalancerNodeSet
. (#126841, @carlory) [SIG API Machinery, Cloud Provider and Network]ZeroLimitedNominalConcurrencyShares
(#126894, @carlory) [SIG API Machinery]--cloud-provider
command line argument to "external", or to the empty string. All other values are invalid. (#128197, @aojea) [SIG API Machinery, Apps and Cloud Provider]--runonce
mode. If you specify the kubelet command line flag --runonce
, this is an error. Setting runOnce
in a kubelet configuration file is also an error, and specifying any value for that configuration option is now deprecated. (#126336, @HirazawaUi) [SIG Node and Scalability]ServerSideApply
and ServerSideFieldValidation
. (#127058, @carlory)KMSv2
and KMSv2KDF
feature gates. The associated features graduated to stable in the Kubernetes v1.29 release. (#126698, @enj) [SIG API Machinery, Auth and Testing]legacy
profile is planned to be deprecated. (#127230, @mochizuki875) [SIG CLI]dynamicResources
has been refactored to DynamicResources
, now users can introduce the DynamicResources
struct outside the dynamicresources
package. (#128399, @JesseStutler) [SIG Node and Scheduling]flowcontrol.apiserver.k8s.io/v1beta3
API version of FlowSchema
and PriorityLevelConfiguration
is no longer served in v1.32. Migrate manifests and API clients to use the flowcontrol.apiserver.k8s.io/v1
API version, available since v1.29. More information is at https://kubernetes.io/docs/reference/using-api/deprecation-guide/#flowcontrol-resources-v132 (#127017, @carlory) [SIG API Machinery and Testing]--tracing-config-file
is now validated strictly (EnableStrict). Duplicate and unknown fields in the configuration will now result in an error. (#128073, @seans3)--healthz-port
and --metrics-port
, which were previously deprecated, have now been removed. (#126889, @aroradaman) [SIG Network and Windows]--healthz-port
and --metrics-port
, which were previously deprecated, have now been removed. (#127930, @aroradaman) [SIG Network and Windows]kubectl top node
from %
to (%)
. (#126995, @googs1025) [SIG CLI]kubectl kustomize
as described at https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv5.4.2 and https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv5.5.0. (#127965, @koba1t)ComponentSLIs
feature is marked as GA and locked. (#128317, @Jefftree) [SIG Architecture and Instrumentation]kubectl apply --server-side
now supports --subresource
congruent to kubectl patch
. (#127634, @deads2k) [SIG CLI and Testing]