Kubernetes v1.32.0 is live!

141 views
Skip to first unread message

Jim Angel

unread,
Dec 11, 2024, 4:51:28 PM12/11/24
to dev, kubernete...@googlegroups.com
Kubernetes Community,

Kubernetes v1.32.0 has been built and pushed using Golang version 1.23.3.

The release notes have been updated in CHANGELOG-1.32.md, with a pointer to them on GitHub:

v1.32.0

Documentation

Downloads for v1.32.0

Source Code

filenamesha512 hash
kubernetes.tar.gz6ff36174fd78b83b7cf2a05ff991725efcd3529f2c8c9924586258d359af5049062c1f4aff6d8e9044981781c80de6cc738365b85e47fd2e2971cd53a36882c2
kubernetes-src.tar.gz3c401843abef2e74c2e20557f1a7165623dc98c1e290cd629035ac323a491125c666966c638e8baf9f1cb039f330e1b80a4795551145dc04c323c487c25ced22

Client Binaries

filenamesha512 hash
kubernetes-client-darwin-amd64.tar.gzadab0d3f2947323dc8690aebe8bf9aca0179a460ee43dd4144677d293d9d75cfe8c363d1f377d03533758aef891bba3fe4c884ec16e94b84dad83c5de1570a98
kubernetes-client-darwin-arm64.tar.gz155376003480f5689a503bd3857606813882bc45bdf7d3b07a002d282cbb74fc585844ccffd00ca5f49ed3e65721c9f63d25d67a19f09a9f3257416017e83e83
kubernetes-client-linux-386.tar.gz96716dcadf056057f9e9e7cda99935a95381333b8dfa101c3c168903c7dbdef2994d59585e8ee2d362c552f04038c3a0b47077ab7506a2a98ccbd1c1d91f183c
kubernetes-client-linux-amd64.tar.gz302e02599f0bdd3665aadb9e16a2f1f50712bf875f7525a0184450c0dcd59cefbfa67c3211aaa4d4eca197bd9fb49e1de35ffb9d579527ed4830d04400b09ef7
kubernetes-client-linux-arm.tar.gzb104c1fcea77ee2c614ee9089e94accc8aa5f915315711a974a51f0f5e0899e4741dcd6a046fea69264cb6933ce5c84ccaa9f7c9c1849def7da098ad5d2cc845
kubernetes-client-linux-arm64.tar.gz378face3b06a2d062aa734ba0b9fd13f20f877bd611556c352be6246fa70067e60ee44fe55c4c0f064b5715b311075b4db540c7cc52d1a2af4b96a563625f4f1
kubernetes-client-linux-ppc64le.tar.gz6172956799cdf4a65fa5450f26ed4e491a935473418daacb51686d93745445747b893eea701af9d8c508ac8cbd3f4cbabef6cb17b94448e5c2732dc13d35e046
kubernetes-client-linux-s390x.tar.gza3ebc9175317aa93acc26edee8b7f5502a0d9405c1b04b39d907bbebe022e23c9fed058f5cede7045e388d1706c2657b0282d14862e01a7b34002e88e7224d8f
kubernetes-client-windows-386.tar.gzaffcae9e4065cdfc130c6bb690539a631c1be0d992e9b02efbd49e0d519275d23e77c2ba5aa563ed5b89498e8bb26ce73019575bd557152b0d554578a96bf945
kubernetes-client-windows-amd64.tar.gz9619c05daa723c7853daae3432f771a31a6fe57887c32e5e592eb3bf619a636a8c3c2dec0eba2ec60382dc3d2ea8b0dc58e5b5f15fa43cdb7371a3ec0a7e4f55
kubernetes-client-windows-arm64.tar.gze4c8c0d70d5c825dccd3dfa4517baf07f863deac440560c176206b626b0ebd585f0c0601e8956e4a73eda41d31d963895fe337a700a9c1853b7ee3ff2bd568e3

Server Binaries

filenamesha512 hash
kubernetes-server-linux-amd64.tar.gz09ffc69de339bb507a9f8fdd2206dcc1e77f58184bfa1f771c715edc200861131e5028ae38ec1f5a1112d3303159fb2b9246266114ce0a502776b2c28354dfba
kubernetes-server-linux-arm64.tar.gz56b04497a022b3cd4efac6d1771ead89aef9e6e33639209bb2c1eaa95f4c01cf6ac5f3fa6e66b5edcbd0cab1c164522ad0585daedf271b27b53a8e2d573f6a82
kubernetes-server-linux-ppc64le.tar.gz75d09f92b6756f1ef96868dd3b83241154729033015544de5e4a881f0ea8bb62bebc326df8c199ab98cb29b7171ff2fce4d4ee15f26d8d68e4545067bbdfa5bb
kubernetes-server-linux-s390x.tar.gz562b42b297a161eded117b5fb0f346c9531e959d4d798e623521703960dccf8841aa261b2678b40d1efc11123af85be1b769ac197a3f89246479486efef85d5b

Node Binaries

filenamesha512 hash
kubernetes-node-linux-amd64.tar.gz37b1c6da21d0b915a8dd372caa2c48715dcc9071191f753b2ebdc812643265b646777ecf781c4d269d5490066968648c3321ce0d56b3ac8d3c528c6357de2e67
kubernetes-node-linux-arm64.tar.gzd6708bf5e5c9e70242af57b20bf64396d419fc6654c090741c508d4c265717b0a1d6e8948de5d6927dd356f22c2085607f7b9549bb0f4ee7aafcb3b2f4b862b3
kubernetes-node-linux-ppc64le.tar.gzc26df8571204a0ae5b18a126c21cd8985b6fd0a8df50c8da4cfd86006b3974fa452ff30de0c4f6ed5cd54e59705a2f639a8ee4201fd681048968cbea416e7e40
kubernetes-node-linux-s390x.tar.gzd5a13e1d13a6d9ff081f691b06ca66b8e9bff7cd12591b1281e7c05382aeeee4cd3ec83a23176e07d21c018ca29795b3944cbff7af5f62700046bf2062912959
kubernetes-node-windows-amd64.tar.gz57f4b842d1637a67ae59e400d237c8d63aea9a7dc018384e3fca9804d457b9125f46bb5776d36f2150642bb70f6f2e8781b4e62e8de84627c076004d1244212a

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.

namearchitectures
registry.k8s.io/conformance:v1.32.0amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.32.0amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.32.0amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.32.0amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.32.0amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.32.0amd64, arm64, ppc64le, s390x

Changelog since v1.31.0

Urgent Upgrade Notes

There are no urgent upgrade notes for the v1.32 release.

Changes by Kind

Deprecation

  • Reverted the DisableNodeKubeProxyVersion feature gate to default-off to give a full year from deprecation announcement in 1.29 to clearing the field by default, per the Kubernetes deprecation policy. (#126720, @liggitt) [SIG Architecture and Node]
  • ServiceAccount metadata.annotations[kubernetes.io/enforce-mountable-secrets]: deprecated since v1.32; no removal deadline. Prefer separate namespaces to isolate access to mounted secrets. (#128396, @ritazh) [SIG API Machinery, Apps, Auth, CLI and Testing]

API Change

  • ACTION REQUIRED for custom scheduler plugin developers: PodEligibleToPreemptOthers in the preemption interface now includes ctx in the parameters. Please update your plugins' implementation accordingly. (#126465, @googs1025) [SIG Scheduling]

  • Changed NodeToStatusMap from a map to a struct and exposed methods to access the entries. Added absentNodesStatus, which informs the status of nodes that are absent in the map. For developers of out-of-tree PostFilter plugins, ensure to update the usage of NodeToStatusMap. Additionally, NodeToStatusMap should eventually be renamed to NodeToStatusReader. (#126022, @macsko) [SIG Node, Scheduling, and Testing]

  • A new /resize subresource was added to request pod resource resizing. Update your k8s client code to utilize the /resize subresource for Pod resizing operations. (#128266, @AnishShah) [SIG API Machinery, Apps, Node and Testing]

  • A new feature that allows unsafe deletion of corrupt resources has been added, it is disabled by default, and it can be enabled by setting the option --feature-gates=AllowUnsafeMalformedObjectDeletion=true. It comes with an API change, a new delete option ignoreStoreReadErrorWithClusterBreakingPotential has been introduced, it is not set by default, this maintains backward compatibility. In order to perform an unsafe deletion of a corrupt resource, the user must enable the option for the delete request. A resource is considered corrupt if it can not be successfully retrieved from the storage due to a) transformation error e.g. decryption failure, or b) the object failed to decode. Normal deletion flow is attempted first, and if it fails with a corrupt resource error then it triggers unsafe delete. In addition, when this feature is enabled, the 'details' field of 'Status' from the LIST response includes information that identifies the corrupt object(s). NOTE: unsafe deletion ignores finalizer constraints, and skips precondition checks. WARNING: this may break the workload associated with the resource being unsafe-deleted, if it relies on the normal deletion flow, so cluster breaking consequences apply. (#127513, @tkashem) [SIG API Machinery, Etcd, Node and Testing]

  • Added singleProcessOOMKill flag to the kubelet configuration. Setting that to true enable single process OOM killing in cgroups v2. In this mode, if a single process is OOM killed within a container, the remaining processes will not be OOM killed. (#126096, @utam0k) [SIG API Machinery, Node, Testing and Windows]

  • Added a /flagz endpoint for kube-apiserver endpoint. (#127581, @richabanker) [SIG API Machinery, Architecture, Auth and Instrumentation]

  • Added a Stream field to PodLogOptions, which allows clients to request certain log stream (stdout or stderr) of the container. Please also note that the combination of a specific Stream and TailLines is not supported. (#127360, @knight42) [SIG API Machinery, Apps, Architecture, Node, Release and Testing]

  • Added alpha support for asynchronous Pod preemption. When the SchedulerAsyncPreemption feature gate is enabled, the scheduler now runs API calls to trigger preemptions asynchronously for better performance. (#128170, @sanposhiho) [SIG Scheduling and Testing]

  • Added driver-owned fields in ResourceClaim.Status to report device status data for each allocated device. (#128240, @LionelJouin) [SIG API Machinery, Network, Node and Testing]

  • Added enforcement of an upper cost bound for DRA evaluations of CEL. The API server and scheduler now enforce an upper bound on the cost and runtime steps required for evaluating a CEL expression. (#128101, @pohly) [SIG API Machinery and Node]

  • Added the ability to change the maximum backoff delay accrued between container restarts for a node for containers in CrashLoopBackOff. To set this for a node, turn on the feature gate KubeletCrashLoopBackoffMax and set the CrashLoopBackOff.MaxContainerRestartPeriod field between "1s" and "300s" in your kubelet config file. (#128374, @lauralorenz) [SIG API Machinery and Node]

  • Allow for Pod search domains to be a single dot . or contain an underscore _ (#127167, @adrianmoisey) [SIG Apps, Network and Testing]

  • Annotation batch.kubernetes.io/cronjob-scheduled-timestamp added to Job objects scheduled from CronJobs is promoted to stable. (#128336, @soltysh)

  • Apply fsGroup policy for ReadWriteOncePod volumes. (#128244, @gnufied) [SIG Storage and Testing]

  • Changed the Pod API to support resources at spec level for pod-level resources. (#128407, @ndixita) [SIG API Machinery, Apps, CLI, Cluster Lifecycle, Node, Release, Scheduling and Testing]

  • ContainerStatus.AllocatedResources is now guarded by a separate feature gate, InPlacePodVerticalSaclingAllocatedStatus (#128377, @tallclair) [SIG API Machinery, CLI, Node, Scheduling and Testing]

  • Coordination.v1alpha1 API is dropped and replaced with coordination.v1alpha2. Old coordination.v1alpha1 types must be deleted before upgrade (#127857, @Jefftree) [SIG API Machinery, Etcd, Scheduling and Testing]

  • DRA: Restricted the length of opaque device configuration parameters. At admission time, Kubernetes enforces a 10KiB size limit. (#128601, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

  • DRA: scheduling pods is up to 16x faster, depending on the scenario. Scheduling throughput depends a lot on cluster utilization. It is higher for lightly loaded clusters with free resources and gets lower when the cluster utilization increases. (#127277, @pohly) [SIG API Machinery, Apps, Architecture, Auth, Etcd, Instrumentation, Node, Scheduling and Testing]

  • DRA: the DeviceRequestAllocationResult struct now has an "AdminAccess" field which should be used instead of the corresponding field in the DeviceRequest field when dealing with an allocation. If a device is only allocated for admin access, allocating it again for normal usage is now supported, as originally intended. To allow admin access, starting with 1.32 the DRAAdminAccess feature gate must be enabled. (#127266, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Network, Node, Scheduling and Testing]

  • Disallow k8s.io and kubernetes.io namespaced extra key in structured authentication configuration. (#126553, @aramase) [SIG Auth]

  • Fixed a bug in the NestedNumberAsFloat64 Unstructured field accessor that could have caused it to return rounded float64 values instead of errors when accessing very large int64 values. (#128099, @benluddy)

  • Fixed the bug where spec.terminationGracePeriodSeconds of the pod will always be overwritten by the MaxPodGracePeriodSeconds of the soft eviction, you can enable the AllowOverwriteTerminationGracePeriodSeconds feature gate, which will restore the previous behavior. If you do need to set this, please file an issue with the Kubernetes project to help contributors understand why you needed it. (#122890, @HirazawaUi) [SIG API Machinery, Architecture, Node and Testing]

  • Graduated Job's ManagedBy field to beta. (#127402, @mimowo) [SIG API Machinery, Apps and Testing]

  • Implemented a new, alpha seLinuxChangePolicy field within a Pod-level securityContext, under SELinuxChangePolicy feature gate. This field allows for opting out from mounting Pod volumes with SELinux label when SELinuxMount feature is enabled (it is alpha and disabled by default now). Please see the KEP how we expect to warn users before any SELinux behavior changes and how they can opt-out before. Note that this field and feature gate is useful only with clusters that run with SELinux enabled. No action is required on clusters without SELinux. (#127981, @jsafrane) [SIG API Machinery, Apps, Architecture, Node, Storage and Testing]

  • Introduced v1alpha1 API for mutating admission policies, enabling extensible # admission control via CEL expressions (KEP 3962: Mutating Admission Policies). # To use, enable the MutatingAdmissionPolicy feature gate and the admissionregistration.k8s.io/v1alpha1 # API via --runtime-config. (#127134, @jpbetz) [SIG API Machinery, Auth, Etcd and Testing]

  • Introduced compressible resource setting on system reserved and kube reserved slices. (#125982, @harche)

  • kube-apiserver: Promoted the StructuredAuthorizationConfiguration feature gate to GA. The --authorization-config flag now accepts AuthorizationConfiguration in version apiserver.config.k8s.io/v1 (with no changes from apiserver.config.k8s.io/v1beta1). (#128172, @liggitt) [SIG API Machinery, Auth and Testing]

  • kube-proxy now reconciles Service/Endpoint changes with conntrack table and cleans up only stale UDP flow entries (#127318, @aroradaman) [SIG Network and Windows]

  • kube-scheduler removed AzureDiskLimits ,CinderLimits EBSLimits and GCEPDLimits plugin. Given the corresponding CSI driver reports how many volumes a node can handle in NodeGetInfoResponse, the kubelet stores this limit in CSINode and the scheduler then knows the limit of the driver on the node. Removed plugins AzureDiskLimits, CinderLimits, EBSLimits and GCEPDLimits if you explicitly enabled them in the scheduler config. (#124003, @carlory) [SIG Scheduling, Storage and Testing]

  • kubelet: the --image-credential-provider-config file was loaded with strict deserialization, which failed if the config file contained duplicate or unknown fields. This protected against accidentally running with malformed config files, unindented files, or typos in field names, and it prevented unexpected behavior. (#128062, @aramase) [SIG Auth and Node]

  • NodeRestriction admission now validates the audience value that kubelet is requesting a service account token for is part of the pod spec volume. This change is introduced with a new kube-apiserver featuregate ServiceAccountNodeAudienceRestriction that's enabled by default. (#128077, @aramase) [SIG Auth, Storage and Testing]

  • Promoted CustomResourceFieldSelectors to stable; the feature was enabled by default. The --feature-gates=CustomResourceFieldSelectors=true flag was no longer needed on kube-apiserver binaries and would be removed in a future release. (#127673, @jpbetz) [SIG API Machinery and Testing]

  • Promoted feature gate StatefulSetAutoDeletePVC from beta to stable. (#128247, @mattcary) [SIG API Machinery, Apps, Auth and Testing]

  • Removed all support for classic dynamic resource allocation (DRA). The DRAControlPlaneController feature gate, formerly alpha, is no longer available. Kubernetes now only uses the structured parameters model (also alpha) for allocating dynamic resources to Pods.

    if and only if classic DRA was enabled in a cluster, remove all workloads (pods, app deployments, etc. ) which depend on classic DRA and make sure that all PodSchedulingContext resources are gone before upgrading. PodSchedulingContext resources cannot be removed through the apiserver after an upgrade and workloads would not work properly. (#128003, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

  • Removed generally available feature gate HPAContainerMetrics (#126862, @carlory) [SIG API Machinery, Apps and Autoscaling]

  • Removed restrictions on subresource flag in kubectl commands (#128296, @AnishShah) [SIG CLI]

  • Revised the kubelet API Authorization with new subresources, that allow finer-grained authorization checks and access control for kubelet endpoints. Provided you enable the KubeletFineGrainedAuthz feature gate, you can access kubelet's /healthz endpoint by granting the caller nodes/helathz permission in RBAC. Similarly you can also access kubelet's /pods endpoint to fetch a list of Pods bound to that node by granting the caller nodes/pods permission in RBAC. Similarly you can also access kubelet's /configz endpoint to fetch kubelet's configuration by granting the caller nodes/configz permission in RBAC. You can still access kubelet's /healthz, /pods and /configz by granting the caller nodes/proxy permission in RBAC but that also grants the caller permissions to exec, run and attach to containers on the nodes and doing so does not follow the least privilege principle. Granting callers more permissions than they need can give attackers an opportunity to escalate privileges. (#126347, @vinayakankugoyal) [SIG API Machinery, Auth, Cluster Lifecycle and Node]

  • The core functionality of Dynamic Resource Allocation (DRA) got promoted to beta. No action is required when upgrading, the previous v1alpha3 API is still supported, so existing deployments and DRA drivers based on v1alpha3 continue to work. Downgrading from 1.32 to 1.31 with DRA resources in the cluster (resourceclaims, resourceclaimtemplates, deviceclasses, resourceslices) is not supported because the new v1beta1 is used as storage version and not readable by 1.31. (#127511, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

  • The default value for node-monitor-grace-period has been increased to 50s (earlier 40s) (Ref - https://github.com/kubernetes/kubernetes/issues/121793) (#126287, @devppratik) [SIG API Machinery, Apps and Node]

  • The resource/v1alpha3.ResourceSliceList filed which should have been named "metadata" but was instead named "listMeta" is now properly "metadata". (#126749, @thockin) [SIG API Machinery]

  • The synthetic "Bookmark" event for the watch stream requests will now include a new annotation: kubernetes.io/initial-events-list-blueprint. THe annotation contains an empty, versioned list that is encoded in the requested format (such as protobuf, JSON, or CBOR), then base64-encoded and stored as a string. (#127587, @p0lyn0mial) [SIG API Machinery]

  • To enhance usability and developer experience, CRD validation rules now support direct use of (CEL) reserved keywords as field names in object validation expressions. Name format CEL library is supported in new expressions. (#126977, @aaron-prindle) [SIG API Machinery, Architecture, Auth, Etcd, Instrumentation, Release, Scheduling and Testing]

  • Updated incorrect description of persistentVolumeClaimRetentionPolicy (#126545, @yangjunmyfm192085) [SIG API Machinery, Apps and CLI]

  • X.509 client certificate authentication to the kube-apiserver now produces credential IDs (derived from the certificate's signature) , for use in audit logging. (#125634, @ahmedtd) [SIG API Machinery, Auth and Testing]

Feature

  • Added Windows support for the node memory manager. (#128560, @marosset) [SIG Node and Windows]

  • Added --concurrent-daemonset-syncs command line flag to kube-controller-manager. This value sets the number of workers for the daemonset controller. (#128444, @tosi3k)

  • Added a /statusz endpoint for the kube-apiserver endpoint. (#125577, @richabanker) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Instrumentation, Network, Node and Testing]

  • Added a health check for the device plugin gRPC registration server. When the registration server is down, kubelet is marked as unhealthy. If systemd watchdog is configured, this will result in a kubelet restart. (#128432, @zhifei92) [SIG Node]

  • Added a kubelet metric container_aligned_compute_resources_count to report the count of containers getting aligned compute resources. (#127155, @ffromani) [SIG Node and Testing]

  • Added a kubelet metrics to report informations about the cpu pools managed by cpumanager when the static policy is in use. (#127506, @ffromani) [SIG Node and Testing]

  • Added a new controller, volumeattributesclass-protection-controller, into the kube-controller-manager. The new controller manages a protective finalizer on VolumeAttributesClass objects. (#123549, @carlory) [SIG API Machinery, Apps, Auth and Storage]

  • Added a new option strict-cpu-reservation for CPU Manager static policy. When this option is enabled, CPU cores in reservedSystemCPUs will be strictly used for system daemons and interrupt processing no longer available for any workload. (#127483, @jingczhang) [SIG Node]

  • Added a one-time random duration of up to 50% of kubelet's nodeStatusReportFrequency to help spread the node status update load evenly over time. (#128640, @mengqiy)

  • Added an option to enable leader election in local-up-cluster.sh via the LEADER_ELECT CLI flag. (#127786, @Jefftree)

  • Added kubelet support for systemd watchdog integration. With this enabled, systemd can automatically recover a hung kubelet. (#127566, @zhifei92) [SIG Cloud Provider, Node and Testing]

  • Added metrics to measure the latency of DRA Node operations and DRA GRPC calls (#127146, @bart0sh) [SIG Instrumentation, Network, Node, and Testing]

  • Added new functionality to the Go client code (client-go) library. The List() method for the metadata client allows enabling API streaming when fetching collections; this improves performance when listing many objects. To request this behavior, your client software must enable the WatchListClient client-go feature gate. Additionally, streaming is only available if supported by the cluster; the API server that you connect to must also support streaming. If the API server does not support or allow streaming, then client-go falls back to fetching the collection using the list API verb. (#127388, @p0lyn0mial) [SIG API Machinery and Testing]

  • Added preemptionPolicy field when using kubectl get PriorityClass -owide (#126529, @googs1025) [SIG CLI]

  • Added status for extended Pod resources within the status.containerStatuses[].resources field. (#124227, @iholder101) [SIG Node and Testing]

  • Added support to the kube-apiserver for an alpha feature enabling external signing of service account tokens and fetching of public verifying keys, by enabling the Alpha ExternalServiceAccountTokenSigner feature gate and specifying --service-account-signing-endpoint. The flag value can either be the location of a Unix domain socket on a filesystem, or be prefixed with an @ symbol and name a Unix domain socket in the abstract socket namespace. (#128190, @HarshalNeelkamal) [SIG API Machinery, Apps, Auth, Etcd, Instrumentation, Node, Release and Testing]

  • Added the feature gate CBORServingAndStorage to allow CBOR as the encoding for API request and response bodies, and as the storage encoding for custom resources. Clients must opt in; programs built with client-go can do this using the client-go feature gates ClientsAllowCBOR and ClientsPreferCBOR. (#128539, @benluddy) [SIG API Machinery, Etcd and Testing]

  • Adopted a new implementation of watch caches for list verbs, using a btree data structure. The new implementation is active by default; you can opt out by disabling the BtreeWatchCache feature gate. (#128415, @serathius) [SIG API Machinery, Auth and Cloud Provider]

  • Allows PreStop lifecycle handler's sleep action to have a zero value (#127094, @sreeram-venkitesh) [SIG Apps, Node and Testing]

  • CRI: Added a field to support CPU affinity on Windows. (#124285, @kiashok) [SIG Node and Windows]

  • Changed OOM score adjustment calculation for sidecar containers: the OOM adjustment for these containers will match or fall below the OOM score adjustment of regular containers in the Pod. (#128029, @bouaouda-achraf)

  • Client-go/rest: contextual logging of request/response with accurate source code location of the caller (#126999, @pohly) [SIG API Machinery and Instrumentation]

  • DRA: The resource claim controller now maintains metrics about the total number of ResourceClaims and the number of allocated ResourceClaims. (#127661, @pohly) [SIG Apps, Instrumentation and Node]

  • Enabled graceful shutdown feature for Windows node (#127404, @zylxjtu) [SIG Node, Testing and Windows]

  • Enabled kube-controller-manager '--concurrent-job-syncs' flag works on orphan Pod processors (#126567, @fusida) [SIG Apps]

  • Ensured resizing for Guaranteed pods with integer CPU requests on nodes with static CPU & Memory policy configured is not allowed for the beta release of in-place resize. The feature gate InPlacePodVerticalScalingExclusiveCPUs defaults to false, but can be enabled to unblock development on (#127262, @tallclair) [SIG Node]. (#128287, @esotsal) [SIG Node, Release and Testing]

  • Extend discovery GroupManager with Group lister interface (#127524, @mjudeikis) [SIG API Machinery]

  • Fixed: Avoid overwriting in-pod vertical scaling updates on systemd daemon reloads when using systemd (#124216, @iholder101) [SIG Node]

  • Fixed an issue where kubectl doesn't print image volume when kubectl describe a pod with that volume. (#126706, @carlory)

  • Graduated the AnonymousAuthConfigurableEndpoints feature gate to beta and enable by default to allow configurable endpoints for anonymous authentication. (#127009, @vinayakankugoyal) [SIG Auth]

  • Graduated the kubelet memory manager to generally available (GA). (#128517, @Tal-or)

  • Graduated SchedulerQueueingHints to beta; the feature gate is now enabled by default. (#128472, @sanposhiho) [SIG Scheduling]

  • Graduated the WatchList feature gate to Beta for kube-apiserver and enabled WatchListClient for KCM. (#128053, @p0lyn0mial) [SIG API Machinery and Testing]

  • Implemented a queueing hint for PersistentVolumeClaim/Add event in the CSILimit plugin. (#124703, @utam0k) [SIG Scheduling and Storage]

  • Implemented new cluster events UpdatePodSchedulingGatesEliminated and UpdatePodTolerations for scheduler plugins. (#127083, @sanposhiho)

  • Improved Node's QueueingHint in the NodeAffinity plugin by ignoring unrelated changes that keep pods unschedulable. (#127444, @dom4ha) [SIG Scheduling and Testing]

  • Improved Node's QueueingHint in the NodeResourceFit plugin by ignoring unrelated changes that keep pods unschedulable. (#127473, @dom4ha) [SIG Scheduling and Testing]

  • Improved performance of the job controller when handling job delete events. (#127378, @hakuna-matatah)

  • Improved performance of the job controller when handling job update events. (#127228, @hakuna-matatah)

  • Included an additional resource labeltransformation in on_operations_total metric which could be used for resource specific validations for example handling of encryption config by the apiserver. (#126512, @kmala) [SIG API Machinery, Auth, Etcd and Testing]

  • Introduced a new metric kubelet_admission_rejections_total to track the number of pods rejected during admission. (#128556, @AnishShah)

  • JWT authenticators now set the jti claim (if present and is a string value) as credential id for use by audit logging. (#127010, @aramase) [SIG API Machinery, Auth and Testing]

  • kube-apiserver: Promoted AuthorizeWithSelectors feature to beta, which includes field and label selector information from requests in webhook authorization calls. Promoted AuthorizeNodeWithSelectors feature to beta, which changes node authorizer behavior to limit requests from node API clients, so that each Node can only get / list / watch its own Node API object, and can also only get / list / watch Pod API objects bound to that node. Clients using kubelet credentials to read other nodes or unrelated pods must change their authentication credentials (recommended), adjust their usage, or obtain broader read access independent of the node authorizer. (#128168, @liggitt) [SIG API Machinery, Auth and Testing]

  • kube-apiserver: a new --requestheader-uid-headers flag allows configuring request header authentication to obtain the authenticating user's UID from the specified headers. The suggested value for the new option is X-Remote-Uid. When specified, the kube-system/extension-apiserver-authentication configmap will include the value in its .data[requestheader-uid-headers] field. (#115834, @stlaz) [SIG API Machinery, Auth, Cloud Provider and Testing]

  • kube-proxy uses field-selector clusterIP!=None on Services to avoid watching for Headless Services, reducing unnecessary network bandwidth (#126769, @Sakuralbj) [SIG Network]

  • : kubeadm upgrade apply now supports phase sub-command, users can use kubeadm upgrade apply phase <phase-name> to execute the specified phase, or use kubeadm upgrade apply --skip-phases <phase-names> to skip some phases during cluster upgrade. (#126032, @SataQiu) [SIG Cluster Lifecycle]

  • kubeadm: kubeadm upgrade node now supports addon and post-upgrade phases. Users can use kubeadm upgrade node phase addon to execute the addon upgrade, or use kubeadm upgrade node --skip-phases addon to skip the addon upgrade. Currently, the post-upgrade phase is no-op, and it is mainly used to handle some release-specific post-upgrade tasks. (#127242, @SataQiu) [SIG Cluster Lifecycle]

  • kubeadm: added a validation warning when the certificateValidityPeriod is more than the caCertificateValidityPeriod (#126538, @SataQiu) [SIG Cluster Lifecycle]

  • kubeadm: added the feature gate NodeLocalCRISocket. When the feature gate is enabled, kubeadm will generate the /var/lib/kubelet/instance-config.yaml file to customize the containerRuntimeEndpoint field in the kubelet configuration for each node and will not write the same CRI socket on the Node object as an annotation. (#128031, @HirazawaUi) [SIG Cluster Lifecycle]

  • kubeadm: allow mixing the flag --config with the special flag --print-manifest of the subphases of 'kubeadm init phase addon'. (#126740, @neolit123) [SIG Cluster Lifecycle]

  • kubeadm: consider --bind-address or --advertise-address and --secure-port for control plane components when the feature gate WaitForAllControlPlaneComponents is enabled. Use /livez for kube-apiserver and kube-scheduler, but continue using /healthz for kube-controller-manager until it supports /livez. (#128474, @neolit123) [SIG Cluster Lifecycle]

  • kubeadm: if an unknown command name is passed to any parent command such as 'kubeadm init phase' return an error. If 'kubeadm init phase' or another command that has subcommands is called without subcommand name, print the available commands and also return an error. (#127096, @neolit123) [SIG Cluster Lifecycle]

  • kubeadm: promoted feature gate EtcdLearnerMode to GA. Learner mode in etcd deployed by kubeadm is now locked to enabled by default. (#126374, @pacoxu) [SIG Cluster Lifecycle]

  • kubelet: add log and event for cgroup v2 with kernel older than 5.8. (#126595, @pacoxu) [SIG Node]

  • Kubernetes is now built with Go 1.23.3. (#128852, @cpanato) [SIG Release and Testing]

  • Kubernetes is now built with go 1.23.0 (#127076, @cpanato) [SIG Release and Testing]

  • Kubernetes was built with Go 1.23.1. (#127611, @haitch) [SIG Release and Testing]

  • Kubernetes was built with Go 1.23.2. (#128110, @haitch) [SIG Release and Testing]

  • Label apps.kubernetes.io/pod-index added to Pod from StatefulSets is promoted to stable Label batch.kubernetes.io/job-completion-index added to Pods from Indexed Jobs is promoted to stable (#128387, @alaypatel07) [SIG Apps]

  • LoadBalancerIPMode feature was marked as GA. (#127348, @RyanAoh) [SIG Apps, Network and Testing]

  • Locked the custom profiling feature in kubectl debug to true. (#127187, @ardaguclu) [SIG CLI and Testing]

  • Output for the ScalingReplicaSet event has changed from: Scaled <up|down> replica set to from to: Scaled <up|down> replica set from to . (#125118, @jsoref) [SIG Apps and CLI]

  • PodLifecycleSleepAction is graduated to GA (#128046, @AxeZhan) [SIG Architecture, Node and Testing]

  • Pods were allowed to use the net.ipv4.tcp_rmem and net.ipv4.tcp_wmem sysctl by default when the kernel version was 4.15 or higher. With the kernel 4.15 the sysctl became namespaced. Pod Security admission allowed these sysctl in v1.32+ versions of the baseline and restricted policies. (#127489, @pacoxu) [SIG Auth, Network and Node]

  • Prepared Pod validation to handle version skew for InPlacePodVerticalScaling's beta graduation. (#128186, @sreeram-venkitesh)

  • Promoted RecoverVolumeExpansionFailure feature gate to beta. (#128342, @gnufied) [SIG Apps and Storage]

  • Promoted RetryGenerateName to stable; the feature is enabled by default. --feature-gates=RetryGenerateName=true not needed on kube-apiserver binaries and will be removed in a future release. (#127093, @jpbetz) [SIG API Machinery]

  • Promoted SizeMemoryBackedVolumes to stable. (#126981, @kannon92) [SIG Node, Storage and Testing]

  • Promoted the RelaxedEnvironmentVariableValidation feature gate to beta and is enabled by default. (#126897, @HirazawaUi)

  • Promoted the feature gates StrictCostEnforcementForVAP and StrictCostEnforcementForWebhooks. (#127302, @cici37) [SIG API Machinery and Testing]

  • Promoted the ServiceAccountTokenJTI feature to GA, which adds a jti claim to issued service account tokens and embeds the jti claim as a authentication.kubernetes.io/credential-id=["JTI=..."] value in user extra info

    • Promoted the ServiceAccountTokenPodNodeInfo feature to GA, which adds the node name and uid as claims into service account tokens mounted into running pods, and embeds that information as authentication.kubernetes.io/node-name and authentication.kubernetes.io/node-uid user extra info when the token is used
    • Promoted the ServiceAccountTokenNodeBindingValidation feature to GA, which validates service account tokens bound directly to nodes. (#128169, @liggitt) [SIG API Machinery, Auth and Testing]

  • Realigned line breaks from kubectl explain descriptions. (#126533, @ah8ad3)

  • Removed attachable volume limits from the capacity of the node for the following volume type when the kubelet was started, affecting the following volume types when the corresponding csi driver was installed:

  • Reverted Go version used to build Kubernetes to 1.23.0. (#127861, @xmudrii) [SIG Release and Testing]

  • Support inflight_events metric in the scheduler for QueueingHint. (#127052, @sanposhiho) [SIG Scheduling]

  • Support specifying a custom network parameter when running e2e-node-tests with the remote option. (#127574, @bouaouda-achraf) [SIG Node and Testing]

  • The Job controller now considers sidecar container restart counts when removing pods. (#124952, @AxeZhan) [SIG Apps and CLI]

  • The TopologyManagerPolicyOptions feature-flag is promoted to GA. (#128124, @PiotrProkop)

  • The scheduler implemented QueueingHint in VolumeBinding plugin's CSIDriver event, which enhanced the throughput of scheduling. (#125171, @YamasouA) [SIG Scheduling and Storage]

  • The scheduler retries gated Pods more appropriately, giving them a backoff penalty too. (#126029, @sanposhiho) [SIG Scheduling]

  • Unallowed label values will show up as "unexpected" in scheduler metrics. (#126762, @richabanker) [SIG Instrumentation and Scheduling]

  • Updated the control plane's trust anchor publisher to create and manage a new ClusterTrustBundle object, associated with the kubernetes.io/kube-apiserver-serving X.509 certificate signer. This ClusterTrustBundle contains a PEM bundle in its payload that you can use to verify kube-apiserver serving certificates. (#127326, @stlaz) [SIG API Machinery, Apps, Auth, Cluster Lifecycle and Testing]

  • Vendor: updated system-validators to v1.9.0. (#128149, @neolit123) [SIG Cluster Lifecycle and Node]

  • Vendor: updated system-validators to v1.9.1. (#128533, @neolit123)

  • When SchedulerQueueingHint is enabled, the scheduler's in-tree plugins now subscribe to specific node events to decide whether to requeue Pods. This allows the scheduler to handle cluster events faster with less memory.

    Specific node events include updates to taints, tolerations or allocatable. In-tree plugins now ignore node updates that don't modify any of these fields. (#127220, @sanposhiho) [SIG Node, Scheduling and Storage]

  • When SchedulerQueueingHints is enabled, clear events cached in the scheduling queue as soon as possible so that the scheduler consumes less memory. (#120586, @sanposhiho) [SIG Scheduling]

  • Windows: Support CPU and Topology manager on Windows. (#125296, @jsturtevant) [SIG Node and Windows]

Documentation

  • Clarified the kube-controller-manager documentation for --allocate-node-cidrs, --cluster-cidr, and --service-cluster-ip-range flags to accurately reflect their dependencies and usage conditions. (#126784, @eminwux) [SIG API Machinery, Cloud Provider and Docs]
  • Documented the --for=create option to kubectl wait. (#127327, @ryanwinter) [SIG CLI]
  • Fixed documentation for the apiserver_admission_webhook_fail_open_count and apiserver_admission_webhook_request_total metrics. The type label can have a value of "admit", not "mutating". (#127898, @modulitos)
  • kubeadm: fixed a misleading output (typo) about control-plane joining instructions when executing the "kubeadm init" command. (#128118, @amaddio)
  • The kubelet, when using --cloud-provider=external can use the --node-ip flag with one of the unspecified addresses 0.0.0.0 or ::, to create the Node with the IP of the default gateway of the corresponding IP family and then delegating the responsibility to the external cloud provider. This solves the bootstrap problems of out of tree cloud providers that are deployed as Pods within the cluster. (#125337, @aojea) [SIG Cloud Provider, Network, Node and Testing]
  • Added request header UID propagation, behind an alpha RemoteRequestHeaderUID feature gate. (#129081, @stalz) [SIG API SIG API Machinery, cluster lifecycle, testing]

Failing Test

  • kubelet plugins are now re-registered properly on Windows if the re-registration period is < 15ms. (#114136, @claudiubelu) [SIG Node, Storage, Testing and Windows]

Bug or Regression

    1. When the kubelet constructs the CRI mounts for the container which references an image volume source type, it passes the missing mount attributes to the CRI implementation, including readOnly, propagation, and recursiveReadOnly. When the readOnly field of the containerMount is explicitly set to false, the kubelet will now take the readOnlyas true to the CRI implementation because the image volume plugin requires the mount to be read-only.
    2. Fixed a bug where the pod is unexpectedly running when the image volume source type is used and mounted to /etc/hosts in the container. (#126806, @carlory) [SIG Node and Storage]
  • Added warnings for overlap paths in ConfigMap, Secret, DownwardAPI, Projected. Added warning for cases when ProjectedVolume with sources is provided. (#121968, @Peac36)
  • Apiserver repair controller is resilient to etcd errors during bootstrap and retries during 30 seconds before failing. (#126671, @fusida) [SIG Network]
  • Applyconfiguration-gen no longer generates duplicate methods and ambiguous member accesses when types end up with multiple members of the same name (through embedded structs). (#127001, @skitt) [SIG API Machinery]
  • Bookmark events are now sent immediately after all items in the watchCache store have been processed, improving consistency in client behavior. (#127012, @Chaunceyctx)
  • DRA: fixed several issues related to allocationMode: all. (#127565, @pohly)
  • DRA: when a DRA driver was started after creating pods which need resources from that driver, no additional attempt was made to schedule such unschedulable pods again. Only affected DRA with structured parameters. (#126807, @pohly) [SIG Node, Scheduling and Testing]
  • DRA: when enabling the scheduler queuing hint feature, pods got stuck as unschedulable for a while unnecessarily because recording the name of the generated ResourceClaim did not trigger scheduling. (#127497, @pohly) [SIG Auth, Node, Scheduling and Testing]
  • Disallowed label values will show up as "unexpected" in all system components' metrics. (#128100, @yongruilin) [SIG Architecture and Instrumentation]
  • Discarded the output streams of destination path check in kubectl cp when copying from local to pod and added a 3 seconds timeout to this check (#126652, @ardaguclu) [SIG CLI]
  • Fixed 1.31 regression that can crash kube-controller-manager's service-lb-controller loop. (#128182, @carlory) [SIG API Machinery, Cloud Provider and Network]
  • Fixed a 1.31 regression starting kubelet on Windows: Revert "fix: handle socket file detection on Windows". (#126976, @jsturtevant)
  • Fixed a 1.31 regression with API emulation versioning honors cohabitating resources. (#127239, @xuzhenglun)
  • Fixed a bug in the endpoints controller that failed to reconcile the Endpoint object after it was truncated (when it received more than 1000 endpoint addresses). (#127417, @aojea) [SIG Apps, Network and Testing]
  • Fixed a bug in the garbage collector controller which could block indefinitely due to a cache sync failure. This fix allows the garbage collector to eventually continue garbage collecting other resources if a given resource cannot be listed or watched. Any objects in the unsynced resource type with owner references with blockOwnerDeletion: true will not be known to the garbage collector. Use of blockOwnerDeletion has always been best-effort and racy on startup and object creation. With this fix, it continues to be best-effort for resources that cannot be synced by the garbage collector controller. (#125796, @haorenfsa) [SIG API Machinery, Apps and Testing]
  • Fixed a bug that occurred when the hostname label of a node did not match the node name, pods bound to a PersistentVolume with nodeAffinity using the hostname may be scheduled to the wrong node or experience scheduling failures. (#125398, @AxeZhan) [SIG Scheduling and Storage]
  • Fixed a bug where podCIDR was released before node was deleted. (#128305, @adrianmoisey) [SIG Apps and Network]
  • Fixed a bug where the kubelet ephemerally failed with failed to initialize top level QOS containers: root container [kubepods] doesn't exist, due to the cpuset cgroup being deleted on cgroup v2 with systemd cgroup manager. (#125923, @haircommander) [SIG Node and Testing]
  • Fixed a bug where the pod(with regular init containers)'s phase was not pending when the regular init container had not finished running after a node restart. (#126653, @zhifei92) [SIG Node and Testing]
  • Fixed a bug which the scheduler didn't correctly tell plugins Node deletion. This bug could impact all scheduler plugins subscribing to Node/Delete event, making the queue keep the Pods rejected by those plugins incorrectly at Node deletion. Among the in-tree plugins, PodTopologySpread is the only victim. (#127464, @sanposhiho) [SIG Scheduling and Testing]
  • Fixed a bug with dual stack clusters using the beta feature MultiCIDRServiceAllocator which could not create dual stack Services or Services with IPs in the secondary range. Users who wanted to use this feature in version 1.30 with dual stack clusters could work around the issue by setting the feature gate DisableAllocatorDualWrite to true. (#127598, @aojea) [SIG Network and Testing]
  • Fixed a possible memory leak in the QueueingHint (alpha feature). (#126962, @sanposhiho)
  • Fixed a potential memory leak in QueueingHint (alpha feature). (#127016, @sanposhiho)
  • Fixed a race condition in the kube-proxy initialization that could cause UDP traffic to service VIP. (#126532, @wedaly)
  • Fixed a race condition that could result in erroneous volume unmounts for flex volume plugins during kubelet restart. (#127669, @olyazavr)
  • Fixed a race condition that could result in erroneous volume unmounts for flex volume plugins on kubelet restart. (#128495, @olyazavr)
  • Fixed a regression in 1.29+ default configurations, where regular init containers may fail to start due to a temporary container runtime failure. (#127162, @gjkim42) [SIG Node]
  • Fixed a regression in default 1.29 configurations with the SidecarContainers feature enabled, where init containers may fail to start due to a temporary container runtime failure. (#126543, @gjkim42)
  • Fixed a regression introduced in v1.29 where conntrack entries for UDP connections to deleted pods did not get cleaned up correctly, which could (among other things) cause DNS problems when DNS pods were restarted. (#127780, @danwinship)
  • Fixed a scheduler preemption issue where the victim pod was not deleted due to incorrect status patching. This issue occurred when the preemptor and victim pods had different QoS classes in their status, causing the preemption to fail entirely. (#126644, @Huang-Wei)
  • Fixed a suboptimal scheduler preemption behavior where potential preemption victims were violating Pod Disruption Budgets. (#128307, @NoicFank) [SIG Scheduling]
  • Fixed an issue in the kubelet that showed when writeable layers and read-only layers were at different paths within the same mount. Kubernetes was previously detecting that the image filesystem was split, even when that was not really the case (#128344, @kannon92) [SIG Node]
  • Fixed an issue in the kubelet that showed when writeable layers and read-only layers were at different paths within the same mount. Kubernetes was previously detecting that the image filesystem was split, even when that was not really the case. (#126562, @kannon92)
  • Fixed an issue where eviction manager was not deleting unused images or containers. (#127874, @AnishShah)
  • Fixed an issue where requests sent by the KMSv2 service would be rejected due to having an invalid authority header. (#126930, @Ruddickmg) [SIG API Machinery and Auth]
  • Fixed data race in kubelet/volumemanager. (#127919, @carlory) [SIG Apps, Node and Storage]
  • Fixed fake client to accept request without metadata.name to better emulate behavior of actual client. (#126727, @jpbetz)
  • Fixed the ability to set the resolvConf option in drop-in kubelet configuration files, which validates that drop-in kubelet configuration files are in a supported version. (#127421, @liggitt)
  • Fixed the bug in NodeUnschedulable that only happens with QHint enabled, which the scheduler might miss some updates for the Pods rejected by NodeUnschedulable plugin and put the Pods in the queue for a longer time than needed. (#127427, @sanposhiho)
  • Fixed the estimated cost in CEL for expressions that perform equality checks on IPs, CIDRs, Quantities, Formats and URLs. (#126359, @jpbetz)
  • Fixed the incorrect help message of a metric "graceful_shutdown_end_time_seconds". Fixed incorrect value set for metrics "graceful_shutdown_start_time_seconds" and "graceful_shutdown_end_time_seconds" in certain cases during graceful node shutdown. (#128189, @zylxjtu) [SIG Node]
  • Fixed the reporting of elapsed times during evaluation of ValidatingAdmissionPolicy decisions and annotations. The apiserver_validating_admission_policy_check_duration metrics will now show elapsed times and no longer be zero. (#128463, @knrc)
  • Fixed the wrong hierarchical structure for both the child span and the parent span (i.e. SerializeObject and List). In the past, some children's spans appeared parallel to their parents. (#127551, @carlory) [SIG API Machinery and Instrumentation]
  • Fixed: dynamic client-go can now handle subresources with an UnstructuredList response (#126809, @ryantxu) [SIG API Machinery]
  • Fixed a bug where restartable and non-restartable init containers were not accounted for in the message and annotations of eviction event. (#124947, @toVersus) [SIG Node]
  • Fixed a kubelet and kube-apiserver memory leak in default 1.29 configurations related to tracing. (#126957, @dashpole) [SIG API Machinery, Architecture, Instrumentation and Node]
  • Fixed the bug in PodTopologySpread that only happens with QHint enabled, which the scheduler might miss some updates for the Pods rejected by PodTopologySpread plugin and put the Pods in the queue for a longer time than needed. (#127447, @sanposhiho) [SIG Scheduling]
  • For Dynamic Resource Allocation, labels in node selectors now are validated. Invalid labels already caused runtime errors before and are unlikely to occur in practice. (#128932, @pohly)
  • For Dynamic Resource Allocation, the new "v1beta1" kubelet gPRC was renamed so that the protobuf package name is unique. (#128764, @pohly) [SIG Node and Testing]
  • HostNetwork pods no longer depend on the PodIPs to be assigned to configure the defined hostAliases on the Pod (#126460, @aojea) [SIG Network, Node and Testing]
  • If a client makes an API streaming requests and specifies an application/json;as=Table content type, the API server now responds with a 406 (Not Acceptable) error. This change helps to ensure that unsupported formats, such as Table representations are correctly rejected. (#126996, @p0lyn0mial) [SIG API Machinery and Testing]
  • If an old pod spec has used image volume source, we must allow it when updating the resource even if the feature-gate ImageVolume is disabled. (#126733, @carlory) [SIG API Machinery, Apps and Node]
  • Improved PVC Protection Controller's scalability by batch-processing PVCs by namespace with lazy live pod listing. (#125372, @hungnguyen243) [SIG Apps, Node, Storage and Testing]
  • Improved the scalability of the PVC Protection Controller by batch-processing PVCs by namespace and implementing lazy live pod listing. (#126745, @hungnguyen243) [SIG Apps, Storage and Testing]
  • kube-apiserver: fixed a 1.31 regression that stopped honoring build ID overrides with the --version flag (#126665, @liggitt) [SIG API Machinery]
  • kubeadm: added "disable success" and "disable denial" as parameters of the "cache" plugin in the Corefile managed by kubeadm. This is to prevent conflicting responses during CoreDNS cache updates. (#128359, @matteriben) [SIG Cluster Lifecycle]
  • kubeadm: ensure that Pods from the upgrade preflight check CreateJob are properly terminated after a timeout. (#127333, @yuyabee) [SIG Cluster Lifecycle]
  • kubeadm: fixed an issue where the wrong member list was being reported when removing an etcd member. (#127650, @SataQiu)
  • kubeadm: when adding new control plane nodes with kubeamd join, ensure that the etcd member addition is performed only if a given member URL does not already exist in the list of members. Similarly, on "kubeadm reset" only remove an etcd member if its ID exists. (#127491, @SataQiu) [SIG Cluster Lifecycle]
  • kubelet now attempts to get an existing node if the request to create it fails with StatusForbidden. (#126318, @hoskeri) [SIG Node]
  • kubelet: Fix - the volume manager didn't check the device mount state in the actual state of the world before marking the volume as detached. It may cause a pod to be stuck in the Terminating state due to the above issue when it was deleted. (#128219, @carlory)
  • kubelet: Fixed a bug where kubelet wrongly drops the QOSClass field of the Pod's status when it rejects a Pod. (#128083, @carlory) [SIG Node and Testing]
  • kubelet: use the CRI stats provider if PodAndContainerStatsFromCRI feature is enabled (#126488, @haircommander) [SIG Node]
  • Made kubelet's /metrics/slis endpoint always available. (#128430, @richabanker) [SIG Architecture, Instrumentation and Node]
  • Node shutdown controller made a best effort to wait for CSI Drivers to complete the volume teardown process according to the pod priority groups. (#125070, @torredil) [SIG Node, Storage and Testing]
  • Reduced memory usage/allocations during wait for volume attachment. (#126575, @Lucaber) [SIG Node and Storage]
  • Removed unneeded permissions for system:controller:persistent-volume-binder and system:controller:expand-controller clusterroles (#125995, @carlory) [SIG Auth and Storage]
  • Reset streams when an error happens during port-forward allowing kubectl to maintain port-forward connection open. (#128318, @soltysh) [SIG API Machinery, CLI and Node]
  • Send an error on ResultChan and close the RetryWatcher when the client is forbidden or unauthorized from watching the resource. (#126038, @mprahl) [SIG API Machinery]
  • Terminated Pods on a node will not be re-admitted on kubelet restart. This fixes the problem of Completed Pods awaiting for the finalizer marked as Failed after the kubelet restart. (#126343, @SergeyKanzhelev) [SIG Node and Testing]
  • The CSI volume plugin stopped watching the VolumeAttachment object if the object is not found or the volume is not attached when kubelet waits for a volume attached. In the past, it would fail due to missing permission. (#126961, @carlory) [SIG Storage]
  • The Usage and VolumeCondition are both optional in the response and if CSIVolumeHealth feature gate is enabled kubelet needs to consider returning metrics if either one is set. (#127021, @Madhu-1) [SIG Storage]
  • The build-tag flag is reintroduced to conversion-gen and defaulter-gen which allow users to inject custom build tag during code generation process. (#128259, @dinhxuanvu)
  • Fixed problem with named ports not being available when specified in sidecar containers. (#127976, @chengjoey)
  • The scheduler started considering the resource requests of existing sidecar containers during the scoring process. (#127878, @AxeZhan) [SIG Scheduling and Testing]
  • Tighten validation on the qosClass field of pod status. This field is immutable but it would be populated with the old status by kube-apiserver if it is unset in the new status when updating this field via the status subsource. (#127744, @carlory) [SIG Apps, Instrumentation, Node, Storage and Testing]
  • Upgraded coreDNS to v1.11.3. (#126449, @BenTheElder) [SIG Cloud Provider and Cluster Lifecycle]
  • Use allocatedResources on PVC for node expansion in kubelet (#126600, @gnufied) [SIG Node, Storage and Testing]
  • When entering a value other than "external" to the "--cloud-provider" flag for the kubelet, kube-controller-manager, and kube-apiserver, the user will now receive a warning in the logs about the disablement of internal cloud providers, this is in contrast to the previous warnings about deprecation. (#127711, @elmiko) [SIG API Machinery, Cloud Provider and Node]
  • StartupProbe was explicitly stopped when the successThreshold was reached. This eliminated the problem of executing StartupProbe more times than the successThreshold. (#121206, @mochizuki875)
  • kubelet: on Windows, consistently resolve filesystem links to volume identifiers instead of inconsistently normalizing to drive letters. (#129103, @liggitt) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Storage and Windows]

Other (Cleanup or Flake)

  • Added a short output format argument for kubectl explain. You could now use -o as an abbreviation for --output in commands such as kubectl explain <resource> --output plaintext-openapiv2. (#127869, @ak20102763)
  • Added an example for kubectl delete with the --interactive flag. (#127512, @bergerhoffer) [SIG CLI]
  • Added: Log Line for Debugging possible merge errors for kubelet related Config requests. (#124389, @holgerson97)
  • Aggregated Discovery v2beta1 fixture is removed in ./api/discovery. Please use v2 (#127008, @Jefftree) [SIG API Machinery]
  • Append the image pull error for the pods status.containerStatuses[*].state.waiting.message when in image pull back-off (reason is ImagePullBackOff) instead of the generic Back-off pulling image… message. (#127918, @saschagrunert) [SIG Node and Testing]
  • CBOR-encoded watch responses now set the Content-Type header to "application/cbor-seq" instead of the nonconformant "application/cbor". (#128501, @benluddy) [SIG API Machinery, Etcd and Testing]
  • CRI client now used the default timeout for ImageFsInfo RPC. (#128052, @saschagrunert)
  • Clarified an API validation error for toleration if operator is Exists and value is not empty. (#128119, @saschagrunert) [SIG API Machinery and Apps]
  • Device manager: stop using annotations to pass CDI device info to runtimes. Containerd versions older than v1.7.2 don't support passing CDI info through CRI and need to be upgraded. (#126435, @bart0sh) [SIG Node]
  • Dropped support for InPlacePodVerticalScaling feature in Windows. (#128623, @AnishShah) [SIG Apps and Node]
  • Enabled CBORServingAndStorage feature gate – built-in APIs can now be served in CBOR format for clients that request it. (#128503, @benluddy) [SIG API Machinery, Etcd and Testing]
  • Fake clientsets now use a common, generic implementation. The corresponding structs are now private; callers must use the corresponding constructors. (#126503, @skitt) [SIG API Machinery, Architecture, Auth and Instrumentation]
  • Feature AllowServiceLBStatusOnNonLB remains deprecated and is now locked to false to support compatibility versions. (#128139, @Jefftree)
  • Feature gate "AllowServiceLBStatusOnNonLB" has been removed. This gate has been stable and unchanged for over a year. (#126786, @thockin) [SIG Apps]
  • Fixed a warning message about the gce in-tree cloud provider state. (#126773, @carlory)
  • Fixed spacing in --validate flag description in kubectl. (#128081, @soltysh)
  • Fixes a bug in the k8s.io/cloud-provider/service controller, it may panic when a service is updated because the event recorder was used before it was initialized. All cloud providers should using the v1.31.0 cloud provider service controller must ensure that the controllers is initialized before the informer start to process events or update it to the version 1.32.0. (#128179, @carlory) [SIG API Machinery, Cloud Provider, Network and Testing]
  • Fully removed PostStartHookContext.StopCh. (#127341, @mjudeikis)
  • kube-apiserver --admission-control-config-file files are now validated strictly (EnableStrict). Duplicate and unknown fields in the configuration will now cause an error. (#128013, @seans3)
  • kube-apiserver --egress-selector-config-file files were validated strictly (EnableStrict). Duplicate and unknown fields in the configuration will now cause an error. (#128011, @seans3) [SIG API Machinery and Testing]
  • kube-apiserver ResourceQuotaConfiguration admission plugin subsection within --admission-control-config-file files were validated strictly (EnableStrict). Duplicate and unknown fields in the configuration would cause an error. (#128038, @seans3)
  • kube-controller-manager --leader-migration-config files were now validated strictly (EnableStrict). Duplicate and unknown fields in the configuration would cause an error. (#128009, @seans3) [SIG API Machinery and Cloud Provider]
  • kube-proxy initialization waits for all pre-sync events from node and serviceCIDR informers to be delivered. (#126561, @wedaly) [SIG Network]
  • kube-proxy will no longer depend on conntrack binary for stale UDP connections cleanup (#126847, @aroradaman) [SIG Cluster Lifecycle, Network and Testing]
  • kubeadm: don't warn if crictl binary does not exist since kubeadm does not rely on crictl since v1.31. (#126596, @saschagrunert) [SIG Cluster Lifecycle]
  • kubeadm: increased the verbosity of API client dry-run actions during the subcommands "init", "join", "upgrade" and "reset". It also allowed dry-run on 'kubeadm join' even if there was no existing cluster by utilizing a faked, in-memory cluster-info ConfigMap. (#126776, @neolit123)
  • kubeadm: make sure the extra environment variables written to a kubeadm managed PodSpec are sorted alpha-numerically by the environment variable name. (#126743, @neolit123) [SIG Cluster Lifecycle]
  • kubeadm: removed the deprecated sub-phase of 'init kubelet-finilize' called experimental-cert-rotation, and use 'enable-client-cert-rotation' instead. (#126913, @pacoxu) [SIG Cluster Lifecycle]
  • kubeadm: removed socat and ebtables from kubeadm preflight checks (#127151, @saschagrunert) [SIG Cluster Lifecycle]
  • kubeadm: removed preflight check for existence of the conntrack binary, as conntrack is no longer a kube-proxy dependency in version 1.32 and newer. (#126953, @aroradaman)
  • kubeadm: removed the deprecated and NO-OP flags --feature-gates for kubeadm upgrade apply and --api-server-manifest, --controller-manager-manifest, and --scheduler-manifest for kubeadm upgrade diff. (#127123, @neolit123) [SIG Cluster Lifecycle]
  • kubeadm: removed the deprecated flag --experimental-output, please use the flag --output instead that serves the same purpose. Affected commands are: kubeadm config images list, kubeadm token list, kubeadm upgrade plan, kubeadm certs check-expiration. (#126914, @carlory) [SIG Cluster Lifecycle]
  • kubeadm: switched the kube-scheduler static Pod to use the endpoints /livez (for startup and liveness probes) and /readyz (for the readiness probe). Previously, /healthz was used for all probes, which is deprecated behavior in the scope of this component. (#126945, @liangyuanpeng) [SIG Cluster Lifecycle]
  • Optimized the code by filtering out empty strings for podUID when calling the getPodAndContainerForDevice method. (#126997, @lengrongfu)
  • Output a log as v4-level when a probe is triggered and shift the periodic timer of ReadinessProbe after manual run. (#119089, @mochizuki875)
  • Removed generally available feature gate ValidatingAdmissionPolicy. (#126645, @cici37) [SIG API Machinery, Auth, and Testing]
  • Removed generally available feature gate CloudDualStackNodeIPs. (#126840, @carlory) [SIG API Machinery and Cloud Provider]
  • Removed generally available feature gate LegacyServiceAccountTokenCleanUp. (#126839, @carlory) [SIG Auth]
  • Removed generally available feature gate MinDomainsInPodTopologySpread. (#126863, @carlory) [SIG Scheduling]
  • Removed generally available feature gate NewVolumeManagerReconstruction. (#126775, @carlory) [SIG Node and Storage]
  • Removed generally available feature gate NodeOutOfServiceVolumeDetach (#127019, @carlory) [SIG Apps and Testing]
  • Removed generally available feature gate StableLoadBalancerNodeSet. (#126841, @carlory) [SIG API Machinery, Cloud Provider and Network]
  • Removed generally available feature-gate ZeroLimitedNominalConcurrencyShares (#126894, @carlory) [SIG API Machinery]
  • Removed legacy cloud provider integration code and the "service-lb-controller", "cloud-node-lifecycle-controller" and the "node-route-controller" from kube-controller-manager. You can now either set the --cloud-provider command line argument to "external", or to the empty string. All other values are invalid. (#128197, @aojea) [SIG API Machinery, Apps and Cloud Provider]
  • Removed support for removing requests and limits during a pod resize. (#128683, @AnishShah) [SIG Apps, Node and Testing]
  • Removed support for the kubelet --runonce mode. If you specify the kubelet command line flag --runonce, this is an error. Setting runOnce in a kubelet configuration file is also an error, and specifying any value for that configuration option is now deprecated. (#126336, @HirazawaUi) [SIG Node and Scalability]
  • Removed the GAed feature gates for ServerSideApply and ServerSideFieldValidation. (#127058, @carlory)
  • Removed the KMSv2 and KMSv2KDF feature gates. The associated features graduated to stable in the Kubernetes v1.29 release. (#126698, @enj) [SIG API Machinery, Auth and Testing]
  • Removed the feature gate ComponentSLIs, which had been promoted to stable since v1.29. (#127787, @Jefftree) [SIG Architecture and Instrumentation]
  • Revised error handling for port forwards to Pods. Added stream resets preventing port-forward from blockage. (#128681, @soltysh) [SIG API Machinery, CLI and Testing]
  • Short circuit if the compaction request from apiserver is disabled. (#126627, @fusida) [SIG Etcd]
  • Show a warning message to inform users that the legacy profile is planned to be deprecated. (#127230, @mochizuki875) [SIG CLI]
  • The dynamicResources has been refactored to DynamicResources, now users can introduce the DynamicResources struct outside the dynamicresources package. (#128399, @JesseStutler) [SIG Node and Scheduling]
  • The flowcontrol.apiserver.k8s.io/v1beta3 API version of FlowSchema and PriorityLevelConfiguration is no longer served in v1.32. Migrate manifests and API clients to use the flowcontrol.apiserver.k8s.io/v1 API version, available since v1.29. More information is at https://kubernetes.io/docs/reference/using-api/deprecation-guide/#flowcontrol-resources-v132 (#127017, @carlory) [SIG API Machinery and Testing]
  • The alpha Dynamic Resource Allocation gRPC API is still available, but might be removed in future releases. Driver authors should update their DRA drivers to use the v1beta1 gRPC API. (#128646, @pohly) [SIG Node and Testing]
  • The feature-gate "PodHostIPs" has been removed. It is GA and its value has been locked since Kubernetes v1.30. (#128634, @thockin) [SIG Apps, Architecture, Node and Testing]
  • The getters for the field name and typeDescription of the Reflector struct were renamed. (#128035, @alexanderstephan)
  • The kube-apiserver --tracing-config-file is now validated strictly (EnableStrict). Duplicate and unknown fields in the configuration will now result in an error. (#128073, @seans3)
  • The kube-proxy command line flags --healthz-port and --metrics-port, which were previously deprecated, have now been removed. (#126889, @aroradaman) [SIG Network and Windows]
  • The kube-proxy command line flags --healthz-port and --metrics-port, which were previously deprecated, have now been removed. (#127930, @aroradaman) [SIG Network and Windows]
  • The members name and typeDescription of the Reflector struct were exported to allow for better user extensibility. (#127663, @alexanderstephan)
  • Changed the percentage marker in kubectl top node from % to (%). (#126995, @googs1025) [SIG CLI]
  • Updated cni-plugins to v1.5.1. (#126966, @saschagrunert) [SIG Cloud Provider, Node and Testing]
  • Updated cni-plugins to v1.6.0. (#128091, @saschagrunert) [SIG Cloud Provider, Node and Testing]
  • Updated cri-tools to v1.31.0. (#126590, @saschagrunert) [SIG Cloud Provider and Node]
  • Upgraded etcd client to v3.5.16. (#127279, @serathius) [SIG API Machinery, Auth, Cloud Provider and Node]
  • Upgraded github.com/coredns/corefile-migration to v1.0.24. (#126851, @BenTheElder) [SIG Architecture and Cluster Lifecycle]
  • Upgraded the functionality of kubectl kustomize as described at https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv5.4.2 and https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv5.5.0. (#127965, @koba1t)
  • ComponentSLIs feature is marked as GA and locked. (#128317, @Jefftree) [SIG Architecture and Instrumentation]
  • kubectl apply --server-side now supports --subresource congruent to kubectl patch. (#127634, @deads2k) [SIG CLI and Testing]
  • kubelet: fixed an issue mounting CSI volumes on Windows nodes in 1.32.0 release candidates. (#129083 liggitt) [SIG API Machinery, architecture, auth, cli, cloud-provider, cluster-lifecycle, instrumentation,network,node, release, storage, windows ]

Dependencies

Added

Changed

Removed

Contributors, the CHANGELOG-1.32.md has been bootstrapped with v1.32.0 release notes and you may edit now as needed.

Published by your Kubernetes Release Managers.

Reply all
Reply to author
Forward
0 new messages