Kubernetes v1.29.0-alpha.1 is live!

Skip to first unread message

Jim Angel

Sep 25, 2023, 5:21:54 PM9/25/23
to dev, kubernete...@googlegroups.com
Kubernetes Community,

Kubernetes v1.29.0-alpha.1 has been built and pushed using Golang version 1.21.1.

The release notes have been updated in CHANGELOG-1.29.md, with a pointer to them on GitHub:


Downloads for v1.29.0-alpha.1

Source Code

filenamesha512 hash

Client Binaries

filenamesha512 hash

Server Binaries

filenamesha512 hash

Node Binaries

filenamesha512 hash

Container Images

All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.


Changelog since v1.28.0

Changes by Kind


  • Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

API Change

  • Added a new ipMode field to the .status of Services where type is set to LoadBalancer. The new field is behind the LoadBalancerIPMode feature gate. (#119937@RyanAoh) [SIG API Machinery, Apps, Cloud Provider, Network and Testing]
  • Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps. The incorrect cost was evident when the result of a function was used in subsequent operations. (#119800@jpbetz) [SIG API Machinery, Auth and Cloud Provider]
  • Go API: the ResourceRequirements struct needs to be replaced with VolumeResourceRequirements for use with volumes. (#118653@pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling, Storage and Testing]
  • Kube-apiserver: adds --authentication-config flag for reading AuthenticationConfiguration files. --authentication-config flag is mutually exclusive with the existing --oidc-* flags. (#119142@aramase) [SIG API Machinery, Auth and Testing]
  • Kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta3 is removed in v1.29. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. (#119994@SataQiu) [SIG Scheduling and Testing]
  • Mark the onPodConditions field as optional in Job's pod failure policy. (#120204@mimowo) [SIG API Machinery and Apps]
  • Retry NodeStageVolume calls if CSI node driver is not running (#120330@rohitssingh) [SIG Apps, Storage and Testing]
  • The kube-scheduler selectorSpread plugin has been removed, please use the podTopologySpread plugin instead. (#117720@kerthcet) [SIG Scheduling]


  • --sync-frequency will not affect the update interval of volumes that use ConfigMaps or Secrets when the configMapAndSecretChangeDetectionStrategy is set to Cache. The update interval is only affected by node.alpha.kubernetes.io/ttl node annotation." (#120255@likakuli) [SIG Node]

  • Add a new scheduler metric, pod_scheduling_sli_duration_seconds, and start the deprecation for pod_scheduling_duration_seconds. (#119049@helayoty) [SIG Instrumentation, Scheduling and Testing]

  • Added apiserver_envelope_encryption_dek_cache_filled to measure number of records in data encryption key(DEK) cache. (#119878@ritazh) [SIG API Machinery and Auth]

  • Added kubectl node drain helper callbacks OnPodDeletionOrEvictionStarted and OnPodDeletionOrEvictionFailed; people extending kubectl can use these new callbacks for more granularity.

  • Adding apiserver identity to the following metrics: apiserver_envelope_encryption_key_id_hash_total, apiserver_envelope_encryption_key_id_hash_last_timestamp_seconds, apiserver_envelope_encryption_key_id_hash_status_last_timestamp_seconds, apiserver_encryption_config_controller_automatic_reload_failures_total, apiserver_encryption_config_controller_automatic_reload_success_total, apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds

    Fix bug to surface events for the following metrics: apiserver_encryption_config_controller_automatic_reload_failures_total, apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds, apiserver_encryption_config_controller_automatic_reload_success_total (#120438@ritazh) [SIG API Machinery, Auth, Instrumentation and Testing]

  • Bump distroless-iptables to 0.3.2 based on Go 1.21.1 (#120527@cpanato) [SIG Testing]

  • Changed kubectl help to display basic details for subcommands from plugins (#116752@xvzf) [SIG CLI]

  • Changed the KMSv2KDF feature gate to be enabled by default. (#120433@enj) [SIG API Machinery, Auth and Testing]

  • Graduated the following kubelet resource metrics to general availability:

    • container_cpu_usage_seconds_total
    • container_memory_working_set_bytes
    • container_start_time_seconds
    • node_cpu_usage_seconds_total
    • node_memory_working_set_bytes
    • pod_cpu_usage_seconds_total
    • pod_memory_working_set_bytes
    • resource_scrape_error

    Deprecated (renamed) scrape_error in favor of resource_scrape_error (#116897@Richabanker) [SIG Architecture, Instrumentation, Node and Testing]

  • Graduation API List chunking (aka pagination) feature to stable (#119503@wojtek-t) [SIG API Machinery, Cloud Provider and Testing]

  • Implements API for streaming for the etcd store implementation

    When sendInitialEvents ListOption is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. (#119557@p0lyn0mial) [SIG API Machinery]

  • Improve memory usage of kube-scheduler by dropping the .metadata.managedFields field that kube-scheduler doesn't require. (#119556@linxiulei) [SIG Scheduling]

  • In a scheduler with Permit plugins, when a Pod is rejected during WaitOnPermit, the scheduler records the plugin. The scheduler will use the record to honor cluster events and queueing hints registered for the plugin, to inform whether to retry the pod. (#119785@sanposhiho) [SIG Scheduling and Testing]

  • In tree cloud providers are now switched off by default. Please use DisableCloudProviders and DisableKubeletCloudCredentialProvider feature flags if you still need this functionality. (#117503@dims) [SIG API Machinery, Cloud Provider and Testing]

  • Introduce new apiserver metric apiserver_flowcontrol_current_inqueue_seats. This metric is analogous to apiserver_flowcontrol_current_inqueue_requests but tracks totals seats as each request can take more than 1 seat. (#119385@andrewsykim) [SIG API Machinery]

  • Kube-proxy don't panic on exit when the Node object changes its PodCIDR (#120375@pegasas) [SIG Network]

  • Kube-proxy will only install the DROP rules for invalid conntrack states if the nf_conntrack_tcp_be_liberal is not set. (#120412@aojea) [SIG Network]

  • Kubeadm: add validation to verify that the CertificateKey is a valid hex encoded AES key (#120064@SataQiu) [SIG Cluster Lifecycle]

  • Kubeadm: promoted feature gate EtcdLearnerMode to beta. Learner mode for joining etcd members is now enabled by default. (#120228@pacoxu) [SIG Cluster Lifecycle]

  • Kubelet exposes latency metrics of different stages of the node startup. (#118568@qiutongs) [SIG Instrumentation, Node and Scalability]

  • Kubernetes is now built with Go 1.21.1 (#120493@cpanato) [SIG Release and Testing]

  • Kubernetes is now built with go 1.21.0 (#118996@cpanato) [SIG Release and Testing]

  • List the pods using as an ephemeral storage volume in "Used by:" part of the output of kubectl describe pvc <PVC> command. (#120427@MaGaroo) [SIG CLI]

  • Migrated the nodevolumelimits scheduler plugin to use contextual logging. (#116884@mengjiao-liu) [SIG Instrumentation, Node, Scheduling, Storage and Testing]

  • Promote ServiceNodePortStaticSubrange to stable and lock to default (#120233@xuzhenglun) [SIG Network]

  • QueueingHint got error in its returning value. If QueueingHint returns error, the scheduler logs the error and treats the event as QueueAfterBackoff so that the Pod wouldn't be stuck in the unschedulable pod pool. (#119290@carlory) [SIG Node, Scheduling and Testing]

  • Remove /livez livezchecks for KMS v1 and v2 to ensure KMS health does not cause kube-apiserver restart. KMS health checks are still in place as a healthz and readiness checks. (#120583@ritazh) [SIG API Machinery, Auth and Testing]

  • The CloudDualStackNodeIPs feature is now beta, meaning that when using an external cloud provider that has been updated to support the feature, you can pass comma-separated dual-stack --node-ips to kubelet and have the cloud provider take both IPs into account. (#120275@danwinship) [SIG API Machinery, Cloud Provider and Network]

  • The Dockerfile for the kubectl image has been updated with the addition of a specific base image and essential utilities (bash and jq). (#119592@rayandas) [SIG CLI, Node, Release and Testing]

  • Use of secret-based service account tokens now adds an authentication.k8s.io/legacy-token-autogenerated-secret or authentication.k8s.io/legacy-token-manual-secret audit annotation containing the name of the secret used. (#118598@yuanchen8911) [SIG Auth, Instrumentation and Testing]

  • Volume_zone plugin will consider beta labels as GA labels during the scheduling process.Therefore, if the values of the labels are the same, PVs with beta labels can also be scheduled to nodes with GA labels. (#118923@AxeZhan) [SIG Scheduling]


  • Added descriptions and examples for the situation of using kubectl rollout restart without specifying a particular deployment. (#120118@Ithrael) [SIG CLI]

Failing Test

  • DRA: when the scheduler has to deallocate a claim after a node became unsuitable for a pod, it might have needed more attempts than really necessary. (#120428@pohly) [SIG Node and Scheduling]
  • E2e framework: retrying after intermittent apiserver failures was fixed in WaitForPodsResponding (#120559@pohly) [SIG Testing]
  • KCM specific args can be passed with /cluster script, without affecting CCM. New variable name: KUBE_CONTROLLER_MANAGER_TEST_ARGS. (#120524@jprzychodzen) [SIG Cloud Provider]
  • This contains the modified windows kubeproxy testcases with mock implementation (#120105@princepereira) [SIG Network and Windows]

Bug or Regression

  • Added a redundant process to remove tracking finalizers from Pods that belong to Jobs. The process kicks in after the control plane marks a Job as finished (#119944@Sharpz7) [SIG Apps]
  • Allow specifying ExternalTrafficPolicy for Services with ExternalIPs. (#119150@tnqn) [SIG API Machinery, Apps, CLI, Cloud Provider, Network, Release and Testing]
  • Exclude nodes from daemonset rolling update if the scheduling constraints are not met. This eliminates the problem of rolling update stuck of daemonset with tolerations. (#119317@mochizuki875) [SIG Apps and Testing]
  • Fix OpenAPI v3 not being cleaned up after deleting APIServices (#120108@tnqn) [SIG API Machinery and Testing]
  • Fix a 1.28 regression in scheduler: a pod with concurrent events could incorrectly get moved to the unschedulable queue where it could got stuck until the next periodic purging after 5 minutes if there was no other event for it. (#120413@pohly) [SIG Scheduling]
  • Fix a bug in cronjob controller where already created jobs may be missing from the status. (#120649@andrewsykim) [SIG Apps]
  • Fix a concurrent map access in TopologyCache's HasPopulatedHints method. (#118189@Miciah) [SIG Apps and Network]
  • Fix kubectl events doesn't filter events by GroupVersion for resource with full name. (#120119@Ithrael) [SIG CLI and Testing]
  • Fixed CEL estimated cost of replace() to handle a zero length replacement string correctly. Previously this would cause the estimated cost to be higher than it should be. (#120097@jpbetz) [SIG API Machinery]
  • Fixed a 1.26 regression scheduling bug by ensuring that preemption is skipped when a PreFilter plugin returns UnschedulableAndUnresolvable (#119778@sanposhiho) [SIG Scheduling and Testing]
  • Fixed a 1.27 scheduling regression that PostFilter plugin may not function if previous PreFilter plugins return Skip (#119769@Huang-Wei) [SIG Scheduling and Testing]
  • Fixed a 1.28 regression around restarting init containers in the right order relative to normal containers (#120281@gjkim42) [SIG Node and Testing]
  • Fixed a regression in default 1.27 configurations in kube-apiserver: fixed the AggregatedDiscoveryEndpoint feature (beta in 1.27+) to successfully fetch discovery information from aggregated API servers that do not check Accept headers when serving the /apis endpoint (#119870@Jefftree) [SIG API Machinery]
  • Fixed an issue where a CronJob could fail to clean up Jobs when the ResourceQuota for Jobs had been reached. (#119776@ASverdlov) [SIG Apps]
  • Fixes a 1.28 regression handling negative index json patches (#120327@liggitt) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
  • Fixes a bug where Services using finalizers may hold onto ClusterIP and/or NodePort allocated resources for longer than expected if the finalizer is removed using the status subresource (#120623@aojea) [SIG Network and Testing]
  • Fixes an issue where StatefulSet might not restart a pod after eviction or node failure. (#120398@aleksandra-malinowska) [SIG Apps]
  • Fixes an issue with the garbagecollection controller registering duplicate event handlers if discovery requests fail. (#117992@liggitt) [SIG API Machinery and Apps]
  • Fixes the bug when images pinned by the container runtime can be garbage collected by kubelet (#119986@ruiwen-zhao) [SIG Node]
  • Fixing issue with incremental id generation for loadbalancer and endpoint in Kubeproxy mock test framework. (#120723@princepereira) [SIG Network and Windows]
  • If a watch with the progressNotify option set is to be created, and the registry hasn't provided a newFunc, return an error. (#120212@p0lyn0mial) [SIG API Machinery]
  • Improved handling of jsonpath expressions for kubectl wait --for. It is now possible to use simple filter expressions which match on a field's content. (#118748@andreaskaris) [SIG CLI and Testing]
  • Incorporating feedback on PR #119341 (#120087@divyasri537) [SIG API Machinery]
  • Kubeadm: Use universal deserializer to decode static pod. (#120549@pacoxu) [SIG Cluster Lifecycle]
  • Kubeadm: fix nil pointer when etcd member is already removed (#119753@pacoxu) [SIG Cluster Lifecycle]
  • Kubeadm: fix the bug that --image-repository flag is missing for some init phase sub-commands (#120072@SataQiu) [SIG Cluster Lifecycle]
  • Kubeadm: improve the logic that checks whether a systemd service exists. (#120514@fengxsong) [SIG Cluster Lifecycle]
  • Kubeadm: print the default component configs for reset and join is now not supported (#119346@chendave) [SIG Cluster Lifecycle]
  • Kubeadm: remove 'system:masters' organization from etcd/healthcheck-client certificate. (#119859@SataQiu) [SIG Cluster Lifecycle]
  • Kubectl prune v2: Switch annotation from contains-group-resources to contains-group-kinds, because this is what we defined in the KEP and is clearer to end-users. Although the functionality is in alpha, we will recognize the prior annotation; this migration support will be removed in beta/GA. (#118942@justinsb) [SIG CLI]
  • Kubectl will not print events if --show-events=false argument is passed to describe PVC subcommand. (#120380@MaGaroo) [SIG CLI]
  • More accurate requeueing in scheduling queue for Pods rejected by the temporal failure (e.g., temporal failure on kube-apiserver.) (#119105@sanposhiho) [SIG Scheduling and Testing]
  • No-op and GC related updates to cluster trust bundles no longer require attest authorization when the ClusterTrustBundleAttest plugin is enabled. (#120779@enj) [SIG Auth]
  • Reintroduce resourcequota.NewMonitor constructor for other consumers (#120777@atiratree) [SIG Apps]
  • Scheduler: Fix field apiVersion is missing from events reported from taint manager (#114095@aimuz) [SIG Apps, Node and Scheduling]
  • Service Controller: update load balancer hosts after node's ProviderID is updated (#120492@cezarygerard) [SIG Cloud Provider and Network]
  • Setting the status.loadBalancer of a Service whose spec.type is not "LoadBalancer" was previously allowed, but any update to the metadata or spec would wipe that field. Setting this field is no longer permitted unless spec.type is "LoadBalancer". In the very unlikely event that this has unexpected impact, you can enable the AllowServiceLBStatusOnNonLB feature gate, which will restore the previous behavior. If you do need to set this, please file an issue with the Kubernetes project to help contributors understand why you need it. (#119789@thockin) [SIG Apps and Testing]
  • Sometimes, the scheduler incorrectly placed a pod in the "unschedulable" queue instead of the "backoff" queue. This happened when some plugin previously declared the pod as "unschedulable" and then in a later attempt encounters some other error. Scheduling of that pod then got delayed by up to five minutes, after which periodic flushing moved the pod back into the "active" queue. (#120334@pohly) [SIG Scheduling]
  • The --bind-address parameter in kube-proxy is misleading, no port is opened with this address. Instead it is translated internally to "nodeIP". The nodeIPs for both families are now taken from the Node object if --bind-address is unspecified or set to the "any" address ( or ::). It is recommended to leave --bind-address unspecified, and in particular avoid to set it to localhost ( or ::1) (#119525@uablrek) [SIG Network and Scalability]

Other (Cleanup or Flake)

  • Add context to "caches populated" log messages. (#119796@sttts) [SIG API Machinery]
  • Add download the cni binary for the corresponding arch in local-up-cluster.sh (#120312@HirazawaUi) [SIG Network and Node]
  • Changes behavior of kube-proxy by allowing to set sysctl values lower than the existing one. (#120448@aroradaman) [SIG Network]
  • Clean up kube-apiserver http logs for impersonated requests. (#119795@sttts) [SIG API Machinery]
  • Dynamic resource allocation: avoid creating a new gRPC connection for every call of prepare/unprepare resource(s) (#118619@TommyStarK) [SIG Node]
  • Fixes an issue where the vsphere cloud provider will not trust a certificate if:
    • The issuer of the certificate is unknown (x509.UnknownAuthorityError)
    • The requested name does not match the set of authorized names (x509.HostnameError)
    • The error surfaced after attempting a connection contains one of the substrings: "certificate is not trusted" or "certificate signed by unknown authority" (#120736@MadhavJivrajani) [SIG Architecture and Cloud Provider]
  • Fixes bug where Adding GroupVersion log line is constantly repeated without any group version changes (#119825@Jefftree) [SIG API Machinery]
  • Generated ResourceClaim names are now more readable because of an additional hyphen before the random suffix (<pod name>-<claim name>-<random suffix> ). (#120336@pohly) [SIG Apps and Node]
  • Improve memory usage of kube-controller-manager by dropping the .metadata.managedFields field that kube-controller-manager doesn't require. (#118455@linxiulei) [SIG API Machinery and Cloud Provider]
  • Kubeadm: remove 'system:masters' organization from apiserver-etcd-client certificate (#120521@SataQiu) [SIG Cluster Lifecycle]
  • Kubeadm: updated warning message when swap space is detected. When swap is active on Linux, kubeadm explains that swap is supported for cgroup v2 only and is beta but disabled by default. (#120198@pacoxu) [SIG Cluster Lifecycle]
  • Makefile and scripts now respect GOTOOLCHAIN and otherwise ensure ./.go-version is used (#120279@BenTheElder) [SIG Release]
  • Optimized NodeUnschedulable Filter to avoid unnecessary calculations (#119399@wackxu) [SIG Scheduling]
  • Previously, the pod name and namespace were eliminated in the event log message. This PR attempts to add the preemptor pod UID in the preemption event message logs for easier debugging and safer transparency. (#119971@kwakubiney) [SIG Scheduling]
  • Promote to conformance a test that verify that Services only forward traffic on the port and protocol specified. (#120069@aojea) [SIG Architecture, Network and Testing]
  • Remove ephemeral container legacy server support for the server versions prior to 1.22 (#119537@ardaguclu) [SIG CLI]
  • Scheduler: handling of unschedulable pods because a ResourceClass is missing is a bit more efficient and no longer relies on periodic retries (#120213@pohly) [SIG Node, Scheduling and Testing]
  • Set the resolution for the job_controller_job_sync_duration_seconds metric from 4ms to 1min (#120577@alculquicondor) [SIG Apps and Instrumentation]
  • Statefulset should wait for new replicas in tests when removing .start.ordinal (#119761@soltysh) [SIG Apps and Testing]
  • The horizontalpodautoscaling and clusterrole-aggregation controllers now assume the autoscaling/v1 and rbac.authorization.k8s.io/v1 APIs are available. If you disable those APIs and do not want to run those controllers, exclude them by passing --controllers=-horizontalpodautoscaling or --controllers=-clusterrole-aggregation to kube-controller-manager. (#117977@liggitt) [SIG API Machinery and Cloud Provider]
  • The metrics controlled by the ComponentSLIs feature-gate and served at /metrics/slis are now GA and unconditionally enabled. The feature-gate will be removed in 1.31. (#120574@logicalhan) [SIG API Machinery, Architecture, Cloud Provider, Instrumentation, Network, Node and Scheduling]
  • Updated CNI plugins to v1.3.0. (#119969@saschagrunert) [SIG Cloud Provider, Node and Testing]
  • Updated cri-tools to v1.28.0. (#119933@saschagrunert) [SIG Cloud Provider]
  • Updated distroless-iptables to use registry.k8s.io/build-image/distroless-iptables:v0.3.1 (#120352@saschagrunert) [SIG Release and Testing]
  • Upgrade coredns to v1.11.1 (#120116@tukwila) [SIG Cloud Provider and Cluster Lifecycle]
  • ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding objects are persisted in etcd using the v1beta1 version. Remove alpha objects or disable the alpha ValidatingAdmissionPolicy feature in a 1.27 server before upgrading to a 1.28 server with the beta feature and API enabled. (#120018@liggitt) [SIG API Machinery and Testing]
  • Yes, kubectl will not support the "/swagger-2.0.0.pb-v1" endpoint that has been long deprecated (#119410@Jefftree) [SIG API Machinery]





Nothing has changed.

Contributors, the CHANGELOG-1.29.md has been bootstrapped with v1.29.0-alpha.1 release notes and you may edit now as needed.

Published by your Kubernetes Release Managers.

Reply all
Reply to author
0 new messages