Kubernetes v1.24.0 has been built and pushed using Golang version 1.18.1.
The release notes have been updated in CHANGELOG-1.24.md, with a pointer to them on GitHub:
filename | sha512 hash |
---|---|
kubernetes.tar.gz | ef3014768ea305a97865ceef486e704083af78841079cad5c589c02711ac12769f5e949cefa2188c80d28a30884a559befc2239e2adfecdca972d1d211a0cb4f |
kubernetes-src.tar.gz | 973ce8ba840125da3d1f45205fd53e82bf5c6736517ff1f27d3ecfb951d413e8db30878194177a76bd73912829a9db3e5948437234f23b269b5e05fd0b29ab75 |
filename | sha512 hash |
---|---|
kubernetes-client-darwin-amd64.tar.gz | acb682468b459bfc51fbc823aa5cef7ed6dbd7441bb3f00e1842c7eb59d15a9c0d0c941700f518e8826d0b3e68d54c9607c94186d624d04402d97ba4b716d384 |
kubernetes-client-darwin-arm64.tar.gz | 066e93a872ea0cef77ab734d7a37ec0ada797da31cd004b29326fc75d4849067ce8a58f11807a4b45643794db4cffa1091820af8c3c476ff9d49c7ce517129b1 |
kubernetes-client-linux-386.tar.gz | 61aba7c6fef077ddf94e6f63958fd4f1c6062460f47098979e8bedf3b1081478f5905cb72e5a6edcb28f46f8e78965dbf4dbc666be49d2a7ad025b39e74e7369 |
kubernetes-client-linux-amd64.tar.gz | 291a4d24a400666ec0e3d91fbaae0605de438abf4878ca81e4e8e923beefa8a434ade3a98e8538f654dfe7d32c2038a1610ca6863efe91d2649ea11c28a20a49 |
kubernetes-client-linux-arm.tar.gz | ac37de6c47ec1e8d8be6e60ef8ca03cd66b6f212b671d619bfa38fe62182c830550d5d6ebe80f4363d2007160ad177c27e7eaab29c2dd5642c4c91f30ce0a347 |
kubernetes-client-linux-arm64.tar.gz | 643b95ff27f275ee7f8999676c141d9a6199d1a60f19ab425be57e6170abf66a5dd4c3712e10db6e11b104e1e41f1fc2b2366507c477b080061cc0ae1fcd788e |
kubernetes-client-linux-ppc64le.tar.gz | 2ce8193c228bb8703d2b42c0e354c489a8c2cf73acfa317c8720f7b1deda68f71ff19c27260ac300d5ab1a70958d082fd4146d09be09505694e65c00b23f103b |
kubernetes-client-linux-s390x.tar.gz | c4124123942178f3371ace92b21e284afd0982d6e2cf8f43db0dc9266a35f5d771d84d829ffa64780213ac7375d0f100463fc990529b28c269527061958098c4 |
kubernetes-client-windows-386.tar.gz | 9973e72958a27b11a02f74147355f8cc4525d283fc148f7a39c2863bb66601839dbfdb27e0da6477dd0afba7afd6a01138ada0d708dda89518d942eb58d1f44c |
kubernetes-client-windows-amd64.tar.gz | 6075bd48040a710395ea6df57f3379984291d90f7d2f75f08d3f24e46abc5b716445dc1d7c26cd877439f213f4efea8e2c03fc362a67db86c2c44326bcc6ff43 |
kubernetes-client-windows-arm64.tar.gz | 17a7d93c0245cdd959845f9c7be95f1f172cae242af09bff03cae161abe828ff2b49cb014b847fff97bdfffd899e78f1a6a0d6c75fd7540c7de5fc9ab7321cef |
filename | sha512 hash |
---|---|
kubernetes-server-linux-amd64.tar.gz | 43a3e68bed60252b588493d07ed85eaa35ff3fec7f9440096fe9af284925f040467d1b31a8948e3035e4738bb689ad6d6fb9208fe77c16b053874d020a3fabd3 |
kubernetes-server-linux-arm.tar.gz | a8ed49f4a6c57b6e0d4a3dc8705fb5d59c8b77e1cc67564bc3825782922bcc2cf431ed762b97f1fd05b4e63d1bf71a3d43f698aae49db4e670b6a7e99384db0c |
kubernetes-server-linux-arm64.tar.gz | dfcf3c4e751b9c174dbc667a87b0f561cb9a0ff4c0503439ca57d4e904db775f19be39605a8f553f9fee6af4e6256fea3eedc71a9cd401aa25d836d722b0f695 |
kubernetes-server-linux-ppc64le.tar.gz | e5d91705c5969a2483314cdb3f80e6b828987036f5fbcf269cce83cbf62b8d73210ec3b469c6e0667432f2f874309622768ecc3df851c9711c4fa51dcaf489e4 |
kubernetes-server-linux-s390x.tar.gz | 1fcbf0e575752cff6a11c8518658454237953227588c7c73efca1a036c73d30246ce57f0b89095b02ccd536b37be2427ddde918763cf63e6a1e6248f38f41689 |
filename | sha512 hash |
---|---|
kubernetes-node-linux-amd64.tar.gz | b6b36973c45986e61d3ab4440b551b145d6279801e388b08f83c7f0369984bd55979504209d3cc70409ff70b200923c680f00302d3410c973c92f657157a7510 |
kubernetes-node-linux-arm.tar.gz | dd57bb241b0468ac78d5eb64770e99a9ad14b2cba345cd762e9552e4e58ea50a92eb037ac6e250983fc5bede37459faee7b9c97eccf7eb3dcc2587eb6f280bae |
kubernetes-node-linux-arm64.tar.gz | b8f339d796644e1bac5e33bb47d14460c012411f28e531a61f3d7b4db939a4722bc8ecafb8193b290b9b4e896b4e52d8332a8e4b010f327b896616614b8695fb |
kubernetes-node-linux-ppc64le.tar.gz | df52fed23afa5bebaaea2fc18482fc718e6e6643d931416cb62f1efbc8bf2e9ba82f6d3372f82b18c8d0562378dfa82aae621e13cb7a1a80a769d843f9279e1e |
kubernetes-node-linux-s390x.tar.gz | dc79e44a9a2879d1dea8c22b2b21965c7a0279e54d8c54af899eecb4f14da0b56eb52e11812e663ed9edc94410b2771f38118bf2936aa15bf66011ee04fad03a |
kubernetes-node-windows-amd64.tar.gz | 4275d4c6ab19433398adf8122923b0dd0a8e1432f966a38f40c9547255e69c952b951e76273b8f3220c777504905480a457a7f0b1a1ed64769f9db0cc9f1b716 |
All container images are available as manifest lists and support the described architectures. It is also possible to pull a specific architecture directly by adding the "-$ARCH" suffix to the container image name.
After its deprecation in v1.20, the dockershim component has been removed from the kubelet. From v1.24 onwards, you will need to either use one of the other supported runtimes (such as containerd or CRI-O) or use cri-dockerd if you are relying on Docker Engine as your container runtime. For more information about ensuring your cluster is ready for this removal, please see this guide.
New beta APIs will not be enabled in clusters by default. Existing beta APIs and new versions of existing beta APIs, will continue to be enabled by default.
Release artifacts are signed using cosign signatures and there is experimental support for verifying image signatures. Signing and verification of release artifacts is part of increasing software supply chain security for the Kubernetes release process.
Kubernetes 1.24 offers beta support for publishing its APIs in the OpenAPI v3 format.
Storage capacity tracking supports exposing currently available storage capacity via CSIStorageCapacity objects and enhances scheduling of pods that use CSI volumes with late binding.
Volume expansion adds support for resizing existing persistent volumes.
This feature adds a new option to PriorityClasses, which can enable or disable pod preemption.
There is work under way to migrate the internals of in-tree storage plugins to call out to CSI Plugins, while maintaining the original API. The Azure Disk and OpenStack Cinder plugins have both been migrated.
With Kubernetes 1.24, the gRPC probes functionality has entered beta and is available by default. You can now configure startup, liveness, and readiness probes for your gRPC app natively within Kubernetes, without exposing an HTTP endpoint or using an extra executable.
Originally released as Alpha in Kubernetes 1.20, the kubelet's support for image credential providers has now graduated to Beta. This allows the kubelet to dynamically retrieve credentials for a container image registry using exec plugins, rather than storing credentials on the node's filesystem.
Kubernetes 1.24 has introduced contextual logging that enables the caller of a function to control all aspects of logging (output formatting, verbosity, additional values and names).
Kubernetes 1.24 introduced a new opt-in feature that allows you to soft-reserve a range for static IP address assignments to Services. With the manual enablement of this feature, the cluster will prefer automatic assignment from the pool of Service IP addresses thereby reducing the risk of collision.
A Service ClusterIP
can be assigned:
Service ClusterIP
are unique, hence, trying to create a Service with a ClusterIP
that has already been allocated will return an error.
Not-ready
state when credentials for vCenter stored in a secret and Zones feature is in use. Zone labels setup moved to KCM component, kubelet skips this step during startup in such case. If credentials stored in cloud-provider config file as plaintext current behaviour does not change and no action required. For proper functioning kube-system:vsphere-legacy-cloud-provider
should be allowed to update node object if vCenter credentials stored in secret and Zone feature used. (#101028, @lobziik)LegacyServiceAccountTokenNoAutoGeneration
feature gate is beta, and enabled by default. When enabled, Secret API objects containing service account tokens are no longer auto-generated for every ServiceAccount. Use the TokenRequest API to acquire service account tokens, or if a non-expiring token is required, create a Secret API object for the token controller to populate with a service account token by following this guide. (#108309, @zshihang)topologyKey
is not node-level. Revisit the node affinity and/or pod selector in the
topology spread constraints to avoid this scenario. (#107009, @kerthcet)--experimental-check-node-capabilities-before-mount
. With CSI now GA, there is a better alternative. Remove any use of --experimental-check-node-capabilities-before-mount
from your kubelet scripts or manifests. (#104732, @mengjiao-liu)kubeadm.k8s.io/v1beta2
has been deprecated and will be removed in a future release, possibly in 3 releases (one year). You should start using kubeadm.k8s.io/v1beta3
for new clusters. To migrate your old configuration files on disk you can use the kubeadm config migrate
command. (#107013, @pacoxu)unix:///var/run/containerd/containerd.sock
, Windows: npipe:////./pipe/containerd-containerd
) instead of the one for Docker. If the Init|JoinConfiguration.nodeRegistration.criSocket
field is empty during cluster creation and multiple sockets are found on the host always throw an error and ask the user to specify which one to use by setting the value in the field. Make sure you update any kubeadm configuration files on disk, to not include the dockershim socket unless you are still using kubelet version < 1.24 with kubeadm >= 1.24. Remove the DockerValidor and ServiceCheck for the docker
service from kubeadm preflight. Docker is no longer special cased during host validation and ideally this task should be done in the now external cri-dockerd project where the importance of the compatibility matters. Use crictl
for all communication with CRI sockets for actions like pulling images and obtaining a list of running containers instead of using the docker CLI in the case of Docker. (#107317, @neolit123)csiMigrationRBD
where it should have been CSIMigrationRBD
to be in parity with other migration plugins. This release correct the same and keep it as CSIMigrationRBD
.
users who have configured this feature gate as csiMigrationRBD
has to reconfigure the same to CSIMigrationRBD
from this release. (#107554, @humblec)second stage
of the plan to migrate kubeadm away from the usage of the word master
in labels and taints. For new clusters, the label node-role.kubernetes.io/master
will no longer be added to control plane nodes, only the label node-role.kubernetes.io/control-plane
will be added. For clusters that are being upgraded to 1.24 with kubeadm upgrade apply
, the command will remove the label node-role.kubernetes.io/master
from existing control plane nodes. For new clusters, both the old taint node-role.kubernetes.io/master:NoSchedule
and new taint node-role.kubernetes.io/control-plane:NoSchedule
will be added to control plane nodes. In release 1.20 (first stage
), a release note instructed to preemptively tolerate the new taint. For clusters that are being upgraded to 1.24 with kubeadm upgrade apply
, the command will add the new taint node-role.kubernetes.io/control-plane:NoSchedule
to existing control plane nodes. Please adapt your infrastructure to these changes. In 1.25 the old taint node-role.kubernetes.io/master:NoSchedule
will be removed. (#107533, @neolit123)csiMigrationRBD
where it should have been CSIMigrationRBD
to be in parity with other migration plugins. This release correct the same and keep it as CSIMigrationRBD
.
users who have configured this feature gate as csiMigrationRBD
has to reconfigure the same to CSIMigrationRBD
from this release. (#107554, @humblec)Deprecated Service.Spec.LoadBalancerIP
. This field was under-specified and its meaning varies across implementations. As of Kubernetes v1.24, users are encouraged to use implementation-specific annotations when available. This field may be removed in a future API version. (#107235, @uablrek)
Kube-apiserver: the --master-count
flag and --endpoint-reconciler-type=master-count
reconciler are deprecated in favor of the lease reconciler (#108062, @aojea)
Kube-apiserver: the insecure address flags --address
, --insecure-bind-address
, --port
and --insecure-port
(inert since 1.20) are removed (#106859, @knight42)
Kubeadm: graduated the UnversionedKubeletConfigMap
feature gate to Beta and enabled the feature by default. This implies that 1) for new clusters kubeadm will start using the kube-system/kubelet-config
naming scheme for the kubelet ConfigMap and RBAC rules, instead of the legacy kubelet-config-x.yy
naming. 2) during upgrade, kubeadm will only write the new scheme ConfigMap and RBAC objects. To disable the feature you can pass UnversionedKubeletConfigMap: false
in the kubeadm config for new clusters. For upgrade on existing clusters you can also override the behavior by patching the ClusterConfiguration object in kube-system/kubeadm-config
. More details in the associated KEP. (#108027, @neolit123)
Remove tolerate-unready-endpoints
annotation in Service deprecated from 1.11, use Service.spec.publishNotReadyAddresses
instead. (#108020, @tossmilestone)
Remove deprecated feature gates ValidateProxyRedirects
and StreamingProxyRedirects
(#106830, @pacoxu)
Remove insecure serving configuration from cloud-provider package, which is consumed by cloud-controller-managers. (#108953, @nckturner)
The --pod-infra-container-image
kubelet flag is deprecated and will be removed in future releases (#108045, @hakman)
The client.authentication.k8s.io/v1alpha1
ExecCredential has been removed. If you are using a client-go credential plugin that relies on the v1alpha1 API please contact the distributor of your plugin for instructions on how to migrate to the v1 API. (#108616, @margocrawf)
The node.k8s.io/v1alpha1
RuntimeClass API is no longer served. Use the node.k8s.io/v1
API version, available since v1.20 (#103061, @SergeyKanzhelev)
The cluster addon for dashboard was removed. To install dashboard, see here. (#107481, @shu-mutou)
The in-tree Azure plugin has been deprecated. The Azure kubelogin plugin serves as an out-of-tree replacement via the kubectl/client-go credential plugin mechanism. Users will now see a warning in the logs regarding this deprecation. (#107904, @sabbey37)
The insecure address flags --address
and --port
in kube-controller-manager have had no effect since v1.20 and are removed in v1.24. (#106860, @knight42)
The metadata.clusterName field is deprecated. This field has always been unwritable and always blank, but its presence is confusing, so we will remove it next release. Out of an abundance of caution, this release we have merely changed the name in the go struct to ensure any accidental client uses are found before complete removal. (#108717, @lavalamp)
VSphere releases less than 7.0u2 are deprecated as of v1.24. Please consider upgrading vSphere to 7.0u2 or above. vSphere CSI Driver requires minimum vSphere 7.0u2.
General Support for vSphere 6.7 will end on October 15, 2022. vSphere 6.7 Update 3 is deprecated in Kubernetes v1.24. Customers are recommended to upgrade vSphere (both ESXi and vCenter) to 7.0u2 or above. vSphere CSI Driver 2.2.3 and higher supports CSI Migration.
Support for these deprecations will be available till October 15, 2022. (#109089, @deepakkinni)
--forward-healthcheck-vip
, if specified as true, health check traffic whose destination is service VIP will be forwarded to kube-proxy's healthcheck service. --root-hnsendpoint-name
specifies the name of the hns endpoint for the root network namespace. This option enables the pass-through load balancers like Google's GCLB to correctly health check the backend services. Without this change, the health check packets is dropped, and Windows node will be considered to be unhealthy by those load balancers. (#99287, @anfernee)webhook_fail_open_count
to monitor webhooks that fail to open. (#107171, @ltagliamonte-dd)InterfaceNamePrefix
and BridgeInterface
as arguments to --detect-local-mode
option and also introduces a new optional --pod-interface-name-prefix
and --pod-bridge-interface
flags to kube-proxy. (#95400, @tssurya)oldSelf
. (#108073, @benluddy)JSONSchemaProps.XValidations
. (#107956, @benluddy)oldSelf
on a part of the schema that does not support it. (#108013, @benluddy)fieldValidation=Strict
consistently require apiVersion
and kind
, matching non-strict requests (#109019, @liggitt)DefaultPodTopologySpread
is graduated to GA (#108278, @kerthcet)NonPreemptingPriority
is graduated to GA (#107432, @denkensk)PodOverhead
is graduated to GA (#108441, @pacoxu)strategic merge patch
-type API requests for the selector
field. Prior to 1.21, these requests would merge matchLabels
content and replace matchExpressions
content. In 1.21, patch requests touching the selector
field started replacing the entire selector. This is consistent with server-side apply and the v1 PodDisruptionBudget behavior, but should not have been changed for v1beta1. (#108138, @liggitt)--audit-log-version
and --audit-webhook-version
now only support the default value of audit.k8s.io/v1
. The v1alpha1 and v1beta1 audit log versions, deprecated since 1.13, have been removed. (#108092, @carlory)metadata.selfLink
field can no longer be populated by kube-apiserver; it was deprecated in 1.16 and has not been populated by default since 1.20+. (#107527, @wojtek-t)PodAffinityNamespaceSelector
is locked and will be removed in 1.26. (#108136, @ahg-g)evictions_number
to evictions_total
and mark it as stable. The original evictions_number
metrics name is marked as "Deprecated" and has been removed in kubernetes 1.23 . (#106366, @cyclinder)SuspendJob
is locked and will be removed in 1.26. (#108129, @ahg-g)spec.expirationSeconds
API field has graduated to GA. The CSRDuration
feature gate for the field is now unconditionally enabled and will be removed in 1.26. (#108782, @cfryanr)ServerSideFieldValidation
feature has graduated to beta and is now enabled by default. Kubectl 1.24 and newer will use server-side validation instead of client-side validation when writing to API servers with the feature enabled. (#108889, @kevindelgado)ServiceLBNodePortControl
feature has graduated to GA. The feature gate will be removed in 1.26. (#107027, @uablrek)DynamicKubeletConfig
has been removed from the kubelet. (#106932, @SergeyKanzhelev)timeZone
field as part of the CronJob spec to support running cron jobs in a specific time zone. (#108032, @deejross)topologySpreadConstraints
includes minDomains
field to limit the minimum number of topology domains. (#107674, @sanposhiho)A new Priority and Fairness metric 'apiserver_flowcontrol_work_estimate_seats_samples' has been added that tracks the estimated seats associated with a request. (#106628, @tkashem)
Add a deprecated cmd flag for the time interval between flushing pods from unschedulable queue to active queue or backoff queue. (#108017, @denkensk)
Add one metrics(kubelet_volume_stats_health_abnormal
) of volume health state to kubelet (#105585, @fengzixu)
Add the metric container_oom_events_total
to kubelet's cAdvisor metric endpoint. (#108004, @jonkerj)
Added SetTransform
to SharedInformer
to allow users to transform objects before they are stored. (#107507, @alexzielenski)
Added a proxy-url
flag into kubectl config set-cluster
. (#105566, @ardaguclu)
Added a metric for measuring end-to-end volume mount timing. (#107006, @gnufied)
Added a new Priority and Fairness metric apiserver_flowcontrol_request_dispatch_no_accommodation_total
to track the number of times a request dispatch attempt results in a no-accommodation status due to lack of available seats. (#106629, @tkashem)
Added a path /header?key=
to agnhost netexec
allowing one to view what the header value is of the incoming request.
Ex:
something``` ([#107796](https://github.com/kubernetes/kubernetes/pull/107796), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu))
Added completion for kubectl config set-context
. (#106739, @kebe7jun)
Added field add_ambient_capabilities
to the Capabilities message in the CRI-API. (#104620, @vinayakankugoyal)
Added label selector flag to all kubectl rollout
commands. (#99758, @aramperes)
Added more message for no PodSandbox container. (#107116, @yxxhero)
Added prune flag into diff
command to simulate apply --prune
. (#105164, @ardaguclu)
Added support for btrfs
resizing (#108561, @RomanBednar)
Added support for kubectl commands (kubectl exec
and kubectl port-forward
) via a SOCKS5 proxy. (#105632, @xens)
Adds OpenAPIV3SchemaInterface
to DiscoveryClient
and its variants for fetching OpenAPI v3 schema documents. (#108992, @alexzielenski)
Allow kubectl to manage resources by filename patterns without the shell expanding it first (#102265, @danielrodriguez)
An alpha flag --subresource
is added to get, patch, edit replace kubectl commands to fetch and update status and scale subresources. (#99556, @nikhita)
Apiextensions_openapi_v3_regeneration_count metric (alpha) will be emitted for OpenAPI V3. (#109128, @Jefftree)
Apply ProxyTerminatingEndpoints to all traffic policies (external, internal, cluster, local). (#108691, @andrewsykim)
CEL regex patterns in x-kubernetes-valiation rules are compiled when CRDs are created/updated if the pattern is provided as a string constant in the expression. Any regex compile errors are reported as a CRD create/update validation error. (#108617, @jpbetz)
CRD x-kubernetes-validations
rules now support the CEL functions: isSorted
, sum
, min
, max
, indexOf
, lastIndexOf
, find
and findAll
. (#108312, @jpbetz)
Changes the kubectl --validate
flag from a bool to a string that accepts the values {true, strict, warn, false, ignore}
Client-go metrics: change bucket distribution for rest_client_request_duration_seconds
and rest_client_rate_limiter_duration_seconds
from [0.001, 0.002, 0.004, 0.008, 0.016, 0.032, 0.064, 0.128, 0.256, 0.512] to [0.005, 0.025, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0, 15.0, 30.0, 60.0}] (#106911, @aojea)
Client-go: add new histogram metric to record the size of the requests and responses. (#108296, @aojea)
CycleState is now optimized for "write once and read many times". (#108724, @sanposhiho)
Enabled beta feature HonorPVReclaimPolicy by default. (#109035, @deepakkinni)
Env var for additional cli flags used in the csi-proxy binary when a Windows nodepool is created with kube-up.sh
(#107806, @mauriciopoppe)
Feature of PreferNominatedNode
is graduated to GA. (#106619, @chendave)
In text format, log messages that previously used quoting to prevent multi-line output (for example, text="some "quotation", a\nline break") will now be printed with more readable multi-line output without the escape sequences. (#107103, @pohly)
Increase default value of discovery cache TTL for kubectl to 6 hours. (#107141, @mk46)
Introduce policy to allow the HPA to consume the external.metrics.k8s.io
API group. (#104244, @dgrisonnet)
Kube-apiserver: Subresources such as status
and scale
now support tabular output content types. (#103516, @ykakarap)
Kube-apiserver: when merging lists, Server Side Apply now prefers the order of the submitted request instead of the existing persisted object. (#107565, @jiahuif)
Kubeadm: added support for dry running kubeadm reset
. The new flag kubeadm reset --dry-run
is similar to the existing flag for kubeadm init/join/upgrade
and allows you to see what changes would be applied. (#107512, @SataQiu)
Kubeadm: added the flag --experimental-initial-corrupt-check
to etcd static Pod manifests to ensure etcd member data consistency (#109074, @neolit123)
Kubeadm: better surface errors during kubeadm upgrade
when waiting for the kubelet to restart static pods on control plane nodes (#108315, @Monokaix)
Kubeadm: improve the strict parsing of user YAML/JSON configuration files. Next to printing warnings for unknown and duplicate fields (current state), also print warnings for fields with incorrect case sensitivity - e.g. controlPlaneEndpoint
(valid), ControlPlaneEndpoint
(invalid). Instead of only printing warnings during init
and join
also print warnings when downloading the ClusterConfiguration, KubeletConfiguration or KubeProxyConfiguration objects from the cluster. This can be useful if the user has patched these objects in their respective ConfigMaps with mistakes. (#107725, @neolit123)
Kubectl now supports shell completion for the / format for specifying resources.
kubectl now provides shell completion for container names following the --container/-c
flag of the exec
command.
kubectl's shell completion now suggests resource types for commands that only apply to pods. (#108493, @marckhouzam)
Kubelet: add kubelet_volume_metric_collection_duration_seconds
metrics for volume disk usage calculation duration (#107201, @pacoxu)
Kubelet: the following dockershim related flags are also removed along with dockershim --experimental-dockershim-root-directory
, --docker-endpoint
, --image-pull-progress-deadline
, --network-plugin
, --cni-conf-dir
, --cni-bin-dir
, --cni-cache-dir
, --network-plugin-mtu
. (#106907, @cyclinder)
Kubernetes 1.24 bumped version of golang it is compiled with to go1.18, which introduced significant changes to its garbage collection algorithm. As a result, we observed an increase in memory usage for kube-apiserver in larger an heavily loaded clusters up to ~25% (with the benefit of API call latencies drop by up to 10x on 99th percentiles). If the memory increase is not acceptable for you you can mitigate by setting GOGC env variable (for our tests using GOGC=63 brings memory usage back to original value, although the exact value may depend on usage patterns on your cluster). (#108870, @dims)
Kubernetes 1.24 is built with go1.18, which will no longer validate certificates signed with a SHA-1 hash algorithm by default. See https://golang.org/doc/go1.18#sha1 for more details. If you are using certificates like this in admission or conversion (#109024, @stlaz)
Kubernetes in now built with go1.18rc1 (#107105, @justaugustus)
Kubernetes is now built with Golang 1.17.4 (#106833, @cpanato)
Kubernetes is now built with Golang 1.17.5. (#106956, @cpanato)
Kubernetes is now built with Golang 1.17.6. (#107612, @palnabarun)
Kubernetes is now built with Golang 1.17.7 (#108091, @xmudrii)
Kubernetes is now built with Golang 1.18.1 (#109461, @cpanato)
Leader Migration is now GA. All new configuration files onwards should use version v1. (#109072, @jiahuif)
Mark AzureDisk CSI migration as GA (#107681, @andyzhangx)
Moving MixedProtocolLBService from alpha to beta (#109213, @bridgetkromhout)
New "field_validation_request_duration_seconds" metric, measures how long requests take, indicating the value of the fieldValidation query parameter and whether or not server-side field validation is enabled on the apiserver (#109120, @kevindelgado)
New feature gate, ServiceIPStaticSubrange, to enable the new strategy in the Service IP allocators, so the IP range is subdivided and dynamic allocated ClusterIP addresses for Services are allocated preferently from the upper range. (#106792, @aojea)
OpenAPI definitions served by kube-apiserver now include enum types by default. (#108898, @jiahuif)
OpenStack Cinder CSI migration is now GA and switched on by default, Cinder CSI driver must be installed on clusters on OpenStack for Cinder volumes to work (has been since v1.21). (#107462, @dims)
PreFilter extension in the scheduler framework now returns not only status but also PreFilterResult (#108648, @ahg-g)
Promoted graceful shutdown based on pod priority to beta (#107986, @wzshiming)
Removed feature gate SetHostnameAsFQDN
. (#108038, @mengjiao-liu)
Removed kube-scheduler insecure flags. You can use --bind-address
and --secure-port
instead. (#106865, @jonyhy96)
Removed the ImmutableEphemeralVolumes
feature gate. (#107152, @mengjiao-liu)
Set PodMaxUnschedulableQDuration
as 5 min. (#108761, @denkensk)
Support in-tree PV deletion protection finalizer. (#108400, @deepakkinni)
The .spec.loadBalancerClass
field for Services is now generally available. (#107979, @XudongLiuHarold)
The NamespaceDefaultLabelName
feature gate, GA since v1.22, is now removed. (#106838, @mengjiao-liu)
The kubectl logs
will now warn and default to the first container in a pod. This new behavior brings it in line with kubectl exec
. (#105964, @kidlj)
The v1
version of LeaderMigrationConfiguration
supports only leases
API for leader election. To use formerly supported mechanisms, please continue using v1beta1
. (#108016, @jiahuif)
The kubelet now creates an iptables chain named KUBE-IPTABLES-HINT
in
the mangle
table. Containerized components that need to modify iptables
rules in the host network namespace can use the existence of this chain
to more-reliably determine whether the system is using iptables-legacy or
iptables-nft. (#109059, @danwinship)
The output of kubectl describe ingress
now includes an IngressClass name if available. (#107921, @mpuckett159)
The scheduler prints info logs when the extender returned an error. (--v>5
) (#107974, @sanposhiho)
The script cluster/gce/gci/configure.sh
now supports downloading crictl
on ARM64 nodes (#108034, @tstapler)
Turn on CSIMigrationAzureFile
by default on 1.24 (#105070, @andyzhangx)
Update the k8s.io/system-validators library to v1.7.0 (#108988, @neolit123)
Updated golang.org/x/net to v0.0.0-20211209124913-491a49abca63. (#106949, @cpanato)
Updates kubectl kustomize
and kubectl apply -k
to Kustomize v4.5.4 (#108994, @KnVerey)
When invoked with -list-images
, the e2e.test
binary now also lists the images that might be needed for storage tests. (#108458, @pohly)
kubectl config delete-user
now supports completion (#107142, @dimbleby)
kubectl create token
can now be used to request a service account token, and permission to request service account tokens is added to the edit
and admin
RBAC roles (#107880, @liggitt)
kubectl version
now includes information on the embedded version of Kustomize (#108817, @KnVerey)
--node-ip
will now be preferred for when determining the node's primary IP and using the external cloud provider (CCM). (#107750, @stephenfin)kubelet_volume_stats_health_abnormal
) of volume health state to kubelet (#108758, @fengzixu)type
to apiserver_flowcontrol_request_execution_seconds
metric - it has the following values: - 'regular': indicates that it is a non long running request - 'watch': indicates that it is a watch request. (#105517, @tkashem)-args $prog_args
in KUBE_TEST_ARGS, when doing make test-integration
. (#107516, @MikeSpreitzer)0.0.0.0/::
when handling a proxy subresource request. (#107402, @anguslees)sigs.k8s.io/apiserver-network-proxy/konnectiv...@v0.0.30
to fix a goroutine leak in kube-apiserver when using egress selctor with the gRPC mode. (#108437, @andrewsykim)NodeExpand
on all nodes in case of RWX volumes (#108693, @gnufied)ResourceVersionMatch
set would fail once paging is kicked in. (#107311, @fasaxc)fsGroup
to be applied for CSI Inline Volumes (#108662, @dobsonj)--retries
functionality for negative values in kubectl cp
(#108748, @atiratree)azureDisk
parameter lowercase translation issue. (#107429, @andyzhangx)azureFile
volumeID
collision issue in CSI migration. (#107575, @andyzhangx).status.nominatedNodeName
is not cleared properly, and thus over-occupied system resources. (#106816, @Huang-Wei)/healthz
request times out. (#107034, @benluddy)EndpointSlice
update could cause node name information to be dropped from endpoints that were not updated. (#108198, @liggitt)create --dry-run
: uid and, if generateName was used, name. (#107088, @joejulian)kubectl create secret
command. (#107221, @rikatz)x-kubernetes-preserve-unknown-fields: true
. (#107688, @liggitt)Service
objects that have not been modified since 1.19 can be rejected with an incorrect spec.clusterIPs: Required value
error. (#107847, @thockin)OutOfCpu
errors if they were rapidly scheduled after other pods were reported as complete in the API. The Kubelet now waits to report the phase of a pod as terminal in the API until all running containers are guaranteed to have stopped and no new containers can be started. Short-lived pods may take slightly longer (~1s) to report Succeeded or Failed after this change. (#108366, @smarterclayton)TopologyManager
for ensuring aligned allocations on machines with more than 2 NUMA nodes (#108052, @klueska)--nodeport-addresses
is empty. (#107413, @tnqn)--context
flag is specified with a value that contains a colon. (#107439, @brianpursley)RunCordonOrUncordon
. (#105297, @jackfrancis)PodTopologySpread
scores to offer better scoring when spreading a low number of pods. (#107384, @sanposhiho)apf_fd
from server logs which could contain data identifying the requesting user (#108631, @jupblb)-v=9
rather than -v=5
. (#108224, @danwinship)certs check-expiration
command to not require the existence of the cluster CA key (ca.key file) when checking the expiration of managed certificates in kubeconfig files. (#106854, @neolit123)certs check-expiration
command, treat the etcd CA as external if there is a missing etcd CA key file (etcd/ca.key) and perform the proper validation on certificates signed by the etcd CA. Additionally, make sure that the CA for all entries in the output table is included - for both certificates on disk and in kubeconfig files. (#106891, @neolit123)KubeletConfiguration
resolvConf
field value does not match /run/systemd/resolve/resolv.conf
(#107785, @chendave)kubeadm init --dry-run
with certificate authority files (ca.key
/ ca.crt
) present in /etc/kubernetes/pki
) (#108410, @Haleygo)kubeadm certs generate-csr
command does not remove duplicated SANs (#107982, @SataQiu)kubectl list
-> unknown command
) that were printed as log message with escaped line breaks instead of a multi-line plain text, making the error hard to read. (#107044, @pohly)"v":0
in JSON output although they were debug messages with a higher verbosity. (#106978, @pohly)resizeStatus
and allocatedResources
when the RecoverVolumeExpansionFailure
feature is enabled. (#107686, @gnufied)--service-account-extend-token-expiration
is true and the requested token audiences are empty or exactly match all values for --api-audiences
. (#105954, @jyotimahapatra)Endpoints
and EndpointSlice
updates caused by Pod ResourceVersion
change (#108078, @tnqn)<default>
as the value in case kubectl describe ingress shows default-backend:80
when no default backend is present (#108506, @jlsong01)rest_client_request_duration_seconds
and rest_client_rate_limiter_duration_seconds
metrics with a host label to prevent cardinality explosions and keep only the useful information. This is a breaking change required for security reasons. (#106539, @dgrisonnet)NumPDBViolations
info of nodes, when HTTPExtender ProcessPreemption
. This info will be used in subsequent filtering steps - pickOneNodeForPreemption
(#105853, @caden2016)spec.internalTrafficPolicy
is no longer defaulted for Services when the type is ExternalName
. The field is also dropped on read when the Service type is ExternalName
. (#104846, @andrewsykim)ServerSideFieldValidation
feature has been reverted to alpha for 1.24. (#109271, @liggitt)TopologyAwareHints
feature gate is now enabled by default. This will allow users to opt-in to Topology Aware Hints by setting the service.kubernetes.io/topology-aware-hints
on a Service. This will not affect any Services without that annotation set. (#108747, @robscott)--really-crash-for-testing
was removed. (#101719, @SergeyKanzhelev)apiserver
, if configured to reconcile the kubernetes.default
service endpoints, checks if the configured Service IP range matches the apiserver public address IP family, and fails to start if not. (#106721, @aojea)kubectl version
now fails when given extra arguments. (#107967, @jlsong01)build/dependencies.yaml
: remove the dependency on Docker. With the dockershim removal, core Kubernetes no longer
has to track the latest validated version of Docker.' (#107607, @neolit123)--experimental-encryption-provider-config
flag is now removed. Adapt your machinery to use the --encryption-provider-config
flag that is available since v1.13. (#108423, @ialidzhikov)--target-ram-mb
flag is now removed. (#108457, @ialidzhikov)kubectl plugin list
command. (#106600, @bergerhoffer)k8s.io/apimachinery/util/clock
. Please use k8s.io/utils/clock
instead. (#106850, @MadhavJivrajani)kube-root-ca.crt
to be populated in namespaces for use with projected service account tokens, reducing delays starting those test pods and errors in the logs. (#107763, @smarterclayton)NewDefaultKubectlCommand
. (#107131, @jonnylangefeld)best
non-preferred hint in the TopologyManager (#108154, @klueska)net.ipv4.conf.all.route_localnet=1
if no IPv4 loopback address is selected by the nodePortAddresses
configuration parameter. (#107684, @aojea){Init|Join}Configuration.nodeRegistration.criSocket
value in the kubeadm configuration to be equal to unix:///var/run/dockershim.sock
on Unix or npipe:////./pipe/dockershim
on Windows. If kubelet version >=1.24 is on the host, kubeadm >=1.24 will treat all container runtimes as "remote" using the kubelet flags --container-runtime=remote --container-runtime-endpoint=scheme://some/path
. The special management for kubelet <1.24 will be removed in kubeadm 1.25. (#106973, @neolit123)kubeadm init/join
always use a URL scheme (unix:// on Linux and npipe:// on Windows) when passing a value to the --container-runtime-endpoint
kubelet flag. This flag's value is taken from the kubeadm configuration criSocket
field or the --cri-socket
CLI flag. Automatically add a missing URL scheme to the user configuration in memory, but warn them that they should also update their configuration on disk manually. During kubeadm upgrade apply/node
mutate the /var/lib/kubelet/kubeadm-flags.env
file on disk and the kubeadm.alpha.kubernetes.io/cri-socket
annotation Node object if needed. These automatic actions are temporary and will be removed in a future release. In the future the kubelet may not support CRI endpoints without an URL scheme. (#107295, @neolit123)IPv6DualStack
feature gate. The feature has been GA and locked to enabled since 1.23. (#106648, @calvin0327)output/v1alpha1
API used for machine readable output by some kubeadm commands. In 1.23 kubeadm started using the newer version output/v1alpha2
for the same purpose. (#107468, @neolit123)ca.crt
can only contain one certificate. If there is more than one certificate in the ca.crt
file, kubeadm will pick the first one by default. (#107327, @SataQiu)-v=99
and not -v=6
(#108053, @eddiezane)--dry-run
, --dry-run=true
, and --dry-run=false
for compatibility with pre-1.23 invocations. (#107003, @julianvmodesto)invalid.registry.k8s.io/invalid
instead invalid.com/invalid
for test that use an invalid registry. (#107455, @aojea)--container-runtime-endpoint
and --image-service-endpoint
CLI flags as stable. (#106954, @saschagrunert)volume/csi/csi-client.go
logs to structured logging. (#99441, @CKchen0726)RuntimeClass
feature gate" if present. Note that this feature has been on by default since 1.14 and was GA'ed in 1.20. (#106882, @cyclinder)--serviceaccount
, --hostport
, --requests
and --limits
from kubectl run. (#108820, @mozillazg)node-expansion
between node-stage
and node-publish
(#108614, @gnufied)generator
and container-port
flags (#106824, @lauchokyip)--non-masquerade-cidr
deprecated CLI flag (#107096, @hakman)--deserialization-cache-size
flag is now removed. (#108448, @ialidzhikov)--container-runtime
kubelet flag is deprecated and will be removed in future releases. (#107094, @adisky)WarningHeaders
feature gate that is GA since v1.22 is unconditionally enabled, and can no longer be specified via the --feature-gates
argument. (#108394, @ialidzhikov)e2e.test
binary supports a new --kubelet-root
parameter to override the default /var/lib/kubelet
path. CSI storage tests use this. (#108253, @pohly)runAllFilters
is removed. (#108829, @kerthcet)--max-resource-write-bytes
& --json-patch-max-copy-bytes
string. (#106875, @warmchang)kube-addon-manager
image version is bumped to 9.1.6 (#108341, @zshihang) --short
. Users requiring full output should use --output=yaml|json
instead. (#108987, @soltysh)
Contributors, the
CHANGELOG-1.24.md has been bootstrapped with
v1.24.0 release notes and you may edit now as needed.
Published by your
Kubernetes Release
Managers.