Week Ending November 13, 2022

Developer News

Two new security fixes last week: CVE-2022-3162 allowed custom resources in the same API group eavesdrop on each other; CVE-2022-3294 allowed users to proxy via the API server to addresses they shouldn’t have had access to. Both issues are fixed in the most recent patch releases and upgrading is recommended.

Release Schedule

Next Deadline: Test Freeze / Docs Ready for Review / Release Blog Ready, November 15th

We are in Code Freeze now, with the countdown of deadlines to final release on December 6th. This week is the trifecta of: have your feature docs ready for review, get the release blog ready for review, and freeze all test work until the final release (except bugfixes).

Patch releases for 1.25.4, 1.24.8, 1.23.14, and 1.22.16 came out last week, including fixes for both of the CVEs. This will be the last patch release for 1.22, so upgrade your cluster now.

Note that the next set of patch releases have been rescheduled to December 7th, with cherry-picks due on the 2nd. SIG-Release has proposed that we stop making release candidates of patch releases given the effort and image cost, so comment there if you have opinions on this change.

Featured PRs

#113599: Aggregated discovery client

The discovery APIs let clients query what APIs exist on the API server. This is used for everything from kubectl shell autocomplete to version negotiation. When Kubernetes was young and there were only a few dozen API types, querying each separately worked well enough, but the core APIs have grown and custom APIs (operators) have become widespread. Some single operators like Crossplane can register more API types than all of Kubernetes core. Suffice to say, this created scaling and performance challenges. We’ve used client-side caching to help mitigate things but this improved API (and a related server-side change), we have a more complete solution. Now kube-apiserver can cache discovery metadata itself and clients can request it all in one go. For the initial testing, this is being added to clients in a relatively drop-in fashion to mirror the existing caching client but in the future the plan is to remove the need for client-side caching entirely when aggregated discovery is in use. This should improve both client and server performance and simplify the code, an impressive win for the API team.

If you work on a non-Go client library, definitely check out the new data you’ve got available, it will likely speed up your tools as well.

#109189: Update redacting functionality to redact all sensitive info in config when printing with view

kubectl config view has long supported a redaction mechanism for a few hardwired config fields. This PR extends things to use our existing datapolicy struct tag to handle things more holistically. This especially helps with user plugin credentials, such as OIDC access tokens. As before, the --raw flag will disabled redaction. If you are using kubectl config view in scripts or automation, you should check if this change will impact you (or if you are currently leaking sensitive information).

#113629: Promote APIServerIdentity to Beta

The leases API was originally added to streamline leader elections but it has found many uses beyond that. A new one added recently, and now promoted to Beta, is that each kube-apiserver creates a lease and renews it as long as it is running. This allows real-time enumeration of what API servers are running. The likely first use for this new data will be automating object storage version updates. Currently storage-version-migrator requires external coordination to ensure it runs to completion between each step when doing a marathon upgrade. Plenty of other things can likely benefit in similar ways though, debugging tools, cluster configuration helpers, etc. A simple and fun extension of an existing API!

Other Merges

Dynamic Resource Allocation, for adding and removing node-level resources at runtime, is complete enough for alpha
StatefulSets support “slices”, allowing sets to start at a non-0 number, usually for sharding a database or storage resource
Priority & Fairness “borrowing” is a complete feature, with a ton of metrics
The unhealthyPodEvictionPolicy allows non-ready pods to be evicted even if it exceeds PDB
OpenStack Cinder storage users, get ready to migrate to CSI
CEL Admission Policy is renamed to ValidatingAdmissionPolicy; metrics change
Add an API for cross-namespace Volumes, feature to come
Prevent a race condition crash between podFailurePolicy and deleted Pods, which does mean that the policy might not execute
Make sure that konnectivity connections don’t hang
NUMA distances work even if nodes don’t start at 0
Revert scheduler pre-filter skipper
Credential provider metrics no longer named kubelet_kubelet
Keep the startReflector from generating memory-eating nil pointers in the kubelet
Kubelet keeps fast-updating until the node is ready

Test Cleanup: add disruption and jobs test, StorageVersion unit tests

Promotions

Retriable pod failures for jobs to beta
kubectl events to beta (no more kubectl alpha events)
API Server Tracing was beta, but got reverted to alpha due to breakage

Version Updates

Last Week In Kubernetes Development (LWKD) is a product of multiple contributors participating in Kubernetes SIG Contributor Experience. All original content is licensed Creative Commons Share-Alike, although linked content and images may be differently licensed. LWKD does collect some information on readers, see our privacy notice for details.

You may contribute to LWKD by submitting pull requests or issues on the LWKD github repo.