Kubernetes v1.37 Turns Deferred Pod Resizes Into a Preemption Decision

A Kubernetes Blog post published this week walks through one of the more granular changes in the v1.37 release: a scheduler feature that finally gives Kubernetes a way to unstick an in-place Pod resize request that has nowhere to go. The feature, alpha and off by default, lets the scheduler evict lower-priority Pods on a node so that a higher-priority Pod’s resource request, one that would otherwise wait indefinitely, can finally be satisfied.

The Problem With a “Deferred” Resize

In-place Pod resize itself is not new. It lets Kubernetes change a running container’s CPU and memory allocation without restarting the container, and it reached General Availability in Kubernetes v1.35 after starting as alpha in v1.27 and passing through beta in v1.33. That graduation removed one kind of disruption, a Pod restart, but left a different one in place: what happens when a running Pod asks for more resources than its node has to spare.

Google engineer Natasha Sarkar, writing on the Kubernetes Blog, explains what happened before v1.37: “if a running Pod requested a resource scale-up that exceeded the host node’s allocatable headroom, the Kubelet was forced to mark the request as Deferred. The Pod would remain parked in this state indefinitely, waiting for resources on the node to naturally free up.” Nothing in the scheduler acted on that stuck request. If the node never freed capacity on its own, whichever controller asked for the resize, a Vertical Pod Autoscaler reacting to a load spike, for instance, would simply keep waiting.

That is a distinct outcome from an outright rejection. Kubernetes docs distinguish two states for a resize request: Deferred, a valid request the Kubelet cannot act on yet because the node lacks room and will keep retrying, and Infeasible, a request the Kubelet has determined cannot be satisfied under any circumstances and will not retry. Only the first case, the recoverable one, is what v1.37’s new feature addresses.

How Scheduler Preemption for Resize Works

The new capability lives behind the InPlacePodVerticalScalingSchedulerPreemption feature gate, alpha in v1.37 and disabled by default. Turning it on changes how the scheduler treats a Pod that is stuck in a Deferred state.

Tracking Deferred Pods, One Node at a Time

Ordinarily, once a Pod has a node assigned (its spec.nodeName is set), the scheduler considers it placed and moves on. Under the new feature gate, the scheduler instead keeps Pods carrying a Deferred resize condition in active scheduling evaluation, specifically so it can consider preempting other Pods on their behalf, and it keeps tracking them until the Kubelet actually completes the resize.

That tracking is deliberately narrow in scope. Standard Kubernetes preemption, the kind that decides where to place a brand new Pod, considers every node in the cluster. Preemption for an in-place resize does not: it is scoped to the single node the deferred Pod is already running on. The scheduler looks only at lower-priority “victim” Pods on that same host and evicts them gracefully to free capacity there. If evicting every eligible lower-priority Pod on that node still is not enough, the resize stays Deferred, exactly as it would have before v1.37; the feature does not migrate a Pod to a different node to complete a resize.

To prevent a race in which two Pods could both lay claim to the same freed capacity, the scheduler treats resources requested by a pending resize as already consumed the moment it commits to preempting for them. That reservation is what lets the Kubelet safely actuate the resize once the victim Pods are gone.

The Kubelet Steps Back

Before this feature existed, the Kubelet already had a way to make room for an important Pod: a local “critical Pod admission handler” that can evict lower-priority Pods on a node so a critical system Pod can be admitted there in the first place. It would have been straightforward to let that same local mechanism also handle resize-driven evictions. Kubernetes’ engineers chose not to.

Under the new feature gate, “the Kubelet’s critical Pod admission handler does not perform local preemption checks or trigger local evictions for in-place resizing operations,” the Kubernetes Blog post states. “Instead, the Kubelet defers the request and delegates the preemption decision entirely to the scheduler.” The stated reason is consistency: a single, centralized decision-maker that already respects PriorityClass values, PodDisruptionBudgets, and graceful termination periods, rather than two separate eviction paths that could disagree about which Pods are safe to remove. If a second, higher-priority resize request lands on the same node while a preemption cycle is already underway, the Kubelet defers to whichever request now has the higher priority, and the scheduler simply runs another preemption pass if more room is needed.

Opting Out, Node by Node

Cluster operators are not required to accept resize-triggered preemption everywhere. A new field on the Node spec, spec.podPreemptionPolicy.disableResizePreemption, lets an operator or an automated controller, a cluster autoscaler is the example Kubernetes gives, turn the behavior off on specific nodes:

apiVersion: v1
kind: Node
metadata:
  name: batch-workload-node
spec:
  podPreemptionPolicy:
    disableResizePreemption:
    - "cluster-autoscaler.kubernetes.io/disable-preemption"
    - "operator.example.com/policy-override"

That is a meaningfully different lever than the one Kubernetes already offered for ordinary preemption. Standard Pod preemption can only be switched off per PriorityClass, by setting preemptionPolicy: Never on the class itself, which then applies to every Pod created under that class everywhere in the cluster. The new Node-level field works the other way around: any Pod’s resize can be shielded from preemption, but only on the specific nodes an operator names, which is the granularity a resizable node pool or a custom autoscaler actually needs. The Kubernetes documentation frames the intended use case directly: it exists for “protecting existing workloads from disruption and allowing node-upsizing to handle the resource deficit instead,” rather than having the scheduler reach for eviction as a first resort.

Watching a Preemption Happen

The Kubernetes Blog post includes a mini-tutorial concrete enough to be worth walking through, since it shows exactly what an operator would see on screen. It starts with a single-node kind cluster built with the feature gate enabled and deliberately constrained CPU headroom, then deploys two Pods: low-priority-pod, requesting 3 CPU under a low-priority PriorityClass, and high-priority-pod, requesting 4 CPU under a high-priority PriorityClass. With both Pods running, the node has exactly 1 CPU of headroom left.

The demo then patches high-priority-pod to ask for 6 CPU instead of 4, a 2-CPU increase the node cannot supply on its own since only that 1 CPU of headroom exists:

kubectl patch pod high-priority-pod --subresource resize --patch \
'{"spec":{"containers":[{"name":"app", "resources":{"requests":{"cpu":"6"}, "limits":{"cpu":"6"}}}]}}'

With the feature gate enabled, the scheduler notices the resulting Deferred condition and targets low-priority-pod for eviction. Checking that Pod’s events shows exactly that:

kubectl get events --field-selector involvedObject.name=low-priority-pod

LAST SEEN   TYPE     REASON      OBJECT                  MESSAGE
5s          Normal   Preempted   pod/low-priority-pod    Preempted by pod 97dba925-6b5f-4e2f-99f9-d51c30016586 on node kind-control-plane
5s          Normal   Killing     pod/low-priority-pod    Stopping container worker

Checking high-priority-pod‘s own event history shows the other half of the exchange: a ResizeDeferred warning citing the CPU shortfall before the preemption, the same mechanism the architecture section describes, now visible as an ordinary Kubernetes event log rather than an abstract description. It is a small demo, one node and two Pods, but it makes the scheduler-Kubelet handoff concrete instead of theoretical.

Alpha, and What Comes Next

Scheduler preemption for in-place resize is alpha in v1.37, off by default, and requires the InPlacePodVerticalScalingSchedulerPreemption feature gate to be enabled consistently across the kube-apiserver, the kube-scheduler, and every kubelet in the cluster, not just one component in isolation. That is the same three-stage path in-place resize itself walked, alpha, then beta, then stable, on its way to v1.35. If scheduler-driven resize preemption follows a similar arc, the feature gate name and the podPreemptionPolicy Node field are worth learning now, since they are likely to become the default way a Kubernetes cluster reclaims capacity for a Pod that outgrows its original request, rather than something operators can put off thinking about.

It ships alongside several other scheduling and capacity-management changes in the same v1.37 release, including HorizontalPodAutoscaler scaling workloads to zero and KubeletInUserNamespace rootless mode graduating to beta, part of a broader pattern in this release of giving the scheduler more direct say over how a fully utilized cluster reallocates capacity, rather than leaving that decision to whichever workload happened to ask first.

Sources

• Kubernetes Blog: Kubernetes v1.37: Scheduler Preemption for In-Place Pod Resize (Alpha)
• Kubernetes Documentation: Pod Priority and Preemption
• Kubernetes Documentation: Resize CPU and Memory Resources assigned to Containers

Image credit: A player’s hand reaches for a chess piece on a wooden board. Photo by Nenad Stojkovic, licensed CC BY 2.0 via Wikimedia Commons; cropped and converted to WebP for sxz.io.