3. Design

3.1 Maintenance Window

What this section defines

The maintenance window controls when rotations may start. In-flight rotations complete past the boundary. The worst-case window period P feeds the ageThreshold derivation (§3.2).

maintenanceWindows:        # a list; effective window = UNION of all entries
  - timezone: Asia/Tokyo   # IANA tz database name
    days: [Wed, Sat]       # Mon/Tue/Wed/Thu/Fri/Sat/Sun
    start: "02:00"
    end:   "06:00"

Semantics

• The reconciler is always running; window membership is evaluated each tick (1-minute).
• maintenanceWindows is a list; the effective window is the union of all entries.
• Outside the union the reconcile loop is a no-op.
• The window controls only rotation starts. In-flight rotations continue past the boundary.
• A freeze annotation (noderotation.io/freeze=<RFC3339 timestamp>) suppresses rotation until that time.

Freeze behavior

Unlike the window (which gates only starts), a freeze also holds an in-flight pending rotation:

• Suspends escalation only — placeholder (re)creation and the draining transition
• Passive bookkeeping keeps running — re-asserting do-not-disrupt/cordon markers, persisting surge-claim identification
• If the freeze outlasts readyTimeout → normal failure-path rollback
• A rotation already in draining continues to completion (aborting mid-drain is unsafe)

Worst-case window period P

P is the largest gap between the start of one occurrence and the start of the next over the recurring cycle.

• Example: union {Wed, Sat} 02:00–06:00 → gaps 3d and 4d → P = 4d
• Continuously-open (24/7) union: P collapses to the reconcile-tick granularity (not 7d)
• DST note: P is computed over the recurring wall-clock cycle. A DST transition can shift an individual gap by ±1h; v1 treats this as a known approximation

3.2 Candidate Selection

What this section defines

Three questions: (1) when does a node become a candidate? (2) how is ageThreshold derived so the controller finishes before expireAfter? (3) what validations ensure feasibility? Core formula: A = E − (K·P + t_rot).

Selection conditions

A NodeClaim becomes a candidate when all of the following hold:

Condition	Notes
now() > deadline − leadTime	Anchored on each NodeClaim's own spec.expireAfter
Belongs to a governed NodePool	Matched by a RotationPolicy (§5.4)
Ready == True	NotReady → left to Auto Repair / backstop
deletionTimestamp unset	Already-deleting → excluded
state empty or failed past backoff	pending/draining in-flight; expired terminal
No operator do-not-disrupt	Operator's own opt-out honored

• Deadline computation: deadline = creationTimestamp + NodeClaim.spec.expireAfter
• Ordering: earliest deadline first, ties broken by oldest creationTimestamp, then name
• Heterogeneous expireAfter: a younger claim with a shorter expireAfter can have an earlier deadline and is rotated first
• Operator opt-out: karpenter.sh/do-not-disrupt: "true" on the Node (without do-not-disrupt-owned marker) excludes the claim from proactive rotation while keeping its expireAfter backstop

The escalated backoff is clamped to the maintenance-window occurrence the failure happened in. Past that occurrence's remaining time the ladder stops discriminating — every step means "skip the rest of this occurrence" — so a pool that fails twice early spends the rest of its window with no eligible candidate. When the escalated retry would land at or after the close, the ladder is walked down to the largest step that still lands strictly inside it, and never below the configured retryBackoff. When not even retryBackoff fits, the escalated value stands and the claim waits for the next occurrence.

The clamp applies only when the failure itself happened inside the occurrence containing the evaluation instant, under the schedule as it stands on that evaluation. It is not a historical identity: an operator edit that merges two occurrences lets a failure from the earlier one be clamped by the merged run, and a split does the converse. retry-count is unaffected — the escalation still climbs on every failure, and only the wait inside one occurrence is shortened.

Deriving ageThreshold

Rather than hand-tuning (error-prone — too loose lets Forceful Expiration fire), the controller derives ageThreshold per NodePool from the schedule and target rotation chances.

The central race: Forceful Expiration fires at each node's deadline regardless of windows or PDBs. The controller must finish a graceful surge rotation before that moment.

Formula:

ageThreshold (A) = E − (K·P + t_rot)

Read leadTime = K·P + t_rot left to right:

• K worst-case window cycles (K·P) to catch a window
• Plus one completion time (t_rot) to finish inside it
• Guarantees K maintenance windows with enough headroom before expireAfter

Symbols (authoritative source)

Symbol	Source
E	Per-node: NodeClaim.spec.expireAfter (authoritative). Template used only as representative for validation
tGP	Per-node: NodeClaim.spec.terminationGracePeriod; template as representative
P	Derived from maintenanceWindows union (§3.1)
t_rot	readyTimeout + tGP + buffer. When tGP unset → fixed 1h fallback
t_rot_est	provisioningEstimate + drainEstimate. Layer-2 only, no deadline terms
buffer	Fixed 4·shortRequeue = 2m. Deadline-side only, not in t_rot_est

Authoritative expiry source

The per-node trigger reads from NodeClaim.spec.expireAfter, anchored at that NodeClaim's creationTimestamp. Later edits to the NodePool template do not propagate to existing NodeClaims (only trigger drift). The template E is used solely as the representative for per-NodePool validation and metrics.

Margin and cooldownAfter

• The bound is tight — exactly K chances, no built-in slack
• Safety margin comes from K itself — K ≥ 2 recommended
• cooldownAfter is the post-success settle between consecutive rotations — not part of t_rot
• failurePause (gate B, §4.4, ADR-0004) is a separate field feeding no throughput term

Validation details (layers 1–3) — click to expand

Layer 1 — scheduling feasibility

Condition	Outcome
P = 0 (no window occurrences)	fatal (NoWindows)
K < 1	fatal — invalid config
K < 2 (i.e. K = 1)	warn — no retry if window missed
A ≤ 0 (E ≤ K·P + t_rot)	fatal — raise E, add windows, or lower K
0 < A < P	warn — extremely aggressive rotation
Explicit override with G < 1	fatal — override rejected
Explicit override with 1 ≤ G < K	warn — weakened chances
E + tGP > 21d	warn (HardCapExceeded) — Auto Mode violation
tGP unset	warn — drain phase unbounded
retryBackoff < readyTimeout	warn — retries faster than attempts
spec.limits no headroom	warn — surge cannot land

Layer 2 — throughput

Per-occurrence capacity: C = m · ceil(D / (t_rot_est + cooldownAfter)).

• C uses t_rot_est (expected service time), not t_rot (deadline bound)
• Every positive-length occurrence admits at least one start (C ≥ 1)
• If arrival exceeds capacity (C < N · P / A): warn — widen windows or add occurrences

Synchronized batch: N nodes with one shared deadline. Completes gracefully only when K · C ≥ N. Otherwise warn (ThroughputBurstShortfall).

Spillover: when t_rot_est + cooldownAfter > gap, a rotation started late carries into the next occurrence. warn (RotationSpansNextWindow) — adjacent occurrences do not each deliver full C.

Layer 3 — per-node runtime

The template-level checks above don't prove every existing claim is satisfiable. On each reconcile the controller also checks every in-scope NodeClaim against its own spec.expireAfter:

• E_node ≤ K·P + t_rot (per-node A ≤ 0) → counted in noderotation_short_lead_nodes, warned via ShortLead Event
• Rotated best-effort at the earliest opportunity
• Once deletionTimestamp is set → excluded from selection, abort path applies (§5.2)

Worked example — click to expand

Setup: Auto Mode, tGP lowered to 1h, E = 14d, union {Wed, Sat} 02:00–06:00.

• P = 4d
• t_rot = 15m + 1h + 2m ≈ 1h17m
• K = 2
• A = 14d − (2·4d + 1h17m) ≈ 5.9d

Nodes become candidates at ~5.9d and are guaranteed 2 windows before 14d.

Throughput:

• provisioningEstimate = min(15m, 5m) = 5m
• drainEstimate = min(1h, 10m) = 10m
• t_rot_est = 15m
• C = ceil(4h / (15m + 10m)) = 10 per occurrence

Fatal example: weekly-only window {Sat} → P = 7d → A = 14d − (14d + 1h17m) ≈ −1h17m ≤ 0 → fatal. Fix: raise E to ~20d or add a window day.

Calibration note: with stock tGP = 24h, t_rot ≈ 24h17m but t_rot_est remains 15m (same C = 10). Lowering tGP helps by: (1) lengthening A, (2) relaxing the 21-day cap. Pick tGP from risk tolerance; pick drainEstimate/provisioningEstimate from observed durations.

3.3 Surge Sequence

What this section defines

A single reconcile cycle handles one node: serial per NodePool (maxUnavailable = 1), concurrent across NodePools. The placeholder Pod induces NodePool-owned replacement capacity.

Why same-NodePool (not a standalone NodeClaim)

A standalone NodeClaim produces an unowned node outside NodePool accounting, expiry, drift, and disruption budgets — breaking intentional NodePool separation.

Static capacity NodePools are out of scope

A NodePool with spec.replicas set (Karpenter static capacity) maintains a fixed node count and is never considered by the provisioner when a Pod is pending. The placeholder pins karpenter.sh/nodepool to the candidate's own pool as a structural invariant, so on a static pool it can be neither absorbed by another pool nor provisioned for — every attempt would stall until readyTimeout and consume one of the node's guaranteed chances.

The controller therefore refuses to start a rotation on a static NodePool and says so once via a StaticNodePool Warning Event (§4.3, gate in §5.2). Those nodes remain subject to Karpenter's forceful expiration.

Karpenter rejects a transition between static and dynamic on an existing NodePool (a CEL rule on spec.replicas), so the remedy is to migrate the workload to a dynamic NodePool or to exclude this one from the RotationPolicy selector — not to edit the field in place.

Surge-less replacement driven by replica reconciliation — raising spec.replicas to surge, or deleting the NodeClaim and letting replica reconciliation refill it — was evaluated and rejected, not deferred. Karpenter's static deprovisioning selects empty nodes first, so a node provisioned by raising replicas is the first candidate for the scale-down that follows rather than merely a possible one; and the concurrency that would make delete-and-refill a one-node dip instead of a stall for the whole drain holds only while limits.nodes exceeds replicas, which is the operator's configuration rather than a property of the mode. surge.forcefulFallback is not a precedent for shipping it anyway: that is an opt-in, window-bounded escape for a pool that would otherwise miss its deadline entirely (ADR-0001), not a pool's steady-state mechanism. Both constraints are properties of Karpenter's current node accounting (v1.14), so the decision is worth revisiting if that changes — not because rotating static pools is undesirable.

The placeholder Pod

Property	Value
Kind	Bare Pod (no controller)
Priority	Dedicated negative PriorityClass
Preemption	preemptionPolicy: Never
Requests	Reschedulable Pod sum (clamped), or a whole node when wholeNodeReservation is on
Node selector	karpenter.sh/nodepool = <pool>
Node affinity	Soft: avoid candidate + near-deadline
Tolerations	From NodePool spec.template.spec.taints

Request sum exclusions

Pods that Karpenter does not need to re-fit:

• DaemonSet Pods — Karpenter adds overhead to every new node (double-count)
• Mirror / static Pods
• Completed Pods (Succeeded / Failed)
• Node-pinned Pods (hostname affinity)

Hostname exclusion (soft, not hard)

• Candidate exclusion enforced by the cordon, not this preference
• Near-deadline exclusion is best-effort
• Why not required: Karpenter rejects required kubernetes.io/hostname affinity (restricted label)
• Exclusion lists recomputed on each (re)creation; stale lifetime ≤ readyTimeout

Two provisioning paths

• New-provision: Karpenter provisions a NodePool-owned node in the same AZ
• Capacity-absorb: scheduler places placeholder onto pre-existing spare capacity

Either way, the host becomes the surge target, frozen for the rotation's duration.

The two paths cost very different amounts of time and reserve very different things, so the controller names the one it took: surgePath ∈ {provisioned, absorbed} on the surge node ready and rotation complete lines and on the RotationCompleted Event, carried between them on the surge-path anchor field (§5.3).

The path is decided by one question — did the surge host's NodeClaim come into existence during this attempt? — evaluated with the same predicate the rollback's reap guard uses, so the two can never disagree about the same host. It reports what is observable, not causality: a node Karpenter provisioned for some other pending Pod inside this attempt's window, which then absorbed the placeholder, reads as provisioned. A host whose NodeClaim cannot be resolved yields no value rather than a guessed one.

Placeholder sizing clamp

Problem: Karpenter caches one allocatable per instance type, but actual allocatable can be higher per-AZ. A node filled past the cached estimate produces an unprovisionable placeholder.

Solution:

limit    = NodeClaim.status.allocatable − DaemonSet overhead   (per resource, floored at 0)
requests = min(reschedulable sum, limit)                        (per resource)

• Uses status.allocatable — no instance-type or cache knowledge needed
• If status.allocatable absent → clamp is a no-op
• surge_headroom (§5.2) tests the clamped footprint
• Both terms are estimates: the cached per-type allocatable, and the DaemonSet overhead observed on the candidate, which Karpenter's own estimate for a fresh node need not match. The clamp aims the placeholder at that ceiling; it does not guarantee the fit

Edge cases:

• Refused (limit ≤ 0 on a resource the drain demands): the DaemonSet overhead observed on the candidate leaves that resource's cached ceiling non-positive → every clamp value under it would reserve no positive amount of that resource, letting surge_ready be satisfied with that dimension of the drain unreserved — so the placeholder keeps the full drain instead. The placeholder need not be empty for this to bite: with a positive cpu ceiling and a zero memory ceiling it still carries cpu, and only the memory has no reservation behind it. (At limit = 0 a zero request for that one resource is arithmetically admissible beside the overhead; what rules it out is that it holds none of the demand, not that it fails to fit.) The refusal is per resource — the clamp returns on the first one it finds, and the drain's others may have positive ceilings and be reservable. And it settles no question of schedulability. Both terms come from the candidate, and the ceiling is Karpenter's cached per-type estimate — the same estimate real nodes exceed, which is the premise of this clamp and the gap the band measures. So a node of the same type carrying the same DaemonSets can still have room, as can a larger allowed instance type or a node carrying less applicable overhead. Those are examples, not an exhaustive set: where nothing can take the placeholder it stays unschedulable and the rotation rolls back at readyTimeout, but that is an outcome, never the complement of a list. The SurgeClampRefused Warning Event (§4.3) states it in exactly that form
• Band-exceeded (shortfall > measured band): SurgeClampBandExceeded Warning Event; rotation proceeds
• Common case (fits under limit): silent

Whole-node reservation (ADR-0005)

Problem: the placeholder reserves the sum of the drain as a single Pod. On the capacity-absorb path that aggregate hole sits on a host already running other Pods, and it is fungible with the individual evicted Pods' placement only if that host can accept all of them. A host can offer a big enough hole while a Pod's own podAntiAffinity or hostPort still refuses it — and Karpenter then provisions for that Pod after the drain has started, behind the surge rather than in front of it.

Solution (opt-in, surge.wholeNodeReservation, default off):

requests = max(requests, limit)   (limit as above)

applied to cpu and memory whenever the candidate has reschedulable Pods, plus every other resource the drain itself requests.

It raises the bar; it is not a guarantee

Every host whose free cpu or memory is short of the candidate-sized footprint is excluded; how much of the fleet that removes depends on the pool's shape. A genuinely empty host (DaemonSets only) still absorbs the placeholder, which is correct: an empty host has nothing for a hostname-topology anti-affinity to bite on.

It does not prove a host is empty. The reservation is sized from the candidate's allocatable, so three ordinary situations let an occupied host take it: a larger host (the placeholder pins the NodePool and the replicated requirements, not the instance type, so on a heterogeneous NodePool an 8-CPU host running 2 CPU has a 4-CPU candidate's worth free — the common case, not a corner one); a host with less DaemonSet overhead than the candidate; and Pods that request no cpu or memory — including Pods requesting only an accelerator or ephemeral storage, since only those two dimensions are raised — which occupy nothing the reservation measures and can still be the ones whose anti-affinity or hostPort refuses an evicted Pod.

There is no way to express "a host with no other Pods": a required kubernetes.io/hostname NotIn term makes Karpenter's provisioner refuse to provision at all, and a required podAntiAffinity matching every Pod would exclude the DaemonSets every node carries. Adding node.kubernetes.io/instance-type to surge.matchNodeRequirements.required narrows the first residual by pinning the candidate's own type, at the cost of Karpenter's freedom to substitute types when capacity is short.

• Both bin-packing dimensions, whatever the drain declares — a cpu-only drain that reserved only cpu could still be absorbed by a host filled with memory-only Pods. pods is never requested (it is not a container request); ephemeral storage and accelerators are raised only when the drain asks for them
• The Pod count decides whether to reserve, not the sum: a candidate carrying only zero-request Pods has an empty drain and real workload. No reschedulable Pods reserves nothing
• Does not force a larger instance: limit + DaemonSet = allocatable, so the request never exceeds the candidate's own class on resource fit — though the placeholder does not pin the instance type, so Karpenter may still choose another by availability, price, or the replicated requirements
• A non-positive limit is left to the clamp's refusal — raising to it would reserve none of that resource, letting surge_ready be satisfied with that dimension unreserved (and where the drain requests nothing else, with an empty placeholder reserving nothing at all): a silent break-before-make either way. A drain already above the limit is likewise left for the clamp to lower and report
• surge_headroom (§5.2) tests the raised footprint, so a pool whose spec.limits are nearly exhausted stops starting rotations and says so (InsufficientHeadroom, §4.3)
• Cost: an extra instance on every rotation whose reservation is not absorbed, and possibly a consolidation cycle once the surge host leaves do-not-disrupt. How often that is depends on the pool's shape (§4.4) — it is not a fixed per-rotation cost

Constraints coarser than the node — zone-level podAntiAffinity, topologySpreadConstraints — are not addressed: they are decided by what is in the zone, not by what is on the host. See ADR-0005 for why modelling the Pods faithfully does not address them either.

Placeholder priority and preemption

• Victim by design: negative priority + preemptionPolicy: Never → workload preempts it; it never preempts anything
• Bare Pod: when preempted, simply gone — state machine detects and recreates (bounded by readyTimeout)
• Hostile preemption: repeated preemption self-terminates at readyTimeout → clean rollback

Sequence diagram (happy path)

surge_ready conditions

The placeholder must be:

• Running and not terminating (deletionTimestamp unset)
• On a Ready host ≠ candidate node
• Host's karpenter.sh/nodepool label == pool name

A preempted placeholder stays Running with deletionTimestamp set → not ready (reservation being removed). Recreated once gone, bounded by readyTimeout.

Capacity reservation semantics

The placeholder reserves one node's worth of capacity. The guard is physical reservation: surge_ready requires the placeholder to be Running on a Ready node other than the candidate. Whether new or pre-existing, the host's admission means reschedulable capacity is physically held.

Absorb-path limitation

On the absorb path, the reservation is aggregate — one node's requests held on a host already running other Pods. An individual displaced Pod can still fail to use it (pod anti-affinity, hostPort collisions), in which case Karpenter provisions for it after the drain has started. The controller guarantees node-level capacity; per-Pod placement is the scheduler's and PDB's domain (§3.5).

surge_wait therefore does not bound the time until the evicted Pods are running on this path — a four-second wait can precede a node launch that has simply moved behind the drain. That is why the path is named alongside the duration (§4.2): the two paths are otherwise indistinguishable in the log.

3.4 Mid-surge Protection

What this section defines

While old and new nodes coexist, the controller prevents Karpenter's optimizer from consolidating/drifting the half-built surge pair.

Mechanism: do-not-disrupt annotation

Applied to both the old node and the surge target for the rotation's duration.

What it blocks

Disruption type	Blocked?
Consolidation	Yes
Drift	Yes
Emptiness	Yes
Forceful Expiration	No
Interruption	No
Node Repair	No

• Winning the race against expireAfter is leadTime's job (§3.2), not this annotation's
• The annotation prevents the optimizer from disrupting the half-built pair

Ownership tracking

• noderotation.io/do-not-disrupt-owned=true written only when the controller applies do-not-disrupt
• Operator's pre-existing do-not-disrupt: true (no marker) is never touched
• Non-true values are not operator protection — overwritten and owned
• Cleanup removes do-not-disrupt only where the owned marker is present

surge-for marker

Each frozen node carries noderotation.io/surge-for=<old NodeClaim name>:

• Attributes the freeze to this rotation
• Finds the surge target after old NodeClaim deletion
• Does not carry do-not-disrupt ownership semantics

Mechanism: cordon

On entering pending, the controller cordons the candidate node:

• Purpose: prevent new Pods landing during surge wait (placeholder requests are a snapshot)
• Marker: noderotation.io/cordoned=true — rollback/sweep undo only the controller's cordon
• Conditional: already unschedulable without marker → no-op (operator's cordon not adopted)
• Not a rotation veto: a cordoned node is still selected. Use freeze to suppress rotation

Residual risk

If the old node's deadline arrives while the surge is still waiting → Karpenter force-expires on schedule. This is a tight-leadTime / last-window edge case that degrades to the native baseline (§3.9), not a prevented scenario.

3.5 Pod-level Behavior

Key point

Make-before-break is at the node level, not Pod level. Pod-level safety is delegated to PDB + replica headroom.

The controller does not perform a rolling update of Pods. The surge node is added as empty capacity.

When the old NodeClaim is deleted:

1. Karpenter's termination controller drains via the Eviction API (PDBs respected)
2. Each evicted Pod is deleted
3. The owning workload controller creates a replacement Pod
4. The scheduler places it onto available capacity (typically the surge node)

This is evict-then-reschedule — a replacement Pod is not guaranteed Ready before the old terminates (§4.1).

The surge node's role: pre-stage a landing zone so PDB-gated eviction proceeds without a long pending window.

• Strict PDB (minAvailable = desired count): eviction blocks until replacements are Ready → effectively Pod-level make-before-break (enabled by the surge node's capacity)
• Loose/absent PDB: evictions proceed in bulk, readyReplicas dips (§4.1)

Summary: the controller guarantees a node-level surge; Pod-level make-before-break is achieved by PDB + replica headroom, which the surge node's capacity enables — consistent with G4.

3.6 Forceful Fallback (opt-in)

What this section defines

When surge.forcefulFallback.enabled: true and a graceful surge cannot finish before the candidate's deadline, the controller deletes the NodeClaim surge-less, inside the window, via the voluntary path (PDBs apply).

Trigger

deadline − now < t_rot

Where deadline = creationTimestamp + spec.expireAfter and t_rot = readyTimeout + tGP + buffer.

Behavior

• Deletes old NodeClaim in-window without make-before-break (break-before-make)
• Still via the voluntary path — PDBs respected up to terminationGracePeriod
• Relaxes only the node-level surge property, not "never bypasses Karpenter" or G4
• Pulls the otherwise-uncontrolled expiration into the window
• Drops readyTimeout and provisioning wait → raises throughput

Constraints

• Serial per NodePool (maxUnavailable = 1)
• A candidate with expireAfter: Never (nil) has no deadline → never qualifies
• Default off — when off, surplus nodes degrade to native expireAfter baseline (§3.9)
• Recorded via noderotation.io/rotation-mode = forceful-fallback on the anchor (§5.3)
• Emits ForcefulFallback Warning Event + noderotation_forceful_fallback_total counter (§4.2)

3.7 Zonal Workloads

What this section defines

Zonal-PV Pods can only reschedule in the same AZ. The placeholder replicates the candidate's zone (and other configurable requirements) so the surge node lands in the correct AZ.

Problem

A Pod bound to a zonal PersistentVolume (EBS gp3/io2) can only reschedule in the same AZ. If the surge node lands in a different AZ, that Pod stays Pending.

Solution: matchNodeRequirements

The placeholder replicates configurable scheduling requirements from the candidate node:

Category	Default keys	Purpose
required	topology.kubernetes.io/zone, kubernetes.io/arch, karpenter.sh/capacity-type	Pin surge to same AZ, arch, capacity type
preferred	(empty)	Relaxed under capacity pressure

• Operators add keys for stricter parity (instance type, custom labels)
• Move keys to preferred to trade strictness for schedulability

Requirement resolution

Keys are read from the candidate NodeClaim.spec.requirements and node labels, intersected with the NodePool's allowed requirements:

• Intersection keeps the placeholder schedulable within the pool
• Node label is authoritative (actual placement) and wins on conflict
• For keys not surfaced as labels → NodeClaim.spec.requirements In values used
• Absent from both sources → skipped
• Numeric operators (Gt/Lt/Gte/Lte) handled correctly — never silently weakened
• Removing topology.kubernetes.io/zone from required → warn (zonal-PV Pods may strand)

Limitations

• Only re-creates a same-AZ landing zone; does not move storage
• CSI driver re-attaches the existing volume once the replacement Pod is scheduled
• If the AZ has no schedulable capacity → rolls back via readyTimeout; backstop applies
• NodePools fronting zonal-PV workloads should ensure per-AZ surge headroom (R3)

3.8 Rollback Behavior

Failure	Action
New node not Ready within timeout	Reap surge claim, delete placeholder, unfreeze, record failure
New node NotReady after old deleted	Drain in flight, cannot reverse; Karpenter reconciles capacity
Karpenter API unavailable	Skip; next reconcile retries
Controller dies mid-surge	Resumes from active-rotation anchor (§5.2); idempotent re-assertion

Timeout rollback detail

When readyTimeout expires:

1. Reap the induced surge NodeClaim — identified from noderotation.io/surge-claim (persisted as soon as placeholder's bind target is observable)
2. Unfreeze nodes — remove controller's do-not-disrupt (by owned marker) and cordon (by cordoned marker)
3. Record failure — write last-failure-at on NodePool, clear anchor, emit failure metric + alert

Surge-claim identification

The surge-claim annotation is persisted by the pending handler as soon as the placeholder's bind target is observable — spec.nodeName (the only scheduler-visible signal for a non-preempting Pod). Fallback resolution order on failure:

1. Re-resolve from a still-present placeholder
2. The pool's NodeClaim created after started-at with no registered Node

Reap guards

Two guards prevent reaping the wrong claim:

• Must be created after started-at (pre-existing capacity-absorb hosts are never surge debris)
• Node must host only the placeholder + DaemonSets (no real Pods)

WARNING

v1 processes one node per cycle. Unprocessed candidates roll to the next window. The expireAfter backstop ensures eventual rotation.

3.9 Backstop Behavior

Key point

Every failure mode degrades onto Karpenter's native Forceful Expiration — never worse than the status quo without this controller.

If the controller is unavailable, the safety net engages in order:

1. Consolidation / Drift — may still rotate some nodes via the voluntary path
2. expireAfter — triggers Forceful drain on overdue nodes
3. terminationGracePeriod — bounds the drain
4. Auto Mode 21-day hard cap — final ceiling

Stale do-not-disrupt is not a risk

A stale karpenter.sh/do-not-disrupt left by a crashed controller:

• Suppresses only voluntary disruption (path 1)
• Does not block expireAfter (path 2) — the node cannot outlive its deadline
• The startup sweep clears it, but the marker was never extending node life

Graceful degradation guarantee

The controller only ever moves rotation earlier and makes it graceful. It never:

• Removes the safety net
• Extends a node's life beyond expireAfter (node-level do-not-disrupt has no effect on it)

The worst case equals today's baseline — forceful, but bounded. Safe to adopt incrementally.

When graceful guarantee is impossible

Once capacity is below demand (C · A < N · P or a batch with N > K·C), some forceful disruption is unavoidable:

• Default: happens at the native, uncontrolled expireAfter deadline
• With forcefulFallback: the controller performs a controlled surge-less rotation inside the window (§3.6)
• NodeClaim.spec is immutable — the controller cannot retime expireAfter; the only lever is replacement