Production runbook

Operational guidance for node-rotation-controller. Each section answers when does this apply, what to look at, and what to do.

For the design rationale, see the specification. Japanese translation: docs/ja/runbook.md.

Incident right now?

Jump to §7 Troubleshooting — a symptom-based index that maps what you see to what you do.

---

Contents

1. Per-AZ surge headroom (zonal PV)
2. Tuning throughput and tGP
3. Metrics reference
4. The freeze workflow
5. Handling a stuck drain
6. Alerting (PrometheusRule)
7. Troubleshooting
8. Upgrading and rolling back
9. Sizing at scale

---

1. Per-AZ surge headroom (zonal PV)

When: your NodePool fronts workloads bound to zonal PersistentVolumes (EBS gp3/io2, or any PV with a topology.kubernetes.io/zone node affinity).

The constraint: the surge node is pinned to the candidate's AZ so the volume can re-attach. A same-AZ capacity shortage cannot fall back to another zone — the rotation rolls back after readyTimeout.

What to do:

• Keep enough pool-wide spec.limits headroom for one additional node, and ensure the NodePool's requirements permit every in-use AZ. spec.limits cannot reserve capacity per AZ.
• Separately confirm provider capacity in each AZ and enough regional EC2 vCPU quota for the surge.
• Consider capacity reservations for the surge instance shape in each AZ.

How to detect a shortfall:

• noderotation_completed_total{outcome="failure"} climbing
• noderotation_retry_count >= 3 for the pool (alert: NodeRotationRetryCountHigh)

When that alert fires on a zonal-PV pool, suspect per-AZ capacity first.

---

2. Tuning throughput and tGP

Raising throughput (window capacity C)

When: ThroughputBelowArrival or ThroughputBurstShortfall warnings appear, or candidates don't clear within a window.

What controls throughput:

C = ceil(D / (provisioningEstimate + drainEstimate + cooldownAfter))

Knob	What it is	How to set it
surge.provisioningEstimate	Expected time for the surge node to reach Ready	Read from noderotation_duration_seconds{phase="surge_wait"}
surge.drainEstimate	Expected time for a healthy drain	Read from noderotation_duration_seconds{phase="drain"}
surge.cooldownAfter	Pause between consecutive rotations	May be 0 if PDBs already serialize drains

What does NOT raise throughput: terminationGracePeriod. It no longer appears in C. Do not lower it for throughput warnings.

Choosing terminationGracePeriod

When: deciding how long Karpenter waits before force-killing pods on a draining node.

Pick it from: the downtime you can tolerate in an incident (a genuinely slow drain that hits the deadline), not from the drain times you observe in normal operation.

Reasons to lower tGP (none of them are throughput):

• Lengthens ageThreshold → nodes rotate later, less churn
• Relaxes the Auto Mode 21-day cap (expireAfter + tGP ≤ 21d)
• Surfaces a stuck drain sooner (noderotation_drain_stuck fires at tGP + buffer)

For the full derivation, see spec §3.2.

---

3. Metrics reference

Exposed on /metrics. Full semantics in spec §4.2.

Per-NodePool series are cleared when the NodePool is deleted or loses its governing RotationPolicy.

Key operational metrics

Metric	Type	What to watch for
noderotation_candidates	Gauge	Should trend to 0 after each window. Stuck > 0 for two windows → falling behind
noderotation_in_backoff	Gauge	Claims still due by age that would be candidates but for a failed attempt's retryBackoff. candidates + in_backoff is what a window has left to do
noderotation_in_progress	Gauge	0 or 1 (serial per pool in v1)
noderotation_completed_total{outcome}	Counter	outcome ∈ {success, failure, expired}. Any failure/expired → investigate
noderotation_forceful_fallback_total	Counter	Rising → graceful surges losing the race to deadlines
noderotation_window_missed_total	Counter	Per pool — window occurrences that closed with candidates outstanding and nothing attributable to them rotated
noderotation_duration_seconds{phase}	Histogram	phase ∈ {surge_wait, drain}. Use these to set your estimates
noderotation_drain_stuck	Gauge	1 → operator action needed (§5)
noderotation_retry_count	Gauge	≥ 3 → systematic failure (preemption or AZ shortage)
noderotation_short_lead_nodes	Gauge	Nodes that can't get K chances before their own expiry

Schedule and policy metrics

Metric	Type	Purpose
noderotation_window_active	Gauge	0/1 per pool — is the window open now?
noderotation_window_period_seconds	Gauge	Worst-case gap P between windows (per pool)
noderotation_age_threshold_seconds	Gauge	Derived ageThreshold A (per pool)
noderotation_rotation_chances	Gauge	Guaranteed chances G
noderotation_throughput_capacity	Gauge	Forecast C — starts per window occurrence
noderotation_t_rot_estimate_seconds	Gauge	t_rot_est — expected healthy rotation time
noderotation_t_rot_bound_seconds	Gauge	t_rot — deadline-side bound (feeds lead time)
noderotation_freeze_until_timestamp	Gauge	Active freeze timestamp (0 = none)
noderotation_policy_conflict	Gauge	1 = pool blocked by a policy conflict

Judging liveness

The controller's warning logs are deduped — a healthy idle loop emits zero log lines. Do not treat log silence as a stall. Use:

• controller_runtime_reconcile_total{controller="rotation"} — rising rate() = alive
• workqueue_depth{name="rotation"} — should stay near 0

---

4. The freeze workflow

Purpose: suppress rotation for a NodePool during a business-critical period.

Set:

kubectl annotate nodepool <name> \
  noderotation.io/freeze='2026-12-31T23:59:59Z' --overwrite

Lift:

kubectl annotate nodepool <name> noderotation.io/freeze-

Behavior while frozen:

• New rotations do not start
• A pending rotation is held (drain hasn't begun, so it's safe to pause)
• A draining rotation runs to completion (cannot abort a drain safely)
• The expireAfter backstop still applies — freezing cannot make nodes live forever

Monitor: noderotation_freeze_until_timestamp{nodepool} (0 = no freeze).

Best practice: manage freezes via GitOps, not ad-hoc kubectl. A forgotten freeze silently disables graceful rotation until its timestamp passes.

---

5. Handling a stuck drain

Symptom: noderotation_drain_stuck == 1 (alert: NodeRotationDrainStuck).

What happened: the controller deleted the old NodeClaim, Karpenter is draining the node via the voluntary path, but the drain exceeds tGP + buffer.

Important: the stuck drain blocks all rotation for that pool on purpose (to respect maxUnavailable = 1). Other pools are unaffected.

Decision flow:

Commands:

# Find the draining node
kubectl get nodeclaim -l karpenter.sh/nodepool=<pool> -o wide | grep -i terminating

# Pods still on it
kubectl get pods -A --field-selector spec.nodeName=<node> -o wide

# PDBs blocking eviction
kubectl get pdb -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed

Do not delete the controller's annotations or placeholder to "unstick" it — the state machine is idempotent and will re-assert. Fix the underlying PDB or finalizer.

---

6. Alerting (PrometheusRule)

The Helm chart ships an optional PrometheusRule (off by default). Enable:

helm upgrade --install node-rotation-controller charts/node-rotation-controller \
  --set prometheusRule.enabled=true

Alert	Fires when	Action
NodeRotationCompletedFailureOrExpired	A rotation failed or expired in the last hour	Check §1 (AZ capacity) and §5 (stuck drain)
NodeRotationCandidatesNotDraining	Candidates haven't cleared across two windows	Check §2 (throughput)
NodeRotationStalledInWindow	Window open and unfrozen, candidates + in_backoff > 0, zero successful completions	Check §1 or §5
NodeRotationDrainStuck	Drain exceeds tGP + buffer	Follow §5
NodeRotationShortLeadNodes	Nodes can't get K chances before expiry	Raise expireAfter or add windows
NodeRotationRetryCountHigh	Same rotation failing ≥ 3 times	Systematic cause — check §1
NodeRotationForcefulFallback	A rotation went surge-less (by design)	Check §2 if rising; single occurrence is expected
NodeRotationWindowMissed	A maintenance window closed with candidates unrotated	See below

Tune schedule-dependent ranges in your values:

• prometheusRule.candidatesNotDraining.windowRange → 2·P (default 8d for {Wed, Sat})
• prometheusRule.stalledInWindow.completionRange → window duration D (default 4h)
• prometheusRule.windowMissed.range → comfortably longer than one window period (default 24h; widen for a weekly schedule)

See values.yaml for all tunable fields.

Responding to a lost window (NodeRotationWindowMissed)

What it means: a maintenance window occurrence closed with candidates still outstanding by age and state (eligible, or inside retryBackoff) and no rotation attributable to that occurrence ever completed. The guaranteed rotation chance for that occurrence was consumed and lost — with minRotationChances: 1 (the floor), no guaranteed graceful chance remains for those nodes, and they may now reach expireAfter without one.

Two words in that sentence are narrower than they look: attributable, not "inside" — an attempt that began in-window and succeeded after the boundary settles the occurrence — and outstanding by age and state, not "the controller could have rotated" — a static or fatally infeasible pool reports every occurrence that closes with age/state-outstanding claims. Spec §4.2 defines both.

What to check:

• The WindowMissed Event on the NodePool — its counts (windowOpenedAt, eligible, inBackoffTriggered) say how much of the window's work went unrotated. The no rotation candidate line's inBackoff is a different number on purpose: it is the raw census bucket, which also holds claims whose age stopped crossing the trigger and which the window therefore owed nothing. Seeing the census line's inBackoff exceed this Event's inBackoffTriggered is that difference, not an inconsistency.
• The preceding rotation attempt failed log lines and their reason — a lost window is usually the tail of one or more failed attempts, not a cold start.
• noderotation_retry_count — climbing toward the escalated backoff cap means attempts are repeatedly failing, not merely running out of time.
• Attempts inside one maintenance window are paced by readyTimeout + failurePause, not by retryBackoff: the window-aware clamp (spec §3.2) keeps a failed claim's retry inside the occurrence it failed in only while a step down to retryBackoff still fits — once not even retryBackoff fits, the escalated wait stands and the claim carries over to the next occurrence instead of retrying again in this one. For a window of duration D the order-of-magnitude ceiling on timeout-driven attempts is 1 + D / (readyTimeout + failurePause). failurePause is the direct, pool-wide pacing control — several claims can be in backoff and retrying independently at once, so retryBackoff is not a reliable bound on the pool-wide rate. Raising retryBackoff can still reduce one claim's attempts, since it is the clamp's floor and a larger floor reaches "nothing fits" sooner in the window; it is just a blunt, per-claim lever that can discard the rest of that claim's window, not the direct control on pool-wide churn.
• Whether the pool is static (StaticNodePool Warning Event, spec §3.3) — a static NodePool never attempts a surge rotation, so it misses every occurrence that closes with age/state-outstanding claims.

What to do: address the underlying failure surfaced by the rotation attempt failed lines (see §1 and §5). If attempts are healthy but genuinely cannot fit the window — the batch is too large for the schedule — widen the maintenance window so more attempts complete per occurrence, or raise minRotationChances (K) so a single missed window still leaves guaranteed chances in reserve before the expireAfter backstop.

Known limits of this signal. The occurrence is identified by the presence of the noderotation.io/window-opened-at annotation on the NodePool, and only what a reconcile observes exists at all. Three consequences, accepted by design and enumerated in spec §5.3: two occurrences can be reported as one; a schedule edit can close a window immediately; and the report is at-most-once, never more. Operationally, only the last one changes what you write: alert on increase(...) > 0, not on an exact count.

Note: NodeRotationStalledInWindow is the in-window early warning for this same failure, but it is not a predictor of it. Both apply the same outstanding-work test (noderotation_candidates + noderotation_in_backoff > 0, both excluding a frozen pool), and spec §4.2 sets out the three ways they still diverge: when they evaluate, how a success suppresses each, and an in-flight retry that neither counts. The one that needs tuning: keep completionRange near one window's duration.

---

7. Troubleshooting

Start from what you see, confirm with the signal, then jump to the fix.

Symptom	Signal	Fix
Placeholder Pod stuck Pending, rotations fail	noderotation_completed_total{outcome="failure"}; retry_count climbing	§1 — same-AZ capacity
Candidates never clear across windows	NodeRotationCandidatesNotDraining alert	§2 — throughput too low
ThroughputBurstShortfall warning every window	Warning Event on NodePool	§2 — window too short for the batch
Drain hangs, no new completions	noderotation_drain_stuck == 1	§5 — PDB or finalizer
noderotation_in_progress stuck at 1	noderotation_drain_stuck == 1; also NodeRotationStalledInWindow when the pool has other claims outstanding — an in-flight rotation is in neither candidates nor in_backoff, so it alone does not raise that alert	Surge not landing (→ §1) or drain stuck (→ §5)
NodePool never rotates, candidates accumulate	noderotation_policy_conflict == 1	Fix the RotationPolicy selector overlap
NodePool never rotates, no surge is ever created	InsufficientHeadroom Warning Event on the NodePool	The pool's spec.limits leave no room for the placeholder, which reserves replacement capacity before draining. The Event names the resource, what the placeholder needs, and what remains of the ceiling: raise spec.limits by at least that shortfall, or reduce the pool's provisioned capacity. If surge.wholeNodeReservation is on, the gate tests a whole node's footprint (§3.3) — the pool needs a whole instance of headroom, not a drain's worth
NodePool never rotates, no attempt is ever made	StaticNodePool Warning Event on the NodePool	The pool sets spec.replicas (static capacity), which surge cannot rotate. Karpenter forbids adding or removing spec.replicas on an existing NodePool, so migrate the workload to a dynamic NodePool or drop this one from the policy selector
NodePool stopped rotating quietly	noderotation_freeze_until_timestamp > 0	Forgotten freeze — §4
A maintenance window closed with candidates still unrotated	NodeRotationWindowMissed alert; WindowMissed Warning Event	§6 — check the preceding failed attempts and retry_count
Nodes reach expireAfter despite the controller	noderotation_completed_total{outcome="expired"}	Lead time too tight — widen windows or lower tGP
noderotation_short_lead_nodes > 0	ShortLead Warning Event	Raise expireAfter on the NodePool or add window days
Rising forceful_fallback_total	ForcefulFallback Warning Event	Expected if throughput is tight; remediate via §2 if excessive
"Reconcile looks stalled" (no log output)	Check controller_runtime_reconcile_total is rising	Not a real problem — logs are deduped in steady state

---

8. Upgrading and rolling back

Upgrading the image

Safe mid-rotation. All state lives on Kubernetes objects — a rolling upgrade hands leadership to the new pod, which resumes any in-flight rotation from the active-rotation annotation. No external state, nothing lost in memory.

To quiesce first (optional): set a short freeze and wait for noderotation_in_progress to reach 0.

CRD schema changes

Helm does not upgrade CRDs. When upgrading into a release that adds a field, apply the CRD first:

kubectl apply -f charts/node-rotation-controller/crds/
helm upgrade --install node-rotation-controller charts/node-rotation-controller ...

Which releases changed the schema, and every behavioral and values change that needs an action on upgrade, is recorded per release in the changelog. Read the entries between your installed version and the target before upgrading.

The chart seals the rotationPolicies[].spec subtree, so a typo that older versions silently dropped now fails the upgrade. Dry-run first:

helm template node-rotation-controller charts/node-rotation-controller -f your-values.yaml >/dev/null

Rolling back

Rolling the image back is safe (same on-object state, older controller resumes). Rolling back across a CRD change may cause the controller to treat the policy as invalid (noderotation_policy_conflict == 1). Rotation pauses for that pool; expireAfter backstop still applies. Fix by also reverting the RotationPolicy objects.

---

9. Sizing at scale

When: clusters with 10k+ Pods.

The controller caches every Pod in the cluster (it needs cross-namespace visibility to size the placeholder). Memory scales with Pod count:

Pods	Cache footprint (lower bound)
~1k	~4 MB
~10k	~37 MB
~50k	~185 MB — budget 500 MB to 1 GB

CPU is not a concern — a 50k-Pod scan takes ~105 µs per call.

Size the Deployment's memory requests/limits accordingly. See the perf note for benchmark details.