Forceful Fallback — Scenario O

What this validates

Window-bounded forceful fallback (#156), earliest-deadline ordering (#157), and do-not-disrupt exclusion (#170) running together on a shared deadline — all on real EKS Auto Mode.

Real-EKS validation of the three features above exercised simultaneously. See §7.2 for the assumptions this flips to validated, and the runbook for what noderotation_forceful_fallback_total and ForcefulFallback mean operationally.

Scenario O · EKS Auto Mode · 2026-07-12 UTC

12-node rotation on a shared deadline — graceful → forceful fallback split

All 12 nodes are created at once and share the same creationTimestamp (deadline). They are processed one at a time, serially (maxUnavailable=1), and graceful vs. forceful is decided purely by whether, at the moment a node's turn comes up, t_rot (12m) or more remains until the deadline. expireAfter stays fixed at 2h throughout (trick-free).

forceful_fallback_total

0 → 3

completed{success}

12 9 graceful + 3 forceful

expireAfter backstop

0 expired

controller restarts

0 counter never reset

Timeline (2h proportional axis)

graceful surge (with placeholder) × 9 forceful fallback (surge-less) × 3 expireAfter backstop (0 this run)

Decision logic (branches on remaining time at pick)

9

Case A graceful

remaining ≥ t_rot (12m)

There is enough time before the deadline to complete the surge. placeholder Pod → new node Ready → voluntary drain of the old node (PDB applies). mode is empty. Nodes 1–9.

3

Case B forceful

remaining < t_rot (12m)

The graceful surge would not finish in time. Since it is in-window and opt-in, the NodeClaim is deleted without a placeholder (voluntary path, PDB applies). mode=forceful-fallback, a Warning event, counter +1. Nodes 10–12.

0

Case C backstop

age reaches 2h

Even forceful fallback would not finish in time and the deadline is reached → Karpenter force-expires the node (outcome=expired). The "fallback of the fallback." This run, forceful fallback handled all remaining nodes by 10:33, so this case is 0.

Chronology (observed log)

Time UTC	age	phase	NodeClaim	behavior / evidence
08:38:20	0	▸ 12-node synchronized batch creation — shared deadline 10:38:20Z. schedule intentionally warns (ThroughputBurstShortfall N=12>K·C=6 / ThroughputBelowArrival)
09:26:20	48m	▸ eligible boundary (ageThreshold A=48m) — from here, candidates become rotation targets
09:26:24	48m	graceful	152t8h	placeholder surge → make-before-break (total 1m2s)
09:33:41	55m	graceful	25rp47	placeholder Pending→Running, mode empty
09:41:29	1h03m	graceful	394f5n	same as above (cooldown 6m → interval ~7m)
09:48:45	1h10m	graceful	4fvd7p	same as above
09:56:34	1h18m	graceful	5gjgg8	same as above
10:04:50	1h26m	graceful	6n8zqt	same as above
10:12:40	1h34m	graceful	7p2vdk	same as above
10:19:54	1h41m	graceful	8phmms	same as above
10:24:21	1h46m	graceful	9rnwb7	last graceful (age 1h46m, just inside the 1h48m boundary)
10:26:20	1h48m	▸ forceful boundary crossed (deadline − t_rot) — cooldown dropped 6m→1m just before, at 10:24:00. Every pick from here on is Case B
10:26:55	1h48m	forceful	10sgh57	first forceful (age 1h48m35s), mode=forceful-fallback, placeholder NotFound, counter 0→1
10:30:59	1h52m	forceful	11t7sb5	surge-less, Warning event, counter →2
10:33:16	1h54m	forceful	12tkkf7	counter →3 — complete, all 12 rotations done (~5m before the 10:38 deadline)
10:38:20	2h	▸ expireAfter backstop — not reached (Case C: 0)

#157 (earliest-deadline order): since all 12 share the same creationTimestamp, ordering degenerates to the Name tiebreak, and they are consumed in exact ascending order — 52t8h < 5rp47 < 94f5n < fvd7p < gjgg8 < n8zqt < p2vdk < phmms < rnwb7 < sgh57 < t7sb5 < tkkf7.

The graceful/forceful mix was shaped via cooldown tuning: a real graceful rotation finishes in ~1 minute (46s–1m51s), so left alone the serial surge would handle all 12 and no forceful fallback would occur. cooldownAfter=6m in the first half held the surplus back until the deadline, then dropping it to 1m at 10:24 (just before the boundary) let forceful fallback clear the rest before the backstop. expireAfter never changes, so this is not a trick.

t_rot = readyTimeout 5m + tGP 5m + buffer 2m = 12m (buffer shrank 15m→2m in #215). The split is decided purely by whether the pick time falls before or after 10:26:20Z; because the deadline is shared, the instant that boundary was crossed, every remaining node flipped to forceful at once.

Scenario coverage

The matrix below tracks every EKS Auto Mode PoC scenario against the specification's roadmap and open-questions. A scenario stays "planned" until re-run and observed on a real cluster — code coverage (unit/envtest/KWOK) is necessary but not sufficient for a row to flip to validated.

EKS Auto Mode PoC · SCENARIOS.md × VALIDATION.md

Scenario coverage matrix

Rows = the capability (behavior layer) under test, columns = scenarios 0–O. ● = that scenario's primary focus, ◯ = passes through as a secondary effect. A row where two scenarios both have ● is overlapping; rows marked ☁ are guarantees from Karpenter/AWS itself (not this controller's logic).

Scenarios

17 0, A–O

Capabilities under test

17 layers

Overlapping capability rows

2 backstop · rollback

Karpenter/AWS-leaning

2 N · A (partial)

primary focus passes through (secondary) not covered✓ validated (observed on real EKS)○ planned (code coverage only)⚠ overlap the same capability is the primary focus of 2 scenarios☁ external a Karpenter/AWS guarantee

Capability / scenario →	Status	0	A	B	C	D	E	F	G	H	I	J	K	L	M-A	M-B	N	O
Surge & placement
Surge — new provisioningmake-before-break	✓
Surge — capacity-absorbbin-packs onto existing spare (no new node)	✓
Placement — NodePool confinementdoesn't leak onto another pool's spare	✓
Gates (don't start / boundary)
Gate — NodePool limitsno surge when there is no headroom	✓
Gate — maintenance window / boundaryin-flight finishes; nothing new starts past the boundary	✓
Backstop, failure & outcome
expireAfter backstop (R6)⚠ D⊂Ilead time wins out	✓
readyTimeout rollback + cleanup⚠ C≈M-Bstate=failed / retry++ / uncordon	✓
Forced-expiry expired outcomecaught while pending	✓
PDB-respecting voluntary drainminAvailable actually blocks	✓
do-not-disrupt (same key, 3 facets)
① the controller writessurge-pair protection + owned marker	✓
② excluded from candidate selection (reads)#170	✓
③ Karpenter honors it (Drift)☁ external	✓
Robustness & other
Resume across leader changecontinues from annotations alone	✓
placeholder preemptionvictim / preemptionPolicy=Never	✓
zonal-EBS (PV) reattach☁ partialstateful workloads · CSI/AWS	✓
Forceful fallback (v0.4+ / #156, #157, #170)
surge-less forceful fallback#156	✓
earliest-deadline order#157	✓

Column legend — 0 core surge · A zonal-EBS · B limits · C readyTimeout · D backstop · E expired · F confinement · G PDB · H dnd-write · I R6 soak · J absorb · K leader · L window · M-A preemption · M-B rollback · N dnd-honor · O forceful-fallback(+#157/#170)

Overlap & containment summary

D ⊂ I backstop

I subsumes D. D is a static snapshot of the backstop metric, while I demonstrates the same property dynamically across a continuous rotation. Proving R6 via I alone is sufficient.

For re-validation, keep I and fold D away (note the static value alongside I's row).

C ≈ M-B rollback

Both verify the same rollback cleanup (state=failed, retry++, uncordon, surge teardown). The only difference is the trigger (C = node not Ready / M-B = placeholder stuck Pending).

Either one alone is enough for the rollback assertions (M-A's preemption is distinct, so keep it).

N · A (partial) external guarantees

N (Karpenter honoring do-not-disrupt) and A's EBS reattach are more a check of Karpenter/AWS's own guarantees (de-risking) than of this controller's logic.

Can be deprioritized under cost pressure (on the assumption that the docs can be trusted).

do-not-disrupt is not a "duplicate" but 3 facets: ① writes it (H) / ② reads and excludes it (#170, O) / ③ Karpenter honors it (N) are separate layers, so this is not redundancy. But since they all revolve around the same key, understanding them together makes the blast radius of a change easier to read.

Rough minimal-cover set: unique = 0, A, B, E, F, G, J, K, L, M-A, O. To fold overlaps: I (subsumes D) plus either C or M-B, with N optional. O is multi-capability (the #156 core plus #157 and #170; it also passes through backstop/expired/PDB/surge secondarily).

Source: test/e2e/eks-automode/SCENARIOS.md (procedure) and VALIDATION.md (observed evidence per scenario), spec §7.2. ●/◯ reflect a judgment call about primary vs. secondary focus, not an exhaustive enumeration of every secondary pass-through.