Tight-race expireAfter Soak — Scenario P

What this validates

A 12-hour real-EKS soak where leadTime genuinely races expireAfter. Demonstrates: (1) the graceful surge always finishes first (expired stays 0), and (2) forceful fallback fires deterministically when a graceful surge no longer fits.

The pool ran with fixed expireAfter: 2h12m — just above derived leadTime of 1h12m — under sub-daily windows (48 × 30 min/day). This validates §3.2 live. See §7.2 for the assumptions validated, Scenario O for the earlier forceful-fallback validation this extends, and the runbook for metric definitions.

Run: 2026-07-14T14:20:29Z (T0) → +12h, EKS Auto Mode, K8s 1.36, us-west-2. Canonical record: test/e2e/eks-automode/VALIDATION.md (§ "Run: 2026-07-15 — Scenario P").

Verdict: CLEAN PASS (12 PASS; criterion 13 N/A — never exercised).

71/71	0	68.3 min	56 s
Graceful rotations (60 in 12h = 5.0/h)	expired · failure · fallback · short_lead · restarts	Min deadline margin	Epilogue: release → fallback

Derived schedule

Earlier PoCs used daily windows with huge expireAfter (E=336h) — the race was always won by a wide margin. This run puts E at 2h12m and holds a genuinely tight race for 12 hours. surge.forcefulFallback stays armed the whole time (demonstrating quiescence).

Quantity	Value
t_rot (bound)	5m + 5m + 2m = 12m
t_rot_est (forecast)	5m + 5m = 10m
leadTime	2·30m + 12m = 1h12m
ageThreshold (auto)	2h12m − 1h12m = 1h
C (per-window)	ceil(28m / 12m) = 3 (6/h)
Steady load	N=5, 5/h → 83% of forecast (tight)
Expected findings	Exactly 1 warn: RotationSpansNextWindow

Derivation pinned by TestDeriveScenarioPSoak in internal/schedule/schedule_test.go.

Margin picture

From T0 to end of recording (T0+14.1h): 71 rotations, all graceful make-before-break surge onto a newly provisioned node. 60 inside [T0, T_end] (= 5.0/h), 11 more during the epilogue at unchanged ~12-min cadence.

Margin (deadline − completion): 68.3–71.2 min across all 71 (spread < 3 min). No degradation accumulated.

Figure 1 — deadline margin of all 72 rotations

x: elapsed time since T0. y: time still left to the deadline at completion (minutes). Dots right of the T_end rule (end of the 12h observation window; 60 rotations inside it) completed while the attended epilogue ran — cadence and margin unchanged. The single orange dot is the epilogue’s deliberate fallback firing (margin 10.1 min) — the drop from the main-pool band is the whole run in one picture.

graceful surge (main pool, 71) forceful fallback (epilogue, 1)

The 13 criteria

#	Criterion	Verdict
1	expired == 0	PASS
2	success ≈5/h, total ≥ 40	PASS (60 in 12h)
3	Main-pool forceful_fallback == 0	PASS (quiescence)
4	short_lead_nodes == 0	PASS (909 scrapes)
5	Restarts 0, seq contiguous	PASS
6	Load present at every snapshot	PASS (desired=available=ready=5)
7	Config-under-test throughout	PASS (6 gauges × 909 scrapes)
8	Per-rotation margin > 0	PASS (min 68.3m)
9	End census clean	PASS (all 5 claims < A)
10	No unexpected Karpenter disruption	PASS
11	Epi: frozen = candidate-only	PASS (~2h hold)
12	Epi: release fires fallback	PASS (56 s)
13	Abort rule (missed release)	N/A

Full criterion observations — click to expand

#	Observed detail
1	0 for full run + tail-follow + epilogue; no Karpenter Expiration events
2	60 in [T0, T_end] (5.0/h); 71 by end (72 incl. epi); steady ~12m cadence
3	0 the entire run — first demonstration of quiescence
4	max 0 across all 909 scrapes
5	controller restartCount=0 for 12h; 0 gaps, 0 restarts, 0 SCRAPE_ERROR
6	desired=available=ready=5 throughout; no Pending backlog
7	Six derived gauges exact at 909/909 scrapes; both policies Accepted; exactly 1 expected warn
8	min 68.3 / median 70.3 / max 71.2 min (n=71)
9	At T_end all 5 claims aged 11–59m (right-censored); 0 stale, 0 failed
10	Only DisruptionBlocked/Unconsolidatable (budgets + do-not-disrupt suppressing)
11	candidates=1, in_progress=0 held ~2h while frozen
12	See epilogue below (56 s, zero placeholder, main pool undisturbed)
13	Release completed inside bounds; fail-closed verified offline only

Anatomy of a rotation

Each of the 71 rotations followed the same sequence:

1. Claim reaches age 1h (= A) → placeholder Pod created
2. Karpenter provisions new node → placeholder binds → surge node Ready (surgeWait median 34 s, range 23–54 s)
3. Both nodes get do-not-disrupt → old NodeClaim deleted → voluntary drain (drain median 44 s, range 18–83 s)
4. Cleanup → total median 81 s (range 45–131 s). Old node collected by WhenEmpty/60s

Figure 2 — time breakdown of each rotation (surgeWait + drain)

y: seconds. The lower segment is the wait from placeholder creation to the surge node going Ready; the upper segment is the old node’s drain. The design bound on t_rot is 720 s — measured totals are 45–131 s (median 81 s), always under a fifth of it; that headroom is what keeps beating the backstop.

surgeWait (placeholder created → surge node Ready) drain (old claim deleted → drained)

Open the full per-rotation ledger — 72 rows (71 main + 1 epilogue)

Birth/completion are UTC. margin = birth + 2h12m − completion. Every surge target is a freshly provisioned EC2 instance. The epilogue row's "—" (no placeholder, no surge node) is itself the evidence of the surge-less path.

#	claim	mode	birth	done	surgeWait	drain	total	margin (min)	surge node
1	fhx2b	surge	07-14 13:18:50	07-14 14:20:29	40s	44s	83s	70.3	i-073168dd1b078525f
2	xdjh8	surge	07-14 13:30:53	07-14 14:32:30	40s	39s	79s	70.4	i-09dcbe1f5d6d88d7c
3	9xfpf	surge	07-14 13:42:57	07-14 14:44:31	41s	35s	76s	70.4	i-0ea97241df5cd15d2
4	xl8fg	surge	07-14 13:55:00	07-14 14:56:34	24s	48s	73s	70.4	i-0e2c826c7f0d82e1e
5	ksjxd	surge	07-14 14:07:04	07-14 15:09:36	53s	77s	131s	69.5	i-07c3fd68b28570fb0
6	s88b7	surge	07-14 14:19:08	07-14 15:20:46	38s	38s	76s	70.4	i-05701db7c4c26e742
7	bvpc6	surge	07-14 14:31:13	07-14 15:32:27	25s	28s	52s	70.8	i-06610ad280717ecb4
8	4txxs	surge	07-14 14:43:18	07-14 15:45:06	38s	49s	87s	70.2	i-0732ee95c07077219
9	7s7dt	surge	07-14 14:55:23	07-14 15:57:11	27s	59s	86s	70.2	i-028415c25f2709a31
10	wbsfb	surge	07-14 15:07:27	07-14 16:09:07	36s	42s	78s	70.3	i-030b61343e3e0405d
11	m4jzw	surge	07-14 15:19:32	07-14 16:21:26	37s	55s	92s	70.1	i-08665c82484969e3b
12	w4r59	surge	07-14 15:31:37	07-14 16:32:58	37s	22s	59s	70.7	i-0ac7d9f8c7de48fd5
13	8bhbp	surge	07-14 15:43:42	07-14 16:45:26	24s	58s	82s	70.3	i-096f1d8ecdea85ea1
14	vrtvs	surge	07-14 15:55:47	07-14 16:57:47	23s	74s	97s	70.0	i-0c9f26564102a018e
15	zwlds	surge	07-14 16:07:52	07-14 17:09:47	34s	59s	94s	70.1	i-01aebd4e635a7a1ef
16	sh6s8	surge	07-14 16:19:56	07-14 17:22:03	24s	80s	104s	69.9	i-01b49d56eb91cc36c
17	xhbsc	surge	07-14 16:32:01	07-14 17:33:55	38s	53s	91s	70.1	i-054e29bf46d2ddb04
18	hghwx	surge	07-14 16:44:06	07-14 17:45:52	24s	61s	85s	70.2	i-02d851cb5b6421813
19	r284b	surge	07-14 16:56:11	07-14 17:57:52	27s	53s	79s	70.3	i-026b29c8cdf04f427
20	q592n	surge	07-14 17:08:15	07-14 18:09:48	27s	44s	71s	70.5	i-0299e5a36131f1a53
21	rwqpb	surge	07-14 17:20:21	07-14 18:22:29	24s	81s	105s	69.9	i-01362e558662da0a8
22	6gdtf	surge	07-14 17:32:26	07-14 18:33:35	24s	24s	48s	70.8	i-01256b5baaa58ea6d
23	76xn9	surge	07-14 17:44:30	07-14 18:46:28	38s	57s	95s	70.0	i-09cc4576bdb2f9954
24	zw4gm	surge	07-14 17:56:35	07-14 18:58:19	38s	43s	82s	70.3	i-0c8662b78afae7195
25	8bcgn	surge	07-14 18:08:40	07-14 19:10:40	26s	71s	97s	70.0	i-09d5061451bc138b9
26	8hld7	surge	07-14 18:20:45	07-14 19:22:13	37s	29s	66s	70.5	i-0053ad9546c899cf6
27	656fc	surge	07-14 18:32:50	07-14 19:34:25	38s	34s	73s	70.4	i-076c2254a9c78ebd7
28	d4d5d	surge	07-14 18:44:55	07-14 19:47:01	24s	81s	105s	69.9	i-0727f919a26d699b2
29	8nkrs	surge	07-14 18:56:59	07-14 19:58:18	23s	33s	56s	70.7	i-0b563214cfb435736
30	pmlpq	surge	07-14 19:09:04	07-14 20:10:38	37s	33s	70s	70.4	i-07e64b99c9d32e18a
31	6wvqx	surge	07-14 19:21:09	07-14 20:22:30	34s	23s	58s	70.7	i-04a2ca7bb98062b91
32	kmh4m	surge	07-14 19:33:14	07-14 20:34:25	25s	23s	48s	70.8	i-01d64316889e57c5c
33	xdtc9	surge	07-14 19:45:20	07-14 20:46:56	33s	39s	73s	70.4	i-026b484c33d3d619c
34	jqmll	surge	07-14 19:57:24	07-14 20:58:35	24s	22s	46s	70.8	i-084de2d8a4645fd56
35	xsxbk	surge	07-14 20:09:29	07-14 21:11:34	36s	64s	100s	69.9	i-00d71584622976033
36	8tp7t	surge	07-14 20:21:35	07-14 21:24:00	39s	83s	122s	69.6	i-0b502fd445ad0c976
37	pdcjt	surge	07-14 20:33:40	07-14 21:35:18	35s	39s	74s	70.4	i-06e1f6f3806f1ec9c
38	jnrdw	surge	07-14 20:45:45	07-14 21:47:21	24s	49s	73s	70.4	i-0ab92c05b557f1add
39	8hbzb	surge	07-14 20:57:50	07-14 22:01:34	42s	38s	81s	68.3	i-0837901c7b5fb0e3b
40	7thv5	surge	07-14 21:09:55	07-14 22:10:56	37s	23s	60s	71.0	i-02f570d8c5fd7fcca
41	j8xsl	surge	07-14 21:22:00	07-14 22:22:54	24s	28s	52s	71.1	i-003922ea46c3fb04f
42	fkwv7	surge	07-14 21:34:06	07-14 22:35:41	39s	55s	94s	70.4	i-084d041ac9a5866a7
43	fjkq9	surge	07-14 21:46:11	07-14 22:47:10	25s	22s	47s	71.0	i-00d419f6d8287f8d8
44	vsd8c	surge	07-14 22:00:16	07-14 23:02:06	39s	61s	100s	70.2	i-09324bed1c04586d9
45	z9dl8	surge	07-14 22:09:58	07-14 23:11:27	39s	44s	83s	70.5	i-0035175e53982ebb5
46	p9qdf	surge	07-14 22:22:03	07-14 23:23:18	30s	38s	68s	70.8	i-05532c42763792a49
47	hwgp8	surge	07-14 22:34:08	07-14 23:35:10	32s	22s	54s	71.0	i-09fdd03b5776ec630
48	kgkmw	surge	07-14 22:46:24	07-14 23:49:06	52s	54s	106s	69.3	i-0fb0f63a1fa25c348
49	mjwgw	surge	07-14 23:00:29	07-15 00:02:01	40s	48s	88s	70.5	i-067d3e9a600db37df
50	jvjhr	surge	07-14 23:10:08	07-15 00:10:59	24s	22s	46s	71.2	i-0027a501d3d908495
51	b9b96	surge	07-14 23:22:13	07-15 00:23:11	31s	23s	54s	71.0	i-0c08f4d5cd021455d
52	rxf2c	surge	07-14 23:34:17	07-15 00:36:09	25s	80s	105s	70.1	i-020c3d914bdd495cf
53	xv8m2	surge	07-14 23:47:22	07-15 00:48:49	54s	28s	81s	70.5	i-00564c34b6324ae46
54	dxq8p	surge	07-15 00:00:35	07-15 01:03:19	41s	65s	106s	69.3	i-06ca63b87ebb447dc
55	kbm2x	surge	07-15 00:10:14	07-15 01:11:15	34s	22s	56s	71.0	i-002e0c626c9965415
56	lnlxb	surge	07-15 00:22:20	07-15 01:23:16	27s	28s	55s	71.1	i-07ed6412f48b14708
57	kgj2d	surge	07-15 00:34:25	07-15 01:36:04	25s	73s	97s	70.3	i-084bb7cca02775ea4
58	tr854	surge	07-15 00:47:30	07-15 01:49:04	24s	69s	93s	70.4	i-0be5128cc2cb31ace
59	d94p8	surge	07-15 01:01:35	07-15 02:02:21	27s	18s	45s	71.2	i-01a1c010ed9bb8abc
60	btlwz	surge	07-15 01:10:20	07-15 02:12:01	31s	34s	64s	70.3	i-0459aab876617973b
61	nssls	surge	07-15 01:22:24	07-15 02:24:09	23s	62s	85s	70.2	i-019f5eed0b9c24fca
62	xwpp9	surge	07-15 01:34:29	07-15 02:36:18	42s	48s	90s	70.2	i-0449ff069770389cb
63	2fmbr	surge	07-15 01:47:34	07-15 02:49:30	38s	58s	96s	70.1	i-03b5247bae327d09a
64	88rjv	surge	07-15 02:01:38	07-15 03:02:57	26s	33s	59s	70.7	i-0adbdb142e012b81d
65	qdh2f	surge	07-15 02:10:59	07-15 03:12:55	35s	62s	97s	70.1	i-07fa42782b1206b04
66	97mm5	surge	07-15 02:22:47	07-15 03:24:26	24s	39s	63s	70.3	i-0eb70b7e8fe3a72c5
67	rbv8l	surge	07-15 02:34:52	07-15 03:36:23	23s	33s	56s	70.5	i-0d032d5175f8d1b1b
68	77m27	surge	07-15 02:47:56	07-15 03:49:49	35s	42s	77s	70.1	i-01b785c0e28e3d678
69	5qwn6	surge	07-15 03:02:01	07-15 04:04:16	36s	62s	98s	69.8	i-0651204cabf6506a8
70	zv9gp	surge	07-15 03:11:20	07-15 04:14:16	42s	79s	121s	69.1	i-07d0545749cf4c983
71	epi-gtx42	forceful-fallback	07-15 02:22:50	07-15 04:24:46	—	48s	—	10.1	—
72	49pqr	surge	07-15 03:23:25	07-15 04:25:24	33s	53s	87s	70.0	i-0507f5b02688986aa

Epilogue — deterministic fallback firing

Why a separate pool

The main run's "never fires" is only half the validation. A frozen single-node pool drives one claim across the boundary deliberately and deterministically. Pool size = 1 because completion can take up to tGP + Buffer = 7m (multi-node evidence is Scenario O).

Time	Event
02:22:50Z	Epi claim gtx42 born (pool created with freeze) → deadline d = 04:34:50Z
03:22:50Z	Becomes candidate at age 1h; frozen → candidates=1, in_progress=0
04:23:50Z	Freeze removed at R (interval search: d−12m < R < d−8m). 11 min left < t_rot 12m
04:24:46Z	56 s after release: claim deleted (drain 48 s < 7m bound). mode=forceful-fallback, no surgeNode

Evidence:

• forceful_fallback_total{nodepool-soak-epi} 0→1
• ForcefulFallback Warning event with spec's message
• Continuous placeholder ledger: zero placeholder for epi claim
• expired stayed 0
• Main pool undisturbed (70→71 across the window, gaps ≤ P + t_rot = 42m)

The abort rule (missed release: if freeze cannot be removed before d−8m, tear down still frozen) was not exercised. Fail-closed behavior verified offline (3 legs, all exit 3).

Reproducing this run

Runbook: test/e2e/eks-automode/SCENARIOS.md § Scenario P.

Pre-flight checks:

go test ./internal/schedule/ -run TestDeriveScenarioPSoak -v  # pins A=1h / C=3 / G=2 / 1 warn
test/e2e/eks-automode/scenarios/soak-analyze-fixture.sh       # analyzer self-check

Operational notes

• JSON logging required (logging.development: false): the analyzer reads zap JSON; console format produces empty ledger
• Check credential lifetime up front — operator credentials expired mid-run, blinding the secondary recorder for 15 min (in-cluster primary stayed gapless)
• Never live-patch expireAfter — recreate the pool (a patch induces Karpenter drift)
• Cost: ≈ $11 (control plane + NAT ~20h, 5 × 2-vCPU nodes × 14h + epilogue). Remember terraform destroy

What this run settles

Spec §7.2 gains two validated rows:

• Under sub-daily windows, leadTime beats a genuinely racing expireAfter for 12h with fallback armed but quiescent
• The moment a claim crosses the point where graceful surge no longer fits, forceful fallback fires deterministically

Remaining open item: genuine same-AZ capacity shortage (ICE) — see roadmap (§6.2) and §7.2.