node-rotation-controller

A Kubernetes controller that gracefully rotates Karpenter-managed nodes inside a maintenance window, before Karpenter's forceful expireAfter fires.

Designed for EKS Auto Mode and any Karpenter v1+ environment.

日本語版: docs/ja/getting-started

---

Quick start

Prerequisites

• A Kubernetes cluster with Karpenter v1+ (karpenter.sh/v1 CRDs served)
• At least one NodePool with expireAfter set
• Helm 3.12+

Install

helm install node-rotation-controller \
  oci://ghcr.io/akashisn/charts/node-rotation-controller \
  --namespace node-rotation-system --create-namespace \
  --set-json 'rotationPolicies=[{
    "spec": {
      "nodePoolSelector": {"matchLabels": {"workload": "api"}},
      "maintenanceWindows": [{
        "timezone": "Asia/Tokyo",
        "days": ["Wed", "Sat"],
        "start": "02:00",
        "end": "06:00"
      }]
    }
  }]'

This installs the controller (2 replicas with leader election), RBAC, the RotationPolicy CRD, and a dedicated negative-priority PriorityClass for the surge placeholder.

Change workload: api to a label your target NodePool carries; adjust maintenanceWindows to your preferred schedule.

Verify

# Check the controller is running
kubectl -n node-rotation-system get pods

# See the derived schedule for your NodePool
kubectl get rotationpolicy -o wide

# Watch rotation metrics (if Prometheus is available)
kubectl port-forward -n node-rotation-system svc/node-rotation-controller-metrics 8080:8080
curl -s localhost:8080/metrics | grep noderotation_

Try the simulator

Before deploying to production, use the policy simulator to visualize which day each node would rotate under your configuration.

---

How it works

Key properties:

• Make-before-break — the replacement node is Ready before the old one is drained
• Never bypasses Karpenter — the controller deletes NodeClaim resources; Karpenter's termination controller drains nodes via the Eviction API (PDBs apply)
• Window-bounded — rotation starts only inside the maintenance window; an in-flight rotation runs to completion past the boundary
• Safe fallback — if the controller is absent, expireAfter still fires as usual (never worse than without the controller)

---

Why this exists

Karpenter classifies node disruption into two categories:

Category	Examples	Disruption Budgets	Pre-provisioned replacement
Graceful	Drift, Consolidation	Applied	Yes (make-before-break)
Forceful	Expiration, Spot Interruption	Not applied	No

Expiration is intentionally forceful — see the upstream forceful-expiration design — so that security patches cannot be blocked by misconfigured PDBs. EKS Auto Mode enforces a 21-day hard cap on node lifetime.

The consequence: nodes will be force-drained at an unpredictable time, Karpenter provisions a replacement only after the drain begins, and this can land in peak business hours. This controller moves that rotation earlier, into your maintenance window, with make-before-break semantics.

---

What it is not

• Not a replacement for Karpenter Consolidation, Drift, or Disruption Budgets — it composes with them
• Not a Spot interruption handler — use AWS Node Termination Handler
• Not an OS-patch reboot tool — use kured
• Not application-side warm-up — readinessProbe / readinessGate / ALB slow start handle that

---

Configuration

The chart renders one RotationPolicy per entry in rotationPolicies. A minimal values file:

rotationPolicies:
  - spec:
      nodePoolSelector:
        matchLabels:
          workload: api
      maintenanceWindows:
        - timezone: Asia/Tokyo
          days: [Wed, Sat]
          start: "02:00"
          end: "06:00"
      # minRotationChances: 2       # K — guaranteed windows before expiry (default 2)
      # surge:
      #   readyTimeout: 15m         # how long to wait for the replacement node
      #   cooldownAfter: 10m        # pause between successive rotations
      #   forcefulFallback:
      #     enabled: false           # opt-in: rotate surge-less if a graceful surge can't finish in time

• Per-NodePool policies. Add more entries to give each NodePool its own window or surge settings.
• Bring your own. Set rotationPolicies: [] and apply your own RotationPolicy objects; see examples/ for ready-to-adapt manifests.
• Full schema. See the specification §5.4 and values.yaml.

---

Compatibility

The compatibility contract is the stable karpenter.sh/v1 CRD surface, not a specific Karpenter minor.

• Runtime: EKS Auto Mode and any Karpenter v1+ cluster
• No cloud APIs: the controller interacts only through Kubernetes API objects (NodeClaim/NodePool, Node, Pod)
• Fail-fast preflight: the controller exits immediately if karpenter.sh/v1 is not served or not readable

See the compatibility policy for the full required-field list.

---

Project status

Pre-1.0 — the CRD schema (v1alpha1) and configuration surface may change between minor releases. What each release changed, and what to do on upgrade, is in the changelog.

The core surge path, forceful fallback, earliest-deadline ordering, and all runtime safety properties are validated end-to-end on real EKS Auto Mode clusters, including a 12-hour tight-race soak. One open item remains before v1.0: a genuine same-AZ capacity shortage (ICE) driving rollback. See the roadmap and validated assumptions.

---

Project layout

├── docs/specification/     Design specification (English)
├── docs/ja/specification/  Japanese translation
├── docs/runbook.md         Production runbook
├── charts/                 Helm chart
├── examples/               Ready-to-adapt RotationPolicy manifests
├── cmd/                    Controller entry point
└── internal/               Reconciler, state machine, surge, window, policy, metrics

---

Development

Requires aqua and make. All tooling is version-pinned in aqua.yaml.

Command	Purpose
make build	Compile the manager binary
make test	Unit tests + envtest smoke test
make lint	golangci-lint
make helm-lint	Lint and render the Helm chart
make helm-rules	Parse the PromQL in the chart's PrometheusRule (promtool)
make docker-build	Build the container image

See CONTRIBUTING.md for the development workflow.

---

Getting involved

This project is pre-1.0 and under active development. Design feedback and implementation contributions are welcome via GitHub Issues and PRs.

See CONTRIBUTING.md for the workflow and CODE_OF_CONDUCT.md for community standards.

---

License

Apache 2.0 — LICENSE.