diff --git a/skills/eks-upgrade-readiness/.skilleval.yaml b/skills/eks-upgrade-readiness/.skilleval.yaml new file mode 100644 index 0000000..686a9c7 --- /dev/null +++ b/skills/eks-upgrade-readiness/.skilleval.yaml @@ -0,0 +1,3 @@ +audit: + ignore: + - STR-016 # README alongside SKILL.md is intentional diff --git a/skills/eks-upgrade-readiness/CHANGELOG.md b/skills/eks-upgrade-readiness/CHANGELOG.md new file mode 100644 index 0000000..428a778 --- /dev/null +++ b/skills/eks-upgrade-readiness/CHANGELOG.md @@ -0,0 +1,178 @@ +# Changelog + +## 2.0.0 (continued) + +### Final PR review corrections + +- Removed the accidental repository-root trigger-test scratch file; the + canonical trigger suite remains `evals/eval_queries.json` inside the skill. +- Reconciled rollback guidance with current EKS behavior: rollback is + conditional for seven days after an eligible upgrade; `ROLLBACK_READINESS` + insights are post-upgrade only and `ERROR`/`UNKNOWN` blocks normal rollback. +- Made kubelet skew bidirectional: no kubelet may be newer than the **current** + control plane, and the target lower bound remains N-3/N-2. Added PF-12 and + an explicit current-control-plane upper-skew eval. +- Dated `addon-version-matrix.md` and `api-deprecations.md` as static fallback + references; live `DescribeAddonVersions` remains the authority. Reframed + kube-proxy exact-minor matching as a post-upgrade recommendation, not a + hard-coded blocker. +- Added explicit discovery and compatibility handling for self-managed VPC + CNI, CoreDNS, and kube-proxy, including custom Corefile/config inspection. +- Added Upgrade Insights freshness semantics: a stale `lastRefreshTime` is + `UNKNOWN`, not PASS. +- Added concrete VPC CNI surge-capacity branches for prefix delegation (/28), + custom networking/ENIConfig, Security Groups for Pods/branch ENIs, and IPv6. +- Documented `PRESERVE` versus `OVERWRITE` add-on conflict handling and the + risk that `OVERWRITE` discards customer configuration. +- Made verdict aggregation mutually exclusive: all PASS = READY; WARNs only = + READY WITH WARNINGS; FAIL = NOT READY; otherwise UNKNOWN = CANNOT DETERMINE. +- Corrected AL2 upstream end-of-life to June 30, 2026; added 5 functional + evals for current-control-plane upper skew, stale Insights, custom + CoreDNS/self-managed core addons, VPC CNI custom configuration, and + read-only add-on conflict resolution (24 total). + +### Follow-up items addressed from the second review round + +Picked off the three lowest-effort items from the "candidate follow-up issue" +list; the remaining five (quorum-aware stateful drain, alternate CNIs/service +meshes, hybrid nodes/Auto Mode depth, GPU/Neuron/Windows accelerated compute, +fleet consistency) are tracked as separate follow-up issues per the +reviewer's suggestion, not folded into this PR. + +1. **Client and CI tooling skew** — Step 2 now checks kubectl (±1 minor per + the upstream Kubernetes version skew policy), eksctl, Helm, and Terraform + AWS provider versions. WARN-level, not a blocker; no hardcoded version + floors since they shift every EKS release. + +2. **Pod Identity awareness** — Step 5 explicitly checks the + `eks-pod-identity-agent` managed addon like any other addon. New "Identity + Migration Considerations" section in `upgrade-troubleshooting.md` contrasts + IRSA (trust policy must be updated per new cluster's OIDC provider) with + Pod Identity (trust policy unchanged, but associations are scoped per + cluster and must be recreated with `create-pod-identity-association`). + +3. **Machine-readable output** — Step 17 now emits an optional structured + JSON verdict alongside the markdown report, with gate IDs matching + `required-check-registry.yaml` prefixes so CI/CD pipelines can gate on + specific check categories, not just the overall verdict. + +4. Added 3 new eval scenarios (19 total) covering CLI tooling skew, Pod + Identity blue-green migration, and machine-readable output requests. + +## 2.0.0 + +Major rewrite addressing PR #48 review feedback. Breaking changes to step +numbering and report format. + +### Must-Fix Items Addressed + +1. **EKS Upgrade Insights as primary signal** — Step 3 now explicitly declares + Insights as the primary authoritative signal. UNKNOWN verdict (not PASS) when + Insights is unavailable or returns no data. Pagination enforced. + +2. **Complete data plane inventory** — New Step 6 inventories ALL node + populations: Managed Node Groups (with DescribeNodegroup details), + self-managed ASGs (via autoscaling API + launch template inspection), + Karpenter (NodePools + EC2NodeClasses), Auto Mode, and Fargate profiles. + Kubelet version map across all nodes with skew validation. + +3. **Live addon API usage** — Step 5 uses `DescribeAddon` + `DescribeAddonVersions` + as primary source. Self-managed addon detection via deployment/Helm scan. + Static addon-version-matrix.md is now explicitly a fallback-only reference. + +4. **AL2→AL2023 comprehensive migration** — New Step 7 covers launch template + analysis, custom AMI detection, user data bootstrap differences (bootstrap.sh + → nodeadm/NodeConfig), cgroup v2 compatibility, IMDSv2 defaults, yum→dnf. + +5. **Mutations separated into Remediation Playbook** — New Step 14 consolidates + ALL mutating commands (helm upgrade, rollout restarts, OVERWRITE addon + updates, PDB patches, Karpenter annotation, CA pause). Clearly marked as + requiring operator approval. Agent never executes these. + +6. **Upgrade ordering refined** — New Step 8 explicitly defines pre-upgrade + alignment (Karpenter, CA, webhooks may need update BEFORE control plane) vs + post-upgrade addon/node-group sequence. + +7. **Deterministic test cases** — evals.json expanded from 6 to 16 scenarios + covering: N-2/N-3 mixed fleet, version skew violations, missing Insights + (AccessDenied), custom bootstrap AL2→AL2023, pagination handling, Karpenter + Drift disabled, GitOps/IaC detection, post-upgrade validation, self-managed + addon detection, pre-upgrade health failure. Assertions enforce UNKNOWN≠PASS. + +### Recommended Additions Addressed + +1. **GitOps/IaC version ownership detection** — New Step 12 detects CF/TF/CDK/ + ArgoCD/Flux/eksctl from cluster/nodegroup tags and routes all remediation + through the owning tool. Never suggests direct CLI if IaC-managed. + +2. **VPC CNI networking modes awareness** — Step 2 subnet check is now mode- + aware: standard, prefix delegation, custom networking, IPv6, Security Groups + for Pods. Includes ENIConfig detection and mode-specific capacity gates. + +3. **Autoscaler pause during rotation** — New Step 13 checks Karpenter + consolidation policy and CA scale-down state. Pause commands in Step 14. + +4. **Pre-upgrade cluster health baseline** — New Step 10 validates all nodes + Ready, no pending CSRs, no crash-looping system pods, DNS resolution working, + metrics-server responding. Any failure blocks the upgrade. + +5. **Post-upgrade functional validation** — New Step 15 provides smoke tests: + DNS, metrics-server, pod scheduling, LB health, IRSA/Pod Identity, baseline + comparison. + +### Additional Improvements + +- Documented required AWS IAM permissions and Kubernetes RBAC ClusterRole +- Added AccessDenied handling protocol (mark UNKNOWN, never PASS) +- Added pagination requirements throughout (ListInsights, ListNodegroups, etc.) +- Helm scanning now checks deployed revision only (not full history) +- CRD checks are vendor-aware and version-aware (compare installed version) +- StatefulSet PVC check corrected (persistentVolumeClaimRetentionPolicy) +- MNG update algorithm documented (for capacity planning accuracy) +- Verdict rules formalized: CANNOT DETERMINE when any gate is UNKNOWN +- Trigger eval expanded to 16 queries (8 positive, 8 negative) +- Skill expanded from 11 to 17 steps +- Version bump to 2.0.0 + +## 1.2.0 + +- Add Helm manifest scanning — detects deprecated APIs in Helm release +- Add version-specific removal gates — AL2 AMI unavailability (≥1.33), + kube-proxy IPVS deprecation (≥1.35/1.36), unmaintained ingress-nginx +- Add service quota headroom checks +- Add StatefulSet safety checks +- Add more comprehensive Karpenter checks +- Add scaled-to-zero workload detection +- Add grading guards with confidence levels +- Add third-party CRD API deprecation checks +- Add cost awareness section +- Add conditional evaluation logic — version-gated checks only when relevant +- Expand pre-upgrade checklist with new checks +- Version bump to 1.2.0 + +## 1.1.0 + +- Add infrastructure prerequisites check (subnet IPs, IAM role, KMS key) +- Add Karpenter Drift and node expiry handling +- Add TopologySpreadConstraints validation +- Add Fargate pod restart requirement (Step 9) +- Add feature-specific removals (Dockershim, PodSecurityPolicy, in-tree storage) +- Add detection tools: kubent, pluto, kubectl-convert, eksup, GoNoGo +- Add blue-green cluster alternative for large upgrades +- Add EKS release calendar and auto-upgrade policy context +- Add EKS Auto Mode awareness +- Add rollback matrix +- Expand from 9 steps to 11 steps +- Align fully with AWS EKS Best Practices Guide cluster-upgrades section + +## 1.0.0 + +- Initial version +- 9-step upgrade readiness assessment workflow +- API deprecation analysis with version-specific removal matrix +- Addon compatibility check against target EKS version +- Node group version skew and AMI readiness validation +- Pod Disruption Budget validation for drain safety +- Capacity planning with surge calculation and reservation guidance +- Structured upgrade plan generation with rollback gates +- Reference documents for API deprecations, addon matrix, capacity planning, and troubleshooting diff --git a/skills/eks-upgrade-readiness/README.md b/skills/eks-upgrade-readiness/README.md new file mode 100644 index 0000000..2d7485a --- /dev/null +++ b/skills/eks-upgrade-readiness/README.md @@ -0,0 +1,212 @@ +# EKS Upgrade Readiness Skill + +A skill for AWS DevOps Agent that performs **read-only** upgrade readiness +assessments for Amazon EKS clusters, aligned with the +[AWS EKS Best Practices Guide — Cluster Upgrades](https://docs.aws.amazon.com/eks/latest/best-practices/cluster-upgrades.html). + +## Purpose + +EKS upgrades can fail or cause downtime when deprecated APIs, incompatible +addons, version-skewed node groups, or misconfigured PDBs are not caught +beforehand. This skill systematically checks every dimension the Best Practices +Guide calls out and produces a READY / NOT READY / READY WITH WARNINGS / +CANNOT DETERMINE verdict with a prioritized remediation plan. + +## Key Capabilities + +- **EKS Upgrade Insights (primary signal)** — UPGRADE_READINESS findings from + the ListInsights/DescribeInsight APIs +- **Infrastructure prerequisites** — subnet IP availability (VPC CNI mode-aware), + IAM role, KMS key, service quotas +- **API deprecation analysis** — maps removed APIs to replacements, scans Helm + stored manifests (deployed revision), vendor-aware CRD checks +- **Addon compatibility (live API)** — validates managed addons via + DescribeAddonVersions, detects self-managed addons via deployment/Helm scan +- **Full data plane inventory** — managed node groups, self-managed ASGs, + Karpenter (Drift, expiry, EC2NodeClass, budgets), Auto Mode, Fargate profiles, + kubelet version map +- **AL2→AL2023 migration** — launch template analysis, custom AMI detection, + bootstrap differences, cgroup v2, IMDSv2 +- **Upgrade ordering** — pre-upgrade alignment (Karpenter/CA/webhooks before CP) + vs post-upgrade sequence +- **PDB and topology spread** — detects drain blockers and availability risks +- **StatefulSet safety** — grace period, PVC retention policy, single-replica risks +- **Pre-upgrade health baseline** — node Ready status, pending CSRs, system pod + health, DNS/metrics baseline +- **Capacity planning** — surge calculation, ODCR/FDCR guidance, blue-green + alternative, autoscaler pause +- **GitOps/IaC detection** — routes remediation through owning tool + (Terraform/CDK/ArgoCD/Flux/eksctl) +- **Post-upgrade validation** — DNS, metrics, scheduling, LB health, IRSA smoke tests +- **Client/CI tooling skew** — kubectl (±1 minor), eksctl, Helm, Terraform provider checks (WARN-level) +- **Pod Identity awareness** — addon version check plus IRSA-vs-Pod-Identity blue-green migration guidance +- **Remediation playbook** — all mutations separated, require operator approval +- **Structured upgrade plan** — ordered execution with rollback gates +- **Machine-readable output** — optional JSON verdict (per-gate status, confidence, + evidence) alongside the markdown report, for CI/CD gating + +## Prerequisites + +### IAM Permissions + +The DevOps Agent role needs read access to EKS, EC2, IAM, and Auto Scaling: + +``` +eks:DescribeCluster +eks:ListClusters +eks:ListInsights +eks:DescribeInsight +eks:ListAddons +eks:DescribeAddon +eks:DescribeAddonVersions +eks:ListNodegroups +eks:DescribeNodegroup +eks:ListFargateProfiles +eks:DescribeFargateProfile +eks:ListUpdates +eks:DescribeUpdate +ec2:DescribeSubnets +ec2:DescribeInstances +ec2:DescribeLaunchTemplateVersions +ec2:DescribeImages +ec2:DescribeCapacityReservations +iam:GetRole +autoscaling:DescribeAutoScalingGroups +autoscaling:DescribeLaunchConfigurations +servicequotas:GetServiceQuota +``` + +### Kubernetes RBAC (if kubectl access available) + +A `ClusterRole` with read-only access to nodes, pods, deployments, statefulsets, +daemonsets, PDBs, configmaps, secrets (Helm), CRDs, CSRs, Karpenter resources, +and ENIConfigs. See the "Required Permissions" section in SKILL.md for the +full `ClusterRole` manifest. `kubectl` access is optional — the assessment +still runs on AWS APIs alone at lower confidence for CRD/Helm/PDB checks. + +### AWS Resources + +- One or more Amazon EKS clusters (any supported version) +- Control plane logging enabled (recommended for post-upgrade debugging) + +## Limitations + +- **EKS clusters only.** Does not cover EKS Anywhere, Outposts, or Local Zones. +- **Read-only by design.** The skill produces recommendations; it never executes + mutating APIs. All mutations are in the Remediation Playbook (Step 14). +- **UNKNOWN ≠ PASS.** Missing data or access denial produces UNKNOWN, never + PASS. The overall verdict cannot be READY while any gate is UNKNOWN. +- **Addon version data may lag.** Static reference table is fallback only — + always prefer live `describe-addon-versions` API. +- **Pagination required.** Large clusters with many node groups or addons + require exhausting API pagination tokens. + +## Agent Types + +- **Chat tasks** — ask for upgrade readiness assessments +- **Evaluation** — periodic upgrade readiness scans + +## Uploading to AWS DevOps Agent + +**Option A: Import from GitHub (recommended)** + +If you have a [GitHub connection configured](https://docs.aws.amazon.com/devopsagent/latest/userguide/connecting-to-cicd-pipelines-connecting-github.html) in your Agent Space, you can import this skill directly from the repository. In the DevOps Agent web app, go to Settings → Add Skill → Import from repository, then +point to `skills/eks-upgrade-readiness`. See [Importing a skill from a repository](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-devops-agent-skills.html#creating-skills) for full instructions. + +> **Note:** You cannot connect the `aws-samples` GitHub organization directly because the GitHub connection setup requires admin rights on the organization. Instead, connect your personal GitHub account and select any repository from it during the connection setup. Once a GitHub connection is established, you can import skills from any public repository, including this one, even if it wasn't selected during the connection setup. + +**Option B: Upload as a zip file** + +1. Zip the `eks-upgrade-readiness/` directory (only including allowed extensions): + +```bash +cd skills +zip -r eks-upgrade-readiness.zip eks-upgrade-readiness/ \ + -i '*.md' '*.json' '*.yaml' '*.yml' \ + -x '*/README.md' '*/.skilleval.yaml' '*/CHANGELOG.md' '*/evals/*' +``` + +2. In the AWS DevOps Agent web app, navigate to the **Skills** page. +3. Click **Add skill** → **Upload skill**. +4. Drag and drop the `eks-upgrade-readiness.zip` file (max 6 MB). +5. Select the agent types: **Chat tasks** and **Evaluation**. +6. Click **Upload**. + +**Option C: Upload via the Asset API** + +Use the DevOps Agent Asset API to programmatically manage skills — useful for CI/CD pipelines or automation workflows. Assign to `CHAT` and `EVALUATION` agent types. See [Managing a skill end-to-end](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-managing-assets.html#managing-a-skill-end-to-end) for the full API workflow. + +## How to Use This Skill + +Describe the task in natural language — you do not need to name the skill. + +### Example Prompts + +``` +"Is my EKS cluster prod-cluster in us-east-1 ready to upgrade to 1.31?" +"Check upgrade readiness for all my EKS clusters" +"What deprecated APIs would break if I upgrade to Kubernetes 1.32?" +"Plan the upgrade of my cluster from 1.30 to 1.31 including node groups" +"Are my addons compatible with EKS 1.31?" +"Will my PDBs block a node group upgrade?" +"I need to upgrade a 50-node cluster — what capacity do I need?" +"Compare in-place vs blue-green strategy for my 200-node cluster" +"My cluster is managed by Terraform — how should I do the upgrade?" +"We use AL2 with custom bootstrap scripts — what breaks going to 1.33?" +"We just upgraded to 1.31 — what should we validate?" +"We use Pod Identity and are planning a blue-green migration — what identity work is needed?" +"Give me the upgrade readiness result as JSON so I can gate our CI/CD pipeline" +``` + +### Modes + +| Mode | Trigger | Behavior | +|------|---------|----------| +| Full assessment | "upgrade readiness", "ready to upgrade" | All 17 steps, scored report | +| Targeted check | "deprecated APIs", "addon compatibility", "PDB check" | One dimension, focused | +| Planning | "upgrade plan", "upgrade runbook" | Execution order with rollback gates | +| Comparison | "blue-green vs in-place" | Strategy recommendation | +| Validation | "post-upgrade check", "validate upgrade" | Smoke tests (Step 15) | + +## Skill Structure + +``` +eks-upgrade-readiness/ +├── SKILL.md # Main skill instructions (17-step workflow) +├── README.md # This file +├── CHANGELOG.md # Version history +├── .skilleval.yaml # Agent Skill Eval config +├── evals/ +│ ├── evals.json # 24 functional evaluation scenarios +│ └── eval_queries.json # Trigger tests (positive and negative) +└── references/ + ├── safety-invariants.md # Hard safety rules, knowledge hierarchy, operation classification + ├── required-check-registry.yaml # All 60+ checks with IDs, categories, and severity + ├── pre-flight-checks.yaml # Blocking vs warning checks, timeouts, soak periods, rollback conditions + ├── api-deprecations.md # K8s API removal schedule by version + ├── addon-version-matrix.md # EKS addon compatibility (static fallback) + ├── capacity-planning.md # FDCR/ODCR surge capacity guidance + ├── upgrade-troubleshooting.md # Tools, feature removals, blue-green + ├── karpenter-checks.md # Full 14-check Karpenter registry (KARP-01 to KARP-14) + ├── pre-drain-safety.md # DRAIN-01 to DRAIN-06 detection and remediation + ├── al2-al2023-migration.md # AL2→AL2023 migration assessment details + └── data-plane-inventory.md # MNG, self-managed, Karpenter, Auto Mode, Fargate inventory commands +``` + +## Safety + +This skill operates in **read-only** mode: + +- No cluster modifications — upgrade actions are recommendations only +- No `update-*`, `delete-*`, or `create-*` API calls +- All mutations isolated in Step 14 Remediation Playbook (operator approval) +- All findings include evidence and specific remediation steps +- The operator reviews the report and decides whether to proceed +- UNKNOWN verdicts prevent false confidence (never marks missing data as PASS) + +## Non-production disclaimer + +> ⚠️ This skill is sample code, not intended for production use without +> additional review and testing. Validate in a non-production environment first. +> Compatibility data and version matrices are point-in-time references — always +> verify with `aws eks describe-addon-versions` for the latest data. diff --git a/skills/eks-upgrade-readiness/SKILL.md b/skills/eks-upgrade-readiness/SKILL.md new file mode 100644 index 0000000..572ade8 --- /dev/null +++ b/skills/eks-upgrade-readiness/SKILL.md @@ -0,0 +1,663 @@ +--- +name: eks-upgrade-readiness +description: Use this skill when a user asks to assess, plan, or validate an + Amazon EKS cluster upgrade. Activate when you see requests mentioning + "EKS upgrade", "Kubernetes version upgrade", "upgrade readiness", + "upgrade plan", "pre-upgrade check", "version skew", "deprecated API", + "addon compatibility", "node group upgrade", "control plane upgrade", + "EKS end of support", "EKS extended support", "Karpenter drift", + "kubelet version skew", or "blue-green cluster migration". This skill + performs a comprehensive pre-upgrade assessment aligned with the AWS EKS + Best Practices Guide covering infrastructure prerequisites, EKS Upgrade + Insights, API deprecations (including Helm stored manifests and third-party + CRDs), addon compatibility (live API + self-managed detection), full data + plane inventory (MNG, self-managed ASGs, Karpenter, Auto Mode, Fargate), + AL2→AL2023 migration, StatefulSet safety, PDB and topology spread + validation, service quota headroom, capacity planning, pre-upgrade cluster + health baseline, and post-upgrade functional validation — then produces a + scored readiness verdict with prioritized remediation (mutations separated + for operator approval) and deterministic test coverage. Do NOT use for ECS, + general EKS troubleshooting unrelated to version upgrades, or EKS + Anywhere/Outpost clusters. +metadata: + author: LearningNewbie + version: "2.0.0" + aws-devops-agent-skills.agent-types: "Chat tasks, Evaluation" + aws-devops-agent-skills.aws-services: "Amazon EKS" + aws-devops-agent-skills.technical-domains: "Containers" +--- + +# EKS Upgrade Readiness + +Assess and plan Amazon EKS cluster upgrades with comprehensive pre-upgrade +validation aligned with the [EKS Best Practices Guide](https://docs.aws.amazon.com/eks/latest/best-practices/cluster-upgrades.html). + +## When to Use + +Activate this skill when the user asks to: +- Check if an EKS cluster is ready to upgrade +- Plan an EKS version upgrade (control plane, node groups, or both) +- Identify deprecated Kubernetes APIs before upgrading +- Validate addon compatibility with a target version +- Assess node group upgrade strategy and capacity requirements +- Review Pod Disruption Budgets or topology spread for upgrade safety +- Understand EKS end-of-support, extended support, or auto-upgrade implications +- Evaluate Karpenter Drift or node expiry upgrade behavior +- Compare in-place vs blue-green upgrade strategies +- Create an upgrade runbook or checklist +- Detect GitOps/IaC version ownership before upgrading + +## Safety First + +**Before doing anything, load `references/safety-invariants.md`.** It defines +the knowledge hierarchy, hard rules, operation classification, and uncertainty +handling. Keep it in context for the entire assessment. + +## Critical Warnings + +- **This skill is read-only.** All commands are `describe*`, `list*`, `get*`. + The agent does NOT execute mutating APIs. Mutations are in Step 14 and + require explicit operator approval. +- **One minor version at a time.** EKS control plane upgrades proceed one + minor version per operation (e.g., 1.30 → 1.31). +- **Version skew policy.** Before planning an upgrade, no kubelet may be newer + than the **current** control plane. For the target version, kubelet may be no + more than N-3 on 1.28+ (N-2 below 1.28). +- **Addons must be upgraded AFTER the control plane** (exceptions in Step 8). +- **Auto-upgrade policy.** Clusters past the 26-month lifecycle will be + auto-upgraded. Proactive upgrade avoids disruption. +- **Control plane rollback (July 2026+).** 7-day rollback window after upgrade. + Conditional, not guaranteed — skill checks eligibility. +- **UNKNOWN ≠ PASS.** Any gate that cannot be assessed MUST be UNKNOWN, never + PASS. Overall verdict cannot be READY while any gate is UNKNOWN. + +## Evidence Completeness + +Uses `references/required-check-registry.yaml` to track checks performed, +skipped, or blocked. EC = checks_performed / total_applicable × 100%. +EC < 50% produces a mandatory warning. + +## Grading and Confidence + +| Level | Meaning | When to Use | +|-------|---------|-------------| +| HIGH (90%+) | Confirmed from authoritative source | EKS Insights API, direct kubectl query, AWS API response | +| MEDIUM (60-89%) | Inferred from available data | Partial kubectl access, version matching heuristics | +| LOW (30-59%) | Limited data, possible gaps | No kubectl, no logging enabled, partial API access | +| UNKNOWN | Cannot determine | Tool unavailable, no data, access denied | + +**False-positive guards:** +- Empty query result ≠ PASS (mark UNKNOWN) +- No kubectl ≠ N/A for everything (AWS APIs still work) +- EKS Insights PASSING ≠ skip other checks (covers a subset only) +- Addon "compatible" ≠ "recommended" +- Pagination not exhausted → confidence LOW + +**Verdict rules (evaluate applicable gates only; `N/A` gates are excluded):** +1. **NOT READY**: one or more applicable gates are FAIL. A known blocker wins + over uncertainty because proceeding is unsafe. +2. **CANNOT DETERMINE**: no gate is FAIL, but one or more applicable gates are + UNKNOWN (including inaccessible, incomplete, stale, or unpaginated data). +3. **READY WITH WARNINGS**: all applicable gates are assessed, none FAIL or + UNKNOWN, and one or more are WARN. +4. **READY**: every applicable gate is PASS. + +Format: `[PASS|FAIL|WARN|UNKNOWN|N/A] (confidence: HIGH) — ` + +## Cost Awareness + +- **EKS Insights API** (Step 3) is free — always use first. +- **CloudWatch Logs Insights** cost ~$0.0076/GB scanned. Default to 60-min windows. +- **Extended support** costs $0.60/cluster/hour — upgrading saves money. +- **Surge nodes** incur temporary EC2 cost during overlap period. + +## Required Permissions + +**AWS IAM** — see README.md "Prerequisites → IAM Permissions" for the full +read-only action list (`eks:Describe*`, `eks:List*`, `ec2:Describe*`, +`autoscaling:Describe*`, `iam:GetRole`, `servicequotas:GetServiceQuota`). + +**Kubernetes RBAC** (only if `kubectl` access is available — the assessment +still runs on AWS APIs alone without it, at lower confidence for CRD/Helm/PDB +checks). Read-only `ClusterRole` covering every `kubectl get`/`describe` used +in this skill: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: eks-upgrade-readiness-readonly +rules: + - apiGroups: [""] + resources: + - nodes + - pods + - configmaps + - secrets + - events + - persistentvolumeclaims + - certificatesigningrequests + verbs: ["get", "list", "watch"] + - apiGroups: ["apps"] + resources: ["deployments", "statefulsets", "daemonsets", "replicasets"] + verbs: ["get", "list", "watch"] + - apiGroups: ["policy"] + resources: ["poddisruptionbudgets"] + verbs: ["get", "list", "watch"] + - apiGroups: ["apiextensions.k8s.io"] + resources: ["customresourcedefinitions"] + verbs: ["get", "list", "watch"] + - apiGroups: ["admissionregistration.k8s.io"] + resources: + - validatingwebhookconfigurations + - mutatingwebhookconfigurations + verbs: ["get", "list", "watch"] + - apiGroups: ["karpenter.sh"] + resources: ["nodepools", "nodeclaims"] + verbs: ["get", "list", "watch"] + - apiGroups: ["karpenter.k8s.aws"] + resources: ["ec2nodeclasses"] + verbs: ["get", "list", "watch"] + - apiGroups: ["crd.k8s.amazonaws.com"] + resources: ["eniconfigs"] + verbs: ["get", "list", "watch"] + - apiGroups: ["storage.k8s.io"] + resources: ["storageclasses", "csinodes"] + verbs: ["get", "list", "watch"] +``` + +Bind with a `ClusterRoleBinding` to the identity the agent assumes (e.g. via +IRSA/Pod Identity or an EKS access entry). `secrets` read access is required +only for the Helm stored-manifest scan (Step 4) — omit that rule and accept +UNKNOWN on Helm checks if a customer's security policy disallows it. + +--- + +## Step 1: Gather Cluster Context + +```bash +aws eks describe-cluster --name --region +``` + +Extract: `cluster.version`, `platformVersion`, `status` (must be ACTIVE), +`kubernetesNetworkConfig`, `logging.clusterLogging` (audit log must be enabled), +`resourcesVpcConfig.subnetIds`, `tags` (IaC ownership detection). + +Determine **target version**: ask user or default to current + 1 minor. +Confirm target is in standard support via the EKS release calendar. + +## Step 2: Verify Infrastructure Prerequisites + +Check these — failures are **BLOCKERs**: + +1. **Subnet IP availability** — need ≥5 IPs per cluster subnet. Mode-aware: + standard IPv4, prefix delegation, custom networking, IPv6, SGP. + Use `aws ec2 describe-subnets` with cluster subnet IDs. +2. **EKS IAM role** — verify role exists with `eks.amazonaws.com` trust. +3. **KMS key** (if encryption enabled) — verify cluster role has key access. +4. **Service quota headroom** — EC2 vCPU (L-1216C47A) and EBS gp3 (L-7A658000) + must have room for surge nodes. Use `aws service-quotas get-service-quota`. + +VPC CNI mode and capacity-input detection: +```bash +kubectl get ds aws-node -n kube-system -o json | jq ' + .spec.template.spec.containers[0].env[] + | select(.name | test("ENABLE_PREFIX_DELEGATION|AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG|ENABLE_POD_ENI|WARM_IP_TARGET|MINIMUM_IP_TARGET|WARM_ENI_TARGET|WARM_PREFIX_TARGET")) + | {name, value}' +``` + +### VPC CNI Surge-Capacity Gate + +Do not treat "mode detected" as capacity validated. First calculate the +managed-node-group surge using `references/capacity-planning.md`, distribute it +by the node group's AZ placement, then verify the relevant subnet/ENI resource +for the selected mode. Record the inputs and calculations as evidence; a missing +mode-specific input is `UNKNOWN`, not PASS. + +| Mode | Required assessment before a node surge | Pass condition | +|------|------------------------------------------|----------------| +| Standard IPv4 | Inspect `WARM_IP_TARGET`, `MINIMUM_IP_TARGET`, and `WARM_ENI_TARGET` on `aws-node`; use node `status.allocatable.pods` and current pod count to calculate the additional secondary-IP demand for every surge node. | Every node subnet has enough free IPv4 addresses for its share of surge nodes, their primary ENIs, and configured warm/allocatable pod-IP demand. | +| Prefix delegation | Confirm `ENABLE_PREFIX_DELEGATION=true`; each IPv4 prefix consumes a `/28` (16 addresses). Calculate required additional prefixes as `ceil(additional_pod_ips / 16)` per affected subnet/AZ. | `floor(availableIpAddressCount / 16)` covers the needed prefixes after allowing for node primary addresses and the configured warm-prefix target. | +| Custom networking | Confirm `AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true`, enumerate `ENIConfig` resources, and map each node AZ to its `spec.subnet` and security groups. | The **ENIConfig pod subnet**, not only the cluster/node subnet, has capacity for the surge pod-IP demand in every used AZ. | +| Security Groups for Pods | Confirm `ENABLE_POD_ENI=true`, inspect trunk ENIs and the instance-type-specific branch-ENI limit for each node type. | Required branch ENIs/pod slots for surge workloads with pod SGs do not exceed the published limit for any instance type. Do not use generic ENI limits as a substitute. | +| IPv6 | Confirm IPv6 family and Nitro-compatible node/Fargate support. IPv6 pod addressing does not consume IPv4 pod IPs, but nodes still need valid ENI/subnet capacity. | Node ENI and subnet capacity cover surge nodes; custom networking is not assumed because it is unsupported with IPv6. | + +```bash +# Custom networking: inspect every AZ-to-pod-subnet mapping +kubectl get eniconfig -o json | jq '.items[] | {name: .metadata.name, subnet: .spec.subnet, securityGroups: .spec.securityGroups}' + +# Security Groups for Pods: inspect trunk/branch interfaces after mode detection +aws ec2 describe-network-interfaces \ + --filters Name=interface-type,Values=trunk,branch \ + --query 'NetworkInterfaces[].{type:InterfaceType,subnet:SubnetId,instance:Attachment.InstanceId,status:Status}' +``` + +### Client and CI Tooling Skew (WARN-level, not a blocker) + +Operator/CI tooling that is too far behind the target version causes confusing +failures during and after the upgrade. Check installed versions where available: + +```bash +kubectl version --client -o json # client minor version +eksctl version # if eksctl-managed +helm version --short # if Helm-managed workloads +``` + +| Tool | Skew Rule | Risk if Violated | +|------|-----------|-------------------| +| kubectl | Must be within ±1 minor of the target `kube-apiserver` version (upstream [Kubernetes version skew policy](https://kubernetes.io/releases/version-skew-policy/#kubectl)) | Unrecognized fields, API calls silently rejected or misinterpreted | +| eksctl | Must support the target EKS version (check release notes for the version that added support) | `eksctl upgrade` commands fail or use stale defaults | +| Helm | 3.8+ recommended for OCI registry support; otherwise not EKS-version-gated | Chart operations may fail independent of the cluster upgrade | +| Terraform AWS provider | Must be new enough to support any target-version-specific attributes in use (e.g. `upgrade_policy`, `compute_config` for Auto Mode) — check the [provider changelog](https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md) for the attribute | `terraform apply` fails validation or silently ignores the new attribute | + +Do not hardcode exact version floors here — they shift every EKS release. +Report the installed version, the rule, and a WARN if it cannot be confirmed +current; never treat "tool not detected" as PASS. + +## Step 3: Check EKS Upgrade Insights + +**Primary authoritative signal.** Always query first. + +```bash +aws eks list-insights --cluster-name \ + --filter '{"categories":["UPGRADE_READINESS"],"kubernetesVersions":[""]}' +aws eks describe-insight --cluster-name --id +``` + +| Status | Gate | Action | +|--------|------|--------| +| ERROR | FAIL | Must fix before upgrade | +| WARNING | WARN | Recommended fix | +| PASSING | PASS | No action | +| UNKNOWN | UNKNOWN | EKS could not evaluate the check; investigate and refresh | +| None returned | UNKNOWN | Continue other checks | + +**Freshness gate:** For every returned summary, capture `lastRefreshTime` and +`lastTransitionTime`, then call `DescribeInsight` to collect the status, +affected resources, and recommendation. Treat data as **stale** when +`lastRefreshTime` is more than 24 hours old at assessment time, or predates a +known relevant workload/addon change. A stale, missing, inaccessible, or +unpaginated insight set is `UNKNOWN`; never reuse it as PASS. The assessment +must not call `StartInsightsRefresh` because this skill is read-only. Instead, +ask the operator to refresh insights through an approved workflow and rerun the +assessment after the refresh completes. + +**Critical:** Insights does NOT cover Helm stored manifests, CRD deprecations, +StatefulSets, Karpenter, service quotas, PDBs, or capacity planning. + +## Step 4: API Deprecation Analysis + +Check version-specific removal gates relevant to user's target: +- ≥1.33: AL2 AMI unavailable (Critical) +- ≥1.35: kube-proxy IPVS deprecated; ≥1.36: removed +- ≥1.25: Dockershim and PodSecurityPolicy removed +- ≥1.23: In-tree EBS provisioner deprecated + +**Helm stored manifests** — the #1 missed blocker. Scan latest deployed +release secrets for deprecated `apiVersion` lines: +```bash +kubectl get secrets -A -l owner=helm,status=deployed +# Decode: base64 -d | base64 -d | gunzip | jq -r '.manifest' +``` + +**Third-party CRD deprecations** — check Istio, cert-manager, Karpenter, +Flux, Argo, Prometheus Operator versions against known deprecation timelines. + +Tools: `kubent`, `pluto detect-all-in-cluster`, `helm mapkubeapis --dry-run`. +See `references/api-deprecations.md` for full removal schedule. + +## Step 5: Addon Compatibility Check + +**Managed addons:** Use live API to build compatibility matrix: +```bash +aws eks list-addons --cluster-name +aws eks describe-addon --cluster-name --addon-name +aws eks describe-addon-versions --addon-name --kubernetes-version +``` + +**Self-managed addons:** First compare `ListAddons` with the actual +`kube-system` workloads. Explicitly detect the three core addons: `aws-node` +(VPC CNI), `coredns`, and `kube-proxy`. If a core component is absent from the +managed-addon inventory but present in-cluster, mark it self-managed/custom and +inspect its image, args, and configuration before assessing target support: + +```bash +kubectl -n kube-system get daemonset aws-node kube-proxy -o json | \ + jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}' +kubectl -n kube-system get deployment coredns -o json | \ + jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}' +kubectl -n kube-system get configmap coredns aws-node -o yaml +``` + +For a custom CoreDNS Corefile, run the Corefile migration check. For VPC CNI, +validate custom environment/config-map values and the mode-specific capacity +gate in Step 2. For kube-proxy, validate its deployed mode and version against +its upstream support policy. Then scan other self-managed components +(aws-load-balancer, external-dns, metrics-server, cluster-autoscaler, +cert-manager, ingress-nginx, argocd, flux); extract image tags and validate +against their K8s support matrices. + +**Pod Identity Agent (`eks-pod-identity-agent`):** Check like any other managed +addon via `DescribeAddon` / `DescribeAddonVersions` — its version gates which +association features are available (e.g. multiple associations per pod, +target IAM role sessions). If not installed but `kubectl get pods -A -o json` +shows service accounts with `eks.amazonaws.com/role-arn` annotations instead, +the cluster is on IRSA, not Pod Identity — note this for blue-green planning +(see `references/upgrade-troubleshooting.md` → Identity Migration Considerations). + +**Upgrade order:** Pre-CP: Karpenter, Cluster Autoscaler, incompatible webhooks. +Post-CP: kube-proxy → vpc-cni → coredns → CSI drivers → others → self-managed. + +See `references/addon-version-matrix.md` for static fallback reference. + +## Step 6: Full Data Plane Inventory + +Inventory ALL node populations. See `references/data-plane-inventory.md` for +complete detection commands. + +- **MNG:** `aws eks list-nodegroups` + `describe-nodegroup` for each +- **Self-managed ASGs:** Find by cluster tag in `describe-auto-scaling-groups` +- **Karpenter:** NodePools, EC2NodeClasses, controller version and health +- **Auto Mode:** Check `cluster.computeConfig.enabled` +- **Fargate:** `aws eks list-fargate-profiles` + describe each +- **Kubelet versions:** `kubectl get nodes` — confirm within skew window + +Version skew requires two independent predicates: + +1. **Current-state upper bound:** no kubelet may be newer than the **current** + control plane (`kubelet_minor <= current_control_plane_minor`). A node + already newer than the current API server is an invalid state and must be + corrected before planning the upgrade. +2. **Target lower bound:** for target 1.X, kubelet must be at least 1.(X-3) + when X>=28, or 1.(X-2) when X<28. + +Any node violating either predicate is a **FAIL**. + +## Step 7: AL2 → AL2023 Migration Assessment + +If AL2 detected and target ≥1.33: **CRITICAL** blocker (EKS releases AL2 +AMIs only through 1.32). If AL2 is detected with a target <1.33: **WARNING** — +upstream Amazon Linux 2 reaches end of life on June 30, 2026. + +Assess: bootstrap method (bootstrap.sh vs nodeadm), custom AMIs, user data +compatibility (yum→dnf, kubelet-extra-args→NodeConfig), cgroup v2 workload +compatibility, IMDSv2 readiness. + +See `references/al2-al2023-migration.md` for full detection commands and +migration strategy. + +## Step 8: Upgrade Ordering and Pre-Upgrade Alignment + +**Pre-CP:** Karpenter (if needed), Cluster Autoscaler (must match target), +admission webhooks with `failurePolicy: Fail`, custom controllers using +deprecated APIs. + +**Post-CP:** Standard addon and node group upgrade order (Step 5). + +Webhook check: +```bash +kubectl get validatingwebhookconfigurations -o json | jq '.items[] | select(.webhooks[].failurePolicy == "Fail")' +kubectl get mutatingwebhookconfigurations -o json | jq '.items[] | select(.webhooks[].failurePolicy == "Fail")' +``` + +## Step 9: PDB, Topology Spread, and Workload Safety + +**PDB blockers:** `maxUnavailable: 0`, `minAvailable` == replicas, orphaned PDBs: +```bash +kubectl get pdb -A -o json | jq '.items[] | select(.status.disruptionsAllowed == 0)' +``` + +**Pre-drain safety (DRAIN-01 to DRAIN-06):** Bare pods, emptyDir data loss, +custom finalizers, EBS AZ-pinning, webhook deadlock, CoreDNS SPOF. +See `references/pre-drain-safety.md` for full detection commands. + +**TopologySpreadConstraints:** Flag multi-replica deployments without topology spread. + +**StatefulSet safety:** Check `terminationGracePeriodSeconds != 0`, PVC retention +policy, single-replica without PDB, update strategy. + +**Scaled-to-zero workloads:** Detect and flag for separate validation. + +## Step 10: Pre-Upgrade Cluster Health Baseline + +Confirm healthy steady state before upgrade. Failures compound on unhealthy clusters. + +- **Node health:** All nodes Ready, no MemoryPressure/DiskPressure/PIDPressure +- **Pending CSRs:** Indicate node registration issues +- **Crash-looping system pods:** Check kube-system, monitoring, ingress namespaces +- **Metrics and DNS baseline:** Verify metrics-server and CoreDNS responding + +Record baselines for post-upgrade comparison. + +## Step 11: Fargate Considerations + +Fargate pods upgrade when redeployed after CP upgrade. All Fargate pods must be +restarted post-upgrade. Restart command is in Step 14 (mutation, requires approval). + +## Step 12: Management Plane and IaC Ownership Detection + +Detect management method to route remediation correctly: + +| Detection | Management Plane | Mutation Routing | +|-----------|-----------------|-----------------| +| ACK CRD + Cluster CR | ACK | Patch ACK Cluster CR | +| ACK CR with `kro.run/owned` | KRO over ACK | Patch kro instance | +| Tags: `terraform:*` | Terraform | Update .tf, `terraform apply` | +| Tags: `aws:cloudformation:*` | CloudFormation | Update template, stack update | +| Tags: `aws:cdk:*` | CDK | Update construct, `cdk deploy` | +| Tags: `eksctl.cluster.k8s.io/*` | eksctl | Update config, `eksctl upgrade` | +| Labels: `argocd.argoproj.io/*` | ArgoCD | Update Git source, sync | +| Labels: `kustomize.toolkit.fluxcd.io/*` | Flux | Update Git source, reconcile | +| Tags: `pulumi:*` | Pulumi | Update program, `pulumi up` | +| None found | unknown | Block mutations until confirmed | + +Route ALL remediation through the owning tool — never suggest direct AWS CLI +when IaC is detected (causes drift). + +## Step 13: Autoscaler Pause During Node Rotation + +During upgrades, autoscalers can interfere with rolling replacement. Check +current Karpenter consolidation config and Cluster Autoscaler scale-down state. +Recommend pausing both before node rotation and re-enabling after completion. + +Pause commands are in Step 14 (mutations, require operator approval). + +## Step 14: Remediation Playbook (Operator Approval Required) + +> ⚠️ **ALL commands in this section are MUTATIONS.** The agent MUST NOT execute +> these — present as a playbook for operator review. + +- **14.1** Helm stored manifest fix: `helm mapkubeapis` + `helm upgrade` +- **14.2** Addon conflict resolution: first capture `DescribeAddon` output and + `configurationValues`; use `--resolve-conflicts PRESERVE` to retain reviewed + custom configuration, or `OVERWRITE` only after approving replacement with + EKS defaults and recording rollback steps. `OVERWRITE` can discard custom + configuration. +- **14.3** Fargate pod restart: `kubectl rollout restart` across namespaces +- **14.4** PDB temporary adjustment: `kubectl patch pdb` (revert after upgrade) +- **14.5** Karpenter pause: `kubectl annotate nodepools --all "karpenter.sh/do-not-disrupt=true"` +- **14.6** CA scale-down pause: patch CA config `scale-down-enabled=false` +- **14.7** Node group upgrade: MNG via `update-nodegroup-version`, Karpenter via + EC2NodeClass patch (drift), self-managed via launch template update + +## Step 15: Post-Upgrade Functional Validation + +Present as validation checklist for operator: +- DNS resolution (nslookup kubernetes.default) +- Metrics server (kubectl top nodes/pods) +- Pod scheduling (run test pod) +- Load balancer health (target group check) +- IRSA / Pod Identity (sts get-caller-identity from pod) +- CoreDNS and kube-proxy pods running +- Compare against Step 10 baseline (node count, no new CrashLoopBackOff) + +## Step 16: Generate Upgrade Plan and Report + +**Execution Order:** +1. Pre-upgrade alignment (Karpenter/CA/webhooks) +2. Pause autoscalers +3. Control plane upgrade (15-40 min) +4. Wait for ACTIVE status +5. kube-proxy → vpc-cni → coredns → other managed addons +6. Self-managed addons +7. Node groups (one at a time, validate between) +8. Karpenter nodes (drift-based) +9. Self-managed nodes (launch template update) +10. Fargate pods (restart) +11. Re-enable autoscalers +12. Post-upgrade validation + +**Rollback Matrix:** + +| Component | Reversibility | Method | +|-----------|--------------|--------| +| Control plane | CONDITIONAL (7-day window) | `aws eks update-cluster-version --kubernetes-version ` | +| Addons | FULL | Downgrade to previous version | +| MNG | PARTIAL | Can halt; completed nodes stay | +| Karpenter nodes | FULL | Revert EC2NodeClass | +| Self-managed | FULL | Revert launch template | +| Fargate | FULL | Redeploy previous config | + +**Rollback eligibility has two phases:** + +- **Pre-upgrade (advisory only):** confirm the planned upgrade is one minor, + document the 7-day window and component rollback order, but do not claim the + future cluster will be eligible. Rollback readiness insights do not exist + until after an eligible upgrade completes. +- **Post-upgrade (authoritative):** while the cluster is ACTIVE and still + inside the 7-day window, run + `aws eks list-insights --cluster-name --filter '{"categories":["ROLLBACK_READINESS"]}'`, + paginate, then `describe-insight` for each entry. `ERROR` blocks a normal + rollback; `UNKNOWN` means EKS could not evaluate readiness and also blocks a + normal rollback. Only `PASSING` insights support an eligible rollback. + +This assessment reports the result but never performs `update-cluster-version` +or a forced rollback. + +## Step 17: Report Format + +``` +## EKS Upgrade Readiness Report +**Cluster:** () +**Current Version:** +**Target Version:** +**Assessment Date:** +**Management Plane:** +**Evidence Completeness:** % (/) +**Overall Readiness:** READY / NOT READY / READY WITH WARNINGS / CANNOT DETERMINE + +### Pre-Upgrade Health Baseline +- [PASS/FAIL] (confidence: HIGH) All nodes Ready +- [PASS/FAIL] (confidence: HIGH) No pending CSRs +- [PASS/FAIL] (confidence: HIGH) No crash-looping system pods +- [PASS/FAIL] (confidence: HIGH) DNS resolution working +- [PASS/FAIL] (confidence: HIGH) Metrics server responding + +### Infrastructure Prerequisites +- [PASS/FAIL] (confidence: HIGH) Subnet IP availability (mode: ) +- [PASS/FAIL] (confidence: HIGH) EKS IAM role valid +- [PASS/FAIL/N/A] (confidence: HIGH) KMS key access +- [PASS/FAIL] (confidence: HIGH) EC2 vCPU quota headroom +- [PASS/FAIL] (confidence: HIGH) EBS volume quota headroom + +### EKS Upgrade Insights +- [PASS/FAIL/UNKNOWN] (confidence: HIGH) + +### Data Plane Inventory +- Managed Node Groups: (versions: ) +- Self-Managed ASGs: (versions: ) +- Karpenter NodePools: (version: ) +- Fargate Profiles: +- Total Nodes: + +### Blockers (must fix) +1. [FAIL] (confidence: HIGH) + +### Warnings (recommended) +1. [WARN] (confidence: MEDIUM) + +### Passing Checks +1. [PASS] (confidence: HIGH) + +### Unknown / Not Assessed +1. [UNKNOWN] + +### Upgrade Plan + + +### Rollback Window +- Rollback eligibility: ELIGIBLE / NOT ELIGIBLE / CHECK AFTER UPGRADE +- Window: 7 days from CP upgrade completion +- Note: Add-ons and node groups must be rolled back BEFORE CP + +### Pre-Drain Risks +- Bare pods (DRAIN-01): +- EmptyDir data loss (DRAIN-02): +- EBS AZ-pinning (DRAIN-04): +- Webhook deadlock (DRAIN-05): +- CoreDNS SPOF (DRAIN-06): + +### Estimated Timeline +- Control plane: ~30 min +- Addons: ~5 min each +- Node groups: ~ min per group +- Total: ~ min +``` + +### Machine-Readable Output + +When the operator asks for a structured result (CI/CD gating, scripted +polling, dashboards), emit this JSON alongside — never instead of — the +markdown report. Every gate in the markdown report must have a matching +entry; the JSON is a serialization of the same evidence, not a summary. + +```json +{ + "cluster": "", + "region": "", + "assessmentTimestamp": "", + "currentVersion": "", + "targetVersion": "", + "overallVerdict": "READY | READY_WITH_WARNINGS | NOT_READY | CANNOT_DETERMINE", + "evidenceCompletenessPct": 0, + "gates": [ + { + "id": "", + "name": "", + "status": "PASS | FAIL | WARN | UNKNOWN | N_A", + "confidence": "HIGH | MEDIUM | LOW", + "evidence": "", + "remediation": "", + "checkedAt": "" + } + ], + "rollback": { + "eligible": true, + "windowExpiresAt": "" + } +} +``` + +`gates[].id` maps 1:1 to the IDs in `references/required-check-registry.yaml` +(prefixes: `PF-` pre-flight, `INFRA-` infrastructure, `NODE-` node assessment, +`ADDON-` addon assessment, `WKLD-` workload assessment, `KARP-` Karpenter, +`DRAIN-` pre-drain safety, `ROLL-` rollback), so a CI pipeline can gate on +specific check categories (e.g. fail only on `NODE-*` or `ADDON-*` FAILs, +warn-only on others) instead of just the overall verdict. `overallVerdict` +follows the same rules as the markdown report — it is never `READY` while +any gate is `UNKNOWN`. + +## References + +See `references/` directory for: +- `safety-invariants.md` — Hard safety rules, knowledge hierarchy, operation classification +- `required-check-registry.yaml` — All 60+ checks with IDs, categories, and severity +- `pre-flight-checks.yaml` — Blocking vs warning checks, timeouts, soak periods, rollback conditions +- `api-deprecations.md` — Full K8s API removal schedule by version +- `addon-version-matrix.md` — EKS addon compatibility per version (static fallback) +- `capacity-planning.md` — FDCR/ODCR and surge capacity guidance +- `upgrade-troubleshooting.md` — Common failures, feature removals, and tools +- `karpenter-checks.md` — Full 14-check Karpenter registry (KARP-01 to KARP-14) +- `pre-drain-safety.md` — DRAIN-01 to DRAIN-06 detection and remediation +- `al2-al2023-migration.md` — AL2→AL2023 migration assessment details +- `data-plane-inventory.md` — MNG, self-managed, Karpenter, Auto Mode, Fargate inventory commands diff --git a/skills/eks-upgrade-readiness/evals/eval_queries.json b/skills/eks-upgrade-readiness/evals/eval_queries.json new file mode 100644 index 0000000..cb9bed3 --- /dev/null +++ b/skills/eks-upgrade-readiness/evals/eval_queries.json @@ -0,0 +1,66 @@ +[ + { + "query": "Which skill would help me assess if my EKS cluster is ready to upgrade? Just name it; do not run it.", + "should_trigger": true + }, + { + "query": "Is there a skill available for planning an EKS Kubernetes version upgrade? Answer yes or no with the skill name; do not execute it.", + "should_trigger": true + }, + { + "query": "Name the skill that covers EKS deprecated API detection before upgrading. Do not run any checks.", + "should_trigger": true + }, + { + "query": "Which skill checks EKS addon compatibility with a target Kubernetes version? Just name it.", + "should_trigger": true + }, + { + "query": "Is there a skill for validating Pod Disruption Budgets before an EKS node group upgrade? Name only.", + "should_trigger": true + }, + { + "query": "Name the skill that helps with capacity planning for EKS cluster upgrades. Do not execute.", + "should_trigger": true + }, + { + "query": "Which skill detects AL2 to AL2023 migration issues before an EKS upgrade? Name only.", + "should_trigger": true + }, + { + "query": "Is there a skill that checks kubelet version skew policy before upgrading EKS? Just name it.", + "should_trigger": true + }, + { + "query": "Which skill helps troubleshoot RDS Aurora database performance issues? Name only.", + "should_trigger": false + }, + { + "query": "Is there a skill for setting up new ECS Fargate services? Just name it.", + "should_trigger": false + }, + { + "query": "Name the skill that handles S3 bucket security configuration audits. Do not run it.", + "should_trigger": false + }, + { + "query": "Which skill investigates Lambda function timeout issues? Name only.", + "should_trigger": false + }, + { + "query": "Is there a skill for VPC networking and subnet configuration? Answer with skill name only.", + "should_trigger": false + }, + { + "query": "Name the skill that covers MSK Kafka consumer lag troubleshooting. Do not execute.", + "should_trigger": false + }, + { + "query": "Which skill helps set up Karpenter from scratch on a new cluster? Name only.", + "should_trigger": false + }, + { + "query": "Is there a skill for EKS pod OOM troubleshooting and memory right-sizing? Just name it.", + "should_trigger": false + } +] diff --git a/skills/eks-upgrade-readiness/evals/evals.json b/skills/eks-upgrade-readiness/evals/evals.json new file mode 100644 index 0000000..4b35bea --- /dev/null +++ b/skills/eks-upgrade-readiness/evals/evals.json @@ -0,0 +1,282 @@ +[ + { + "name": "basic_upgrade_readiness_check", + "prompt": "Check if my EKS cluster prod-cluster in us-east-1 is ready to upgrade to version 1.31", + "assertions": [ + "should call eks describe-cluster or equivalent API", + "should check EKS upgrade insights (ListInsights) as primary signal", + "should produce a readiness verdict (READY, NOT READY, READY WITH WARNINGS, or CANNOT DETERMINE)", + "should mention addon compatibility check via DescribeAddonVersions", + "should inventory all node populations (MNG, self-managed, Karpenter, Fargate)", + "should check kubelet version skew" + ] + }, + { + "name": "deprecated_api_investigation", + "prompt": "What deprecated Kubernetes APIs would break if I upgrade my EKS cluster from 1.30 to 1.31?", + "assertions": [ + "should reference the API deprecation schedule", + "should identify specific APIs removed in 1.31", + "should suggest replacement APIs", + "should recommend checking EKS upgrade insights first", + "should mention Helm stored manifest scanning for deployed revisions", + "should mention third-party CRD deprecation checks" + ] + }, + { + "name": "addon_compatibility_check", + "prompt": "Are my EKS addons compatible with Kubernetes 1.31? My cluster is running vpc-cni v1.16.0, coredns v1.10.1, and kube-proxy v1.29.0", + "assertions": [ + "should use DescribeAddonVersions API to check each addon against target", + "should identify kube-proxy version mismatch (1.29 vs target 1.31)", + "should recommend upgrade order (control plane first, then addons)", + "should mention using describe-addon-versions for live verification", + "should detect self-managed addons not in EKS addon list" + ] + }, + { + "name": "capacity_planning_large_cluster", + "prompt": "I need to upgrade a 50-node EKS cluster across 3 AZs with m6g.2xlarge instances. What capacity do I need?", + "assertions": [ + "should calculate surge node requirements based on maxUnavailable", + "should mention capacity reservation options (ODCR or FDCR)", + "should discuss instance type availability", + "should recommend an upgrade strategy appropriate for cluster size", + "should mention pausing Karpenter consolidation or CA scale-down during rotation" + ] + }, + { + "name": "pdb_validation", + "prompt": "Will my Pod Disruption Budgets block the EKS node group upgrade? I have PDBs with maxUnavailable: 0 on critical services", + "assertions": [ + "should identify maxUnavailable: 0 as a blocker for node drains", + "should recommend adjusting PDBs before upgrade", + "should suggest at least maxUnavailable: 1 during upgrade window", + "should place PDB mutation in remediation section (not inline)", + "should warn this is a read-only assessment, operator must approve changes" + ] + }, + { + "name": "full_upgrade_plan", + "prompt": "Create a complete upgrade plan for my EKS cluster from 1.29 to 1.30 including control plane, addons, and node groups", + "assertions": [ + "should produce a step-by-step ordered plan", + "should list pre-upgrade alignment steps (Karpenter/CA if needed)", + "should include autoscaler pause before node rotation", + "should specify addon upgrade order after control plane", + "should include node group upgrade after addons", + "should include post-upgrade validation smoke tests", + "should include rollback gates between steps", + "should include a pre-upgrade checklist" + ] + }, + { + "name": "n2_version_skew_mixed_fleet", + "prompt": "My EKS cluster is on 1.27 with kubelet versions ranging from 1.25 to 1.27 across nodes. Some are managed node groups, some are self-managed ASGs with custom AMIs. Can I upgrade to 1.28?", + "assertions": [ + "should identify that 1.27 uses N-2 skew policy (kubelet >= 1.25 required)", + "should confirm 1.25 kubelet is within tolerance for current 1.27 CP", + "should warn that after upgrade to 1.28, N-3 policy applies but 1.25 nodes become N-3 (borderline)", + "should inventory both managed and self-managed node populations", + "should flag custom AMIs as requiring manual launch template update", + "should NOT mark missing data as PASS" + ] + }, + { + "name": "n3_version_skew_violation", + "prompt": "My cluster is on 1.30 and I want to go to 1.31. I have some nodes still running kubelet 1.27. Is that okay?", + "assertions": [ + "should identify version skew violation (1.27 kubelet with 1.31 CP = N-4, exceeds N-3)", + "should mark this as FAIL/BLOCKER", + "should recommend upgrading those nodes before control plane upgrade", + "should distinguish between MNG and self-managed remediation paths", + "should NOT produce READY verdict while this violation exists" + ] + }, + { + "name": "missing_insights_unknown_verdict", + "prompt": "Check upgrade readiness for my cluster staging-app. I don't have eks:ListInsights permission.", + "assertions": [ + "should mark Insights gate as UNKNOWN (not PASS, not N/A)", + "should continue with remaining checks despite missing Insights", + "should produce CANNOT DETERMINE overall verdict (not READY)", + "should list the missing permission and recommend granting it", + "should still check infrastructure prerequisites, addons, node groups" + ] + }, + { + "name": "custom_bootstrap_al2_to_al2023", + "prompt": "We use AL2 with custom bootstrap scripts (kubelet-extra-args, custom yum packages, cgroup v1 assumptions). Target is 1.33. Assess readiness.", + "assertions": [ + "should identify AL2 as CRITICAL blocker for 1.33+ (AMIs unavailable)", + "should flag bootstrap.sh to nodeadm migration requirement", + "should identify kubelet-extra-args needing conversion to NodeConfig YAML", + "should flag yum to dnf change", + "should mention cgroup v2 compatibility concerns", + "should mention IMDSv2 default on AL2023", + "should route remediation through launch template update" + ] + }, + { + "name": "pagination_and_access_denied", + "prompt": "I have a cluster with 200 node groups and 50 addons. Check if it's ready to upgrade. Note: I don't have permission to describe subnets.", + "assertions": [ + "should paginate ListNodegroups results (not stop at first page)", + "should paginate ListAddons results", + "should mark subnet IP check as UNKNOWN due to AccessDenied", + "should NOT mark subnet check as PASS or N/A", + "should produce CANNOT DETERMINE overall verdict (UNKNOWN gate exists)", + "should note which specific permission is missing" + ] + }, + { + "name": "karpenter_drift_disabled", + "prompt": "My cluster runs Karpenter v0.32 with Drift feature gate disabled and expireAfter set to Never on all NodePools. Planning to upgrade from 1.30 to 1.31.", + "assertions": [ + "should flag Drift disabled as a problem (nodes won't auto-replace)", + "should flag expireAfter Never as a problem (nodes stay on old AMI forever)", + "should check Karpenter version compatibility with target (v0.32 may not support 1.31)", + "should recommend upgrading Karpenter before control plane (pre-upgrade alignment)", + "should place mutation commands in remediation section only", + "should check disruption budgets" + ] + }, + { + "name": "gitops_iac_detection", + "prompt": "My cluster is managed by Terraform and ArgoCD deploys all workloads. Check upgrade readiness and tell me how to proceed.", + "assertions": [ + "should detect Terraform ownership from cluster tags", + "should detect ArgoCD from addon labels or deployments", + "should route control plane upgrade through terraform apply (not direct aws CLI)", + "should route workload changes through ArgoCD Git source", + "should warn against direct aws eks update-cluster-version (causes drift)", + "should still perform all readiness checks regardless of IaC tool" + ] + }, + { + "name": "post_upgrade_validation", + "prompt": "We just finished upgrading our cluster to 1.31. What should we validate?", + "assertions": [ + "should include DNS resolution test", + "should include metrics-server check", + "should include pod scheduling test", + "should include load balancer health check", + "should include IRSA or Pod Identity validation", + "should recommend comparing against pre-upgrade baseline", + "should check for new CrashLoopBackOff pods" + ] + }, + { + "name": "self_managed_addon_detection", + "prompt": "My cluster shows only vpc-cni and kube-proxy in EKS managed addons, but I know we have AWS Load Balancer Controller, external-dns, and cert-manager installed via Helm. Check addon compatibility for 1.31.", + "assertions": [ + "should use describe-addon-versions for managed addons", + "should detect self-managed addons by scanning deployments/Helm releases", + "should check AWS LBC, external-dns, cert-manager versions against target compatibility", + "should recommend migrating self-managed to EKS managed where possible", + "should flag any self-managed addon with unknown K8s version support" + ] + }, + { + "name": "pre_upgrade_health_failure", + "prompt": "Before upgrading, I ran kubectl get nodes and two nodes show NotReady. Also, coredns pods are in CrashLoopBackOff. Should I proceed with the upgrade?", + "assertions": [ + "should FAIL the pre-upgrade health baseline gate", + "should recommend investigating NotReady nodes before upgrade", + "should flag CrashLoopBackOff coredns as critical (DNS will break)", + "should produce NOT READY verdict", + "should NOT recommend proceeding with the upgrade", + "should suggest remediation for each issue before retrying assessment" + ] + }, + { + "name": "cli_tooling_skew_check", + "prompt": "My local kubectl is v1.26 and I'm upgrading a cluster from 1.30 to 1.31. I also use eksctl and an old Terraform AWS provider pinned to 4.x. Anything I should fix first?", + "assertions": [ + "should flag kubectl v1.26 as outside the +/-1 minor skew window for a v1.31 target (WARN, not a blocker)", + "should recommend upgrading kubectl before running further commands against the upgraded cluster", + "should note eksctl version should be checked against target EKS version support", + "should flag the Terraform AWS provider as likely too old to support target-version-specific attributes", + "should NOT hardcode a specific 'minimum' version number as gospel — should recommend checking current release notes/changelog", + "should NOT block the overall assessment on tooling skew alone (WARN-level only)" + ] + }, + { + "name": "pod_identity_blue_green_migration", + "prompt": "We use EKS Pod Identity (not IRSA) for most workloads and are considering a blue-green cluster migration. What identity work is needed?", + "assertions": [ + "should explain that Pod Identity IAM role trust policies do NOT need to change (cluster-agnostic principal)", + "should explain that Pod Identity associations are scoped per-cluster and must be explicitly recreated with create-pod-identity-association on the new cluster", + "should contrast this with IRSA, where existing role trust policies must be updated to trust the new cluster's OIDC provider", + "should recommend inventorying existing associations via list-pod-identity-associations before cutover", + "should place the recreate-associations step in the Remediation Playbook (mutation, operator approval), not execute it automatically", + "should check the eks-pod-identity-agent addon version via DescribeAddonVersions if Pod Identity Agent add-on is in use" + ] + }, + { + "name": "machine_readable_output_request", + "prompt": "Check upgrade readiness for cluster ci-cluster and give me the result as JSON so I can gate our pipeline on it.", + "assertions": [ + "should produce the standard markdown report AND a structured JSON verdict block", + "should include gate-level entries in the JSON, each with an id matching required-check-registry.yaml prefixes (PF-, INFRA-, NODE-, ADDON-, WKLD-, KARP-, DRAIN-, ROLL-)", + "should include an overallVerdict field using the same READY/NOT_READY/READY_WITH_WARNINGS/CANNOT_DETERMINE values as the markdown report", + "should NOT set overallVerdict to READY if any gate status is UNKNOWN", + "should include confidence and evidence fields per gate, not just a boolean pass/fail" + ] + }, + { + "name": "current_control_plane_upper_skew_violation", + "prompt": "My EKS control plane is 1.31 and I plan to upgrade to 1.32. One self-managed node reports kubelet 1.32 already. Is that okay because 1.32 is my target?", + "assertions": [ + "should identify kubelet 1.32 as newer than the current 1.31 control plane and therefore an existing version-skew violation", + "should mark the current-state upper-bound gate as FAIL", + "should explain that the planned target does not make an already-invalid current state acceptable", + "should recommend correcting the node version before proceeding with the control-plane upgrade", + "should produce NOT READY" + ] + }, + { + "name": "stale_upgrade_insights_unknown", + "prompt": "Assess upgrade readiness for a cluster targeting 1.32. ListInsights returned PASSING entries, but their lastRefreshTime is 36 hours old and the VPC CNI was updated yesterday.", + "assertions": [ + "should mark the Upgrade Insights gate UNKNOWN because lastRefreshTime is older than the skill's 24-hour freshness threshold and predates a relevant addon change", + "should NOT treat stale PASSING insights as proof that the upgrade is safe", + "should recommend an operator-triggered insights refresh followed by a rerun, rather than invoking StartInsightsRefresh during this read-only assessment", + "should produce CANNOT DETERMINE if no independent gate has failed" + ] + }, + { + "name": "custom_coredns_corefile_and_self_managed_core_addons", + "prompt": "Our cluster has no managed CoreDNS, kube-proxy, or VPC CNI addons in ListAddons. aws-node and kube-proxy are DaemonSets, CoreDNS is a Deployment, and the Corefile has custom rewrite and forward rules. Can we upgrade to 1.32?", + "assertions": [ + "should identify aws-node, kube-proxy, and CoreDNS as self-managed or custom core addons rather than assuming ListAddons is complete", + "should inspect their deployed image versions, arguments, and configuration", + "should require a Corefile migration/compatibility check for the custom Corefile", + "should mark compatibility UNKNOWN rather than PASS if target-version support cannot be established from the self-managed components' documentation", + "should NOT recommend an unreviewed OVERWRITE of the custom CoreDNS configuration" + ] + }, + { + "name": "vpc_cni_custom_networking_prefix_and_pod_sg_capacity", + "prompt": "Our EKS 1.31 cluster is moving to 1.32. aws-node has ENABLE_PREFIX_DELEGATION=true, AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true, and ENABLE_POD_ENI=true. We use ENIConfigs in two AZs and need a 6-node surge. Validate network capacity.", + "assertions": [ + "should enumerate ENIConfig resources and evaluate the alternate pod subnets per AZ, not only the cluster subnets", + "should calculate prefix delegation capacity using /28 prefixes and 16 IPv4 addresses per prefix", + "should check instance-type-specific branch-ENI limits for Security Groups for Pods rather than using generic ENI limits", + "should include the surge-node and additional pod-IP demand in the capacity calculation", + "should mark the capacity gate UNKNOWN if required subnet, branch-ENI, or warm-prefix inputs are unavailable" + ] + }, + { + "name": "managed_addon_configuration_conflict", + "prompt": "DescribeAddon shows our VPC CNI has custom configurationValues, and the target add-on update reports ConfigurationConflict. Should I use PRESERVE or OVERWRITE?", + "assertions": [ + "should state that this is an operator-approved mutation and must not be executed by the read-only assessment", + "should require capturing the current DescribeAddon output and configurationValues before selecting a resolution", + "should recommend PRESERVE first when the custom configuration is intentional and target-compatible", + "should explain that OVERWRITE replaces conflicting customer configuration with EKS defaults and can discard custom behavior", + "should require a reviewed backup and rollback plan before recommending OVERWRITE", + "should not present OVERWRITE as the only or default resolution" + ] + } +] diff --git a/skills/eks-upgrade-readiness/references/addon-version-matrix.md b/skills/eks-upgrade-readiness/references/addon-version-matrix.md new file mode 100644 index 0000000..463e152 --- /dev/null +++ b/skills/eks-upgrade-readiness/references/addon-version-matrix.md @@ -0,0 +1,116 @@ +# EKS Addon Version Compatibility Matrix + +> Static fallback reference. Last verified: 2026-08-17. `DescribeAddonVersions` +> for the current cluster and target Kubernetes version is the compatibility +> authority. Do not select a version, or mark a blocker, from this table alone. + +This reference shows historically recommended addon version families per EKS +Kubernetes version. Use it only when live API data is unavailable, and report +that the relevant compatibility gate as `UNKNOWN` rather than `PASS`. + +## Core Addons + +| EKS Version | kube-proxy | vpc-cni | coredns | aws-ebs-csi-driver | +|-------------|------------|---------|---------|-------------------| +| 1.32 | v1.32.x | v1.19+ | v1.12+ | v1.38+ | +| 1.31 | v1.31.x | v1.18+ | v1.11+ | v1.35+ | +| 1.30 | v1.30.x | v1.18+ | v1.11+ | v1.33+ | +| 1.29 | v1.29.x | v1.16+ | v1.11+ | v1.28+ | +| 1.28 | v1.28.x | v1.15+ | v1.10+ | v1.25+ | + +## Addon Upgrade Rules + +### kube-proxy +- Treat matching the target control-plane minor as the normal **post-upgrade + alignment recommendation**, not a hard-coded compatibility rule. +- Use `DescribeAddonVersions` to decide whether the installed version blocks + the target upgrade. The live API result overrides this static reference. +- Upgrade promptly after the control plane when the live API identifies a + target-compatible version. During the transition, respect the Kubernetes + kube-proxy skew policy rather than assuming an exact minor match is the only + valid state. + +### vpc-cni (amazon-vpc-cni-k8s) +- Generally backward compatible across 2-3 minor versions +- New features (prefix delegation, Security Groups for Pods, network policy) + may require specific minimum versions +- Safe to run a newer vpc-cni on an older control plane + +### coredns +- Backward compatible across multiple minor versions +- New EKS versions may require minimum coredns for new features +- Check `coredns:coredns/corefile-migration` for Corefile compatibility + +### aws-ebs-csi-driver +- Version constraints driven by CSI spec version and sidecar compatibility +- Newer versions add volume snapshot, resize, and topology awareness features +- Check for deprecation of `kubernetes.io/aws-ebs` in-tree provisioner + +## How to Check Compatibility + +```bash +# List available versions for an addon on target EKS version +aws eks describe-addon-versions \ + --addon-name vpc-cni \ + --kubernetes-version 1.31 \ + --query 'addons[0].addonVersions[*].{version:addonVersion,default:compatibilities[0].defaultVersion}' \ + --output table + +# Check current addon versions on a cluster +aws eks list-addons --cluster-name --output text +for addon in $(aws eks list-addons --cluster-name --output text --query 'addons[]'); do + echo "$addon: $(aws eks describe-addon --cluster-name --addon-name $addon --query 'addon.addonVersion' --output text)" +done +``` + +## Self-Managed and Custom-Configured Addons + +First compare `aws eks list-addons` with the in-cluster inventory. A core +component that exists in `kube-system` but is absent from `ListAddons` is +self-managed (or replaced) and must not be assumed compatible from EKS managed +addon APIs alone. + +```bash +# Core components that may be self-managed: images, args, and configuration +kubectl -n kube-system get daemonset aws-node kube-proxy -o json | \ + jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}' +kubectl -n kube-system get deployment coredns -o json | \ + jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}' +kubectl -n kube-system get configmap coredns aws-node -o yaml +``` + +- **VPC CNI (`aws-node`)** — inspect image, environment variables, and the + `aws-node` ConfigMap. A custom image or configuration needs its own + compatibility validation; use the mode-aware capacity gate in SKILL.md Step 2. +- **CoreDNS** — inspect image and Corefile. A custom Corefile requires the + CoreDNS migration check for the target release; do not overwrite it without + a reviewed backup and migration plan. +- **kube-proxy** — inspect image, mode/configuration, and DaemonSet arguments. + Use the upstream component documentation plus target-version testing when it + is not EKS managed. +- For other self-managed addons, check the addon's release notes for Kubernetes + version support: Karpenter, AWS Load Balancer Controller, ExternalDNS, + cert-manager, ingress-nginx, Argo CD, and Flux. + +## Upgrade Order + +1. Pre-control-plane compatibility remediation where needed (Karpenter, + Cluster Autoscaler, webhooks, and controllers that must span source/target). +2. Control plane. +3. kube-proxy, VPC CNI, CoreDNS, CSI drivers, other managed addons, then + self-managed addons — use target-compatible versions returned by live APIs + or each self-managed addon's support matrix. + +### Configuration-Conflict Strategy (operator approval required) + +Before updating a managed addon, save `DescribeAddon` output and any +`configurationValues`. Select a conflict mode deliberately: + +- `PRESERVE` retains customer configuration values. Use it first when custom + configuration is intentional and compatible with the target addon. +- `OVERWRITE` replaces conflicting customer configuration with EKS defaults. + It can discard custom behavior; use it only after review, backup, and a + documented rollback plan. + +Neither option is part of the read-only assessment. They belong in the +operator-approved remediation playbook. diff --git a/skills/eks-upgrade-readiness/references/al2-al2023-migration.md b/skills/eks-upgrade-readiness/references/al2-al2023-migration.md new file mode 100644 index 0000000..b4e60a5 --- /dev/null +++ b/skills/eks-upgrade-readiness/references/al2-al2023-migration.md @@ -0,0 +1,162 @@ +# AL2 → AL2023 Migration Assessment + +This document covers the full AL2 to AL2023 migration checks for EKS upgrade +readiness assessments. If any node group or EC2NodeClass uses AL2, and the target +version is 1.33+, this is a **CRITICAL** blocker because EKS stopped releasing +AL2 AMIs after 1.32. For targets < 1.33, flag AL2 usage as a **WARNING**: +upstream Amazon Linux 2 reaches end of life on June 30, 2026. + +## Detection Commands + +```bash +# Check MNG AMI types +aws eks list-nodegroups --cluster-name --query 'nodegroups[]' --output text | \ + xargs -I {} aws eks describe-nodegroup --cluster-name --nodegroup-name {} \ + --query 'nodegroup.amiType' --output text + +# Check Karpenter EC2NodeClass amiFamily +kubectl get ec2nodeclasses -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.amiFamily}{"\n"}{end}' +``` + +## Launch Template Analysis + +```bash +# Get launch template user data for bootstrap method detection +aws ec2 describe-launch-template-versions --launch-template-id \ + --versions --query 'LaunchTemplateVersions[0].LaunchTemplateData.UserData' | \ + base64 -d +``` + +## Bootstrap Differences + +| Aspect | AL2 (bootstrap.sh) | AL2023 (nodeadm) | +|--------|--------------------|--------------------| +| Bootstrap script | `/etc/eks/bootstrap.sh` | `nodeadm` with YAML NodeConfig | +| Config format | CLI flags | `/etc/nodeadm/nodeconfig.yaml` | +| Cgroup driver | cgroup v1 | cgroup v2 (unified) | +| IMDS | v1 enabled by default | v2 only (IMDSv2) | +| Container runtime | containerd (since 1.24) | containerd | +| Kernel | 5.10 | 6.1 | + +## Custom AMI Detection + +If a node group uses a custom AMI (not EKS-optimized): + +```bash +# Check if AMI is EKS-optimized or custom +AMI_ID=$(aws ec2 describe-launch-template-versions --launch-template-id \ + --versions --query 'LaunchTemplateVersions[0].LaunchTemplateData.ImageId' --output text) +aws ec2 describe-images --image-ids $AMI_ID --query 'Images[0].Name' --output text +# EKS-optimized pattern: amazon-eks-node--* +# Custom: anything else +``` + +If custom AMI is detected: +- Flag that automated AMI updates won't work +- User must rebuild their AMI pipeline for AL2023 base +- Check if user data scripts are AL2-specific (yum vs dnf, systemd units, etc.) + +## User Data / Bootstrap Compatibility + +Check user data for AL2-specific patterns that break on AL2023: + +- `--kubelet-extra-args` in bootstrap.sh → must convert to NodeConfig YAML +- `/etc/docker/daemon.json` → irrelevant on AL2023 (containerd only) +- `yum install` → must change to `dnf install` +- `/etc/sysctl.d/` settings → verify cgroup v2 compatibility +- IMDSv1 assumptions → AL2023 defaults to IMDSv2 only + +### Example: Converting bootstrap.sh to nodeadm NodeConfig + +**AL2 (bootstrap.sh):** +```bash +/etc/eks/bootstrap.sh my-cluster \ + --kubelet-extra-args '--max-pods=110 --node-labels=workload=compute' +``` + +**AL2023 (nodeadm NodeConfig):** +```yaml +apiVersion: node.eks.aws/v1alpha1 +kind: NodeConfig +spec: + cluster: + name: my-cluster + apiServerEndpoint: https://... + certificateAuthority: ... + kubelet: + config: + maxPods: 110 + flags: + - --node-labels=workload=compute +``` + +## Cgroup v2 Compatibility + +AL2023 uses cgroup v2 (unified hierarchy). Check for workloads that assume cgroup v1: + +- **Java apps with `-XX:+UseContainerSupport`:** Works on both, but check JDK version. + JDK 15+ has full cgroup v2 support. JDK 8u372+ and 11.0.16+ have partial support. +- **Monitoring agents reading `/sys/fs/cgroup/memory/`:** This is the v1 path; v2 uses + `/sys/fs/cgroup/memory.max` etc. Agents that hardcode v1 paths will break. +- **Custom init containers manipulating cgroup files directly:** Any direct cgroup + filesystem manipulation needs updating. +- **Resource monitoring tools:** cAdvisor < 0.43 has limited cgroup v2 support. + +### Detection + +```bash +# Find pods that mount cgroup filesystem directly +kubectl get pods -A -o json | jq '.items[] | select(.spec.volumes[]?.hostPath.path | test("/sys/fs/cgroup")) | { + ns: .metadata.namespace, + name: .metadata.name, + mounts: [.spec.volumes[] | select(.hostPath.path | test("/sys/fs/cgroup")) | .hostPath.path] +}' + +# Check Java version in common images (requires exec access) +# kubectl exec -- java -version 2>&1 | head -1 +``` + +## IMDSv2 Compatibility + +AL2023 defaults to IMDSv2 only (requires token-based requests). Check for +workloads that use IMDSv1 (simple HTTP GET without token): + +### Common IMDSv1 Patterns That Break + +- AWS SDK versions before credential provider chain update (SDK v1 < 1.11.x) +- Custom scripts using `curl http://169.254.169.254/latest/meta-data/` + without first obtaining a session token +- Legacy EC2 metadata queries without `X-aws-ec2-metadata-token` header + +### Detection + +```bash +# Check launch template IMDS settings +aws ec2 describe-launch-template-versions --launch-template-id \ + --versions --query 'LaunchTemplateVersions[0].LaunchTemplateData.MetadataOptions' + +# Check node group IMDS configuration +aws eks describe-nodegroup --cluster-name --nodegroup-name \ + --query 'nodegroup.launchTemplate' +``` + +### Remediation + +- Update AWS SDK to latest version (all modern SDKs support IMDSv2) +- Replace `curl http://169.254.169.254/...` with token-based access: + ```bash + TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600") + curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/ + ``` +- Use IRSA or Pod Identity instead of IMDS for AWS credentials (preferred) + +## Migration Strategy Summary + +1. **Inventory:** Identify all AL2 node groups and Karpenter EC2NodeClasses +2. **Bootstrap:** Convert all bootstrap.sh args to nodeadm NodeConfig YAML +3. **Custom AMIs:** Rebuild AMI pipelines with AL2023 base +4. **User Data:** Update package managers (yum→dnf), systemd units, scripts +5. **Cgroup v2:** Validate workloads with cgroup v2 compatibility +6. **IMDSv2:** Ensure all metadata access uses tokens or IRSA/Pod Identity +7. **Test:** Deploy AL2023 node group in parallel, migrate workloads gradually +8. **Cutover:** Drain AL2 nodes after validation diff --git a/skills/eks-upgrade-readiness/references/api-deprecations.md b/skills/eks-upgrade-readiness/references/api-deprecations.md new file mode 100644 index 0000000..ab77ad2 --- /dev/null +++ b/skills/eks-upgrade-readiness/references/api-deprecations.md @@ -0,0 +1,92 @@ +# Kubernetes API Deprecations and Removals by EKS Version + +> Static fallback reference. Last verified: 2026-08-17. Confirm the target +> version against live EKS Upgrade Insights and the Kubernetes release notes; +> this table is not a compatibility authority. + +This reference maps deprecated and removed Kubernetes APIs to EKS versions. +Use this to identify workloads that must be updated before upgrading. + +## How to Read This Table + +- **Deprecated**: API still works but emits warnings in audit logs +- **Removed**: API returns 404 — workloads using it will break + +## Removals by Target Version + +### EKS 1.32 (Kubernetes 1.32) + +| API | Replacement | Resources Affected | +|-----|-------------|-------------------| +| `flowcontrol.apiserver.k8s.io/v1beta3` | `flowcontrol.apiserver.k8s.io/v1` | FlowSchema, PriorityLevelConfiguration | + +### EKS 1.29 (Kubernetes 1.29) + +| API | Replacement | Resources Affected | +|-----|-------------|-------------------| +| `flowcontrol.apiserver.k8s.io/v1beta2` | `flowcontrol.apiserver.k8s.io/v1` | FlowSchema, PriorityLevelConfiguration | + +### EKS 1.27 (Kubernetes 1.27) + +| API | Replacement | Resources Affected | +|-----|-------------|-------------------| +| `storage.k8s.io/v1beta1` (CSIStorageCapacity) | `storage.k8s.io/v1` | CSIStorageCapacity | + +### EKS 1.26 (Kubernetes 1.26) + +| API | Replacement | Resources Affected | +|-----|-------------|-------------------| +| `flowcontrol.apiserver.k8s.io/v1beta1` | `flowcontrol.apiserver.k8s.io/v1beta3` | FlowSchema, PriorityLevelConfiguration | +| `autoscaling/v2beta2` | `autoscaling/v2` | HorizontalPodAutoscaler | + +### EKS 1.25 (Kubernetes 1.25) + +| API | Replacement | Resources Affected | +|-----|-------------|-------------------| +| `policy/v1beta1` | `policy/v1` | PodDisruptionBudget, PodSecurityPolicy (removed entirely) | +| `batch/v1beta1` | `batch/v1` | CronJob | +| `discovery.k8s.io/v1beta1` | `discovery.k8s.io/v1` | EndpointSlice | +| `events.k8s.io/v1beta1` | `events.k8s.io/v1` | Event | +| `autoscaling/v2beta1` | `autoscaling/v2` | HorizontalPodAutoscaler | +| `node.k8s.io/v1beta1` | `node.k8s.io/v1` | RuntimeClass | + +### EKS 1.22 (Kubernetes 1.22) + +| API | Replacement | Resources Affected | +|-----|-------------|-------------------| +| `networking.k8s.io/v1beta1` | `networking.k8s.io/v1` | Ingress, IngressClass | +| `rbac.authorization.k8s.io/v1beta1` | `rbac.authorization.k8s.io/v1` | ClusterRole, ClusterRoleBinding, Role, RoleBinding | +| `admissionregistration.k8s.io/v1beta1` | `admissionregistration.k8s.io/v1` | MutatingWebhookConfiguration, ValidatingWebhookConfiguration | +| `apiextensions.k8s.io/v1beta1` | `apiextensions.k8s.io/v1` | CustomResourceDefinition | + +## Detection Methods + +### Via EKS Upgrade Insights (recommended) +``` +aws eks list-insights --cluster-name \ + --filter '{"categories":["UPGRADE_READINESS"]}' +``` + +### Via Kubernetes audit logs (if enabled) +Search for `k8s.io/deprecated=true` annotation in API server audit logs: +``` +fields @timestamp, objectRef.resource, objectRef.apiVersion, user.username +| filter annotations.`k8s.io/deprecated` = "true" +| stats count() by objectRef.apiVersion, objectRef.resource +``` + +### Via kubectl (requires cluster access) +```bash +# Check for deprecated APIs using kubectl +kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis +``` + +## Remediation Pattern + +For each deprecated API usage: +1. Identify the controller/workload using it (from insight or audit log) +2. Update the manifest `apiVersion` field to the replacement +3. Check if the resource spec changed between versions (some fields moved) +4. Apply the updated manifest +5. Verify the workload is healthy +6. Confirm no more deprecation warnings in audit logs diff --git a/skills/eks-upgrade-readiness/references/capacity-planning.md b/skills/eks-upgrade-readiness/references/capacity-planning.md new file mode 100644 index 0000000..b0f3235 --- /dev/null +++ b/skills/eks-upgrade-readiness/references/capacity-planning.md @@ -0,0 +1,124 @@ +# EKS Upgrade Capacity Planning + +This reference covers capacity planning for EKS node group upgrades, +including surge node calculations and Capacity Reservation strategies. + +## Surge Node Calculation + +This follows the actual Amazon EKS managed node group update algorithm +(see [Understand each phase of node updates](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-update-behavior.html)), +not a simplified per-AZ percentage model. There are four phases: + +1. **Setup** — a new launch template version is created and applied to the + ASG. `updateConfig` (`maxUnavailable` or `maxUnavailablePercentage`, capped + at 100 nodes) determines how many nodes can be replaced in parallel. +2. **Scale up** — the ASG's max and desired size are incremented **before** + any old node is touched, so capacity never drops during the upgrade + (default strategy). New nodes land in the same AZs as the nodes they + replace, using EC2 Auto Scaling Availability Zone Rebalancing. +3. **Upgrade** — old nodes are cordoned once a replacement is `Ready`, + drained (15-minute timeout, `PodEvictionFailure` without `--force`), then + terminated after a 60-second post-eviction wait. This repeats in batches + of `maxUnavailable` until every node runs the new launch template version. +4. **Scale down** — the ASG max/desired size is decremented back to the + pre-upgrade value once the rollout completes (skipped if Cluster + Autoscaler is actively scaling the group at that moment). + +### Formula + +The scale-up increment is **not** a per-AZ percentage of existing nodes — it +is the larger of two values, applied once to the whole ASG: + +``` +maxUnavailable_count = min(100, maxUnavailable OR ceil(desired_size * maxUnavailablePercentage / 100)) +Total surge at peak = max(2 * number_of_azs, maxUnavailable_count) +``` + +Because EKS guarantees at least one new node per AZ where old nodes exist +(and up to two per AZ to satisfy AZ Rebalancing), a node group spread across +many AZs can surge by more than `maxUnavailable` even when `maxUnavailable` +is small — plan capacity for `2 * numAZs`, not just `maxUnavailable`. + +### Examples (3 AZs) + +| Desired Size | updateConfig | maxUnavailable_count | Surge at Peak (max of 2×AZ, maxUnavailable) | +|--------------|-------------|----------------------|----------------------------------------------| +| 15 | maxUnavailable: 1 | 1 | 6 (2×3 AZ dominates) | +| 15 | maxUnavailable: 10 | 10 | 10 (maxUnavailable dominates) | +| 30 | maxUnavailablePercentage: 20% | 6 | 6 (maxUnavailable dominates) | +| 150 | maxUnavailablePercentage: 33% | 50 | 50 (maxUnavailable dominates) | + +Use `--force` awareness when planning timelines: if `PodEvictionFailure` +occurs (aggressive PDBs, taint-tolerant deployments), the batch stalls at +the 15-minute drain timeout until an operator intervenes — factor this into +maintenance-window sizing, don't assume `force` is used automatically (it +requires an explicit operator-approved `update-nodegroup-version --force`). + +## Capacity Reservation Strategies + +For large clusters or instance types with limited availability, +use EC2 Capacity Reservations to guarantee surge capacity. + +### On-Demand Capacity Reservations (ODCR) + +- Immediate availability, billed whether used or not +- Best for: short upgrade windows where you want guaranteed capacity +- Create just before upgrade, cancel immediately after + +### Flexible Duration Capacity Reservations (FDCR) + +- Scheduled future capacity, minimum 24-hour duration +- Best for: planned upgrades with known schedules +- Create days in advance, auto-activate at scheduled time + +### Targeting Strategies + +| Strategy | How It Works | When to Use | +|----------|-------------|-------------| +| Open match | Any instance in the AZ consumes slots | Single workload in the AZ | +| Targeted + Resource Group | Only ASG instances consume slots | Multiple workloads in same AZ | + +### Resource Group + ASG Targeting (Recommended) + +```bash +# 1. Create resource group +aws resource-groups create-group \ + --name eks-upgrade-capacity \ + --configuration \ + '{"Type":"AWS::EC2::CapacityReservationPool"}' \ + '{"Type":"AWS::ResourceGroups::Generic","Parameters":[{"Name":"allowed-resource-types","Values":["AWS::EC2::CapacityReservation"]}]}' + +# 2. Add CRs to group +aws resource-groups group-resources \ + --group eks-upgrade-capacity \ + --resource-arns arn:aws:ec2:::capacity-reservation/ + +# 3. Configure ASG to target the group +aws autoscaling update-auto-scaling-group \ + --auto-scaling-group-name \ + --capacity-reservation-specification \ + '{"CapacityReservationTarget":{"CapacityReservationResourceGroupArn":"arn:aws:resource-groups:::group/eks-upgrade-capacity"}}' +``` + +### Important Notes + +- FDCRs start as "targeted" — must switch to "open" after activation OR use resource group +- Cannot modify instance eligibility while instances are consuming the reservation +- If using "open" match, other workloads with the same instance type in the AZ may consume slots +- Calculate reservation size as: existing nodes + surge nodes (all get replaced during rolling update) + +## When NOT to Use Capacity Reservations + +- Instance types with broad availability (t3, m5, m6i in major regions) +- Small clusters (< 10 nodes) where InsufficientCapacity is unlikely +- Clusters using diversified instance types (Karpenter with multiple types) +- Spot-based node groups (CRs are for On-Demand only) + +## Troubleshooting Capacity Issues During Upgrade + +| Symptom | Cause | Resolution | +|---------|-------|-----------| +| `InsufficientInstanceCapacity` during upgrade | AZ lacks capacity for instance type | Use FDCR or switch to open CR match | +| CR shows "Available: 0" but no instances running | Other workloads consumed open CR slots | Switch to targeted + resource group | +| ASG not consuming targeted CR | Launch template missing CR specification | Use resource group targeting on ASG instead | +| FDCR not activating | Still in "Scheduled" state | Wait until start time; cannot modify while scheduled | diff --git a/skills/eks-upgrade-readiness/references/data-plane-inventory.md b/skills/eks-upgrade-readiness/references/data-plane-inventory.md new file mode 100644 index 0000000..7e0b438 --- /dev/null +++ b/skills/eks-upgrade-readiness/references/data-plane-inventory.md @@ -0,0 +1,185 @@ +# Data Plane Inventory + +A complete picture of the data plane is required for upgrade readiness assessment. +Missing any node population means the upgrade plan has blind spots. + +## Managed Node Groups (MNG) + +```bash +# List all node groups (paginate!) +NODEGROUPS=$(aws eks list-nodegroups --cluster-name --query 'nodegroups[]' --output text) + +# For each node group, get full details +for ng in $NODEGROUPS; do + aws eks describe-nodegroup --cluster-name --nodegroup-name "$ng" \ + --query '{name:nodegroup.nodegroupName, version:nodegroup.version, + amiType:nodegroup.amiType, instanceTypes:nodegroup.instanceTypes, + desiredSize:nodegroup.scalingConfig.desiredSize, + maxSize:nodegroup.scalingConfig.maxSize, + updateConfig:nodegroup.updateConfig, + launchTemplate:nodegroup.launchTemplate, + health:nodegroup.health.issues}' +done +``` + +For each MNG, record: +- Current K8s version vs target (version skew check) +- AMI type (AL2, AL2023, BOTTLEROCKET, WINDOWS_CORE, CUSTOM) +- Update strategy (`maxUnavailable` or `maxUnavailablePercentage`) +- Launch template ID and version (for custom AMI detection) +- Health issues (any existing problems block upgrade) + +### MNG Update Algorithm + +When you initiate a node group update, EKS: +1. Creates new nodes with the updated config (up to `maxUnavailable` count) +2. Cordons old nodes +3. Drains old nodes (respects PDBs — will wait/retry for up to 15 min) +4. If drain fails after timeout, ForceEviction applies (pods deleted) +5. Old nodes are terminated +6. Repeats until all nodes are updated + +Understanding this is critical for capacity planning — at peak, you have +`existing_nodes + maxUnavailable` nodes running simultaneously. + +## Self-Managed Node Groups (ASGs) + +Self-managed nodes are EC2 instances in ASGs that joined the cluster via +bootstrap script but aren't tracked by EKS node group APIs. + +### Detection + +```bash +# Find ASGs with EKS cluster tag +aws autoscaling describe-auto-scaling-groups \ + --query "AutoScalingGroups[?Tags[?Key=='kubernetes.io/cluster/' || Key=='eks:cluster-name']].[AutoScalingGroupName,LaunchTemplate.LaunchTemplateId,LaunchTemplate.Version,DesiredCapacity]" \ + --output table + +# Get launch template details for AMI ID +aws ec2 describe-launch-template-versions --launch-template-id \ + --versions --query 'LaunchTemplateVersions[0].LaunchTemplateData.ImageId' + +# Resolve AMI to K8s version +aws ec2 describe-images --image-ids --query 'Images[0].[Name,Description]' +``` + +### Kubelet Version from Self-Managed Nodes + +```bash +# Extract kubelet version from node labels (if kubectl available) +kubectl get nodes -l eks.amazonaws.com/nodegroup!= \ + -o jsonpath='{range .items[*]}{.metadata.name}: {.status.nodeInfo.kubeletVersion}{"\n"}{end}' +``` + +For self-managed nodes: +- Check if AMI is custom or EKS-optimized (from AMI name pattern) +- Identify bootstrap method (see `al2-al2023-migration.md`) +- Note: self-managed nodes require manual launch template updates + +## Karpenter Managed Nodes + +```bash +# Check Karpenter version +kubectl get deploy karpenter -n kube-system -o jsonpath='{.spec.template.spec.containers[0].image}' + +# List NodePools and their config +kubectl get nodepools -o json | jq '.items[] | { + name: .metadata.name, + expireAfter: .spec.disruption.expireAfter, + consolidateAfter: .spec.disruption.consolidateAfter, + budgets: .spec.disruption.budgets +}' + +# List EC2NodeClasses (AMI config) +kubectl get ec2nodeclasses -o json | jq '.items[] | { + name: .metadata.name, + amiFamily: .spec.amiFamily, + amiSelectorTerms: .spec.amiSelectorTerms +}' + +# Check feature gates (Drift) +kubectl get deploy karpenter -n kube-system -o json | jq '.spec.template.spec.containers[0].env[] | select(.name=="FEATURE_GATES")' +``` + +See `karpenter-checks.md` for the full 14-check registry (KARP-01 to KARP-14). + +## EKS Auto Mode + +If `cluster.computeConfig.enabled` is `true`: +- Data plane upgrades happen automatically after control plane upgrade +- Monitor with: `aws eks describe-cluster --name --query 'cluster.computeConfig'` +- Verify PDBs won't block automatic rotation + +### Detection + +```bash +aws eks describe-cluster --name --query 'cluster.computeConfig' +``` + +If Auto Mode is enabled, the node rotation happens without operator action +after the control plane upgrade completes. The key checks become: +- PDBs must allow disruption +- Workloads must tolerate rolling replacement +- No bare pods or emptyDir-dependent workloads on Auto Mode nodes + +## Fargate Profiles + +```bash +# List Fargate profiles +aws eks list-fargate-profiles --cluster-name + +# Describe each profile +aws eks describe-fargate-profile --cluster-name --fargate-profile-name \ + --query '{name:fargateProfile.fargateProfileName, selectors:fargateProfile.selectors, subnets:fargateProfile.subnets}' +``` + +Fargate pods: +- Are automatically upgraded when redeployed after control plane upgrade +- Support the same version skew as managed node groups (N-3 for 1.28+) +- Require explicit restart after CP upgrade (see Remediation Playbook) + +## Kubelet Version Inventory + +Regardless of node management method, confirm actual kubelet versions running: + +```bash +# Full kubelet version map (requires kubectl) +kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}: kubelet={.status.nodeInfo.kubeletVersion}, os={.status.nodeInfo.osImage}, arch={.metadata.labels.kubernetes\.io/arch}{"\n"}{end}' + +# Summary: versions that violate skew policy +kubectl get nodes -o json | jq --arg target "" '.items[] | select(.status.nodeInfo.kubeletVersion | test("v1\\.(\\d+)") | not) | {name: .metadata.name, version: .status.nodeInfo.kubeletVersion}' +``` + +### Version Skew Check + +Evaluate the current control-plane version and target version separately: + +1. **Current-state upper bound:** kubelet must never be newer than the current + control plane (`kubelet_minor <= current_control_plane_minor`). A node at + 1.32 with a current control plane at 1.31 is already invalid, even if 1.32 + is the intended target. +2. **Target lower bound:** for target version 1.X, kubelet must be >= 1.(X-3) + when X >= 28 (N-3), or >= 1.(X-2) when X < 28 (N-2). + +Any node violating either predicate is a **FAIL** — correct it before the +control-plane upgrade proceeds. + +## Inventory Summary Template + +After running all discovery commands, produce a summary: + +``` +### Data Plane Inventory +- Managed Node Groups: (versions: ) + - : , , , / nodes +- Self-Managed ASGs: (versions: ) + - : , , nodes +- Karpenter NodePools: (Karpenter version: ) + - : , expireAfter=, budgets= +- EKS Auto Mode: +- Fargate Profiles: + - : selectors= +- Total Nodes: +- Kubelet Versions: +- Version Skew Violations: nodes outside allowed window +``` diff --git a/skills/eks-upgrade-readiness/references/karpenter-checks.md b/skills/eks-upgrade-readiness/references/karpenter-checks.md new file mode 100644 index 0000000..8e240a5 --- /dev/null +++ b/skills/eks-upgrade-readiness/references/karpenter-checks.md @@ -0,0 +1,136 @@ +# Karpenter Upgrade Readiness Checks (KARP-01 to KARP-14) + +This document contains the full Karpenter check registry and detection commands +for EKS upgrade readiness assessments. + +## Discovery Commands + +```bash +# Check Karpenter version +kubectl get deploy karpenter -n kube-system -o jsonpath='{.spec.template.spec.containers[0].image}' + +# List NodePools and their config +kubectl get nodepools -o json | jq '.items[] | { + name: .metadata.name, + expireAfter: .spec.disruption.expireAfter, + consolidateAfter: .spec.disruption.consolidateAfter, + budgets: .spec.disruption.budgets +}' + +# List EC2NodeClasses (AMI config) +kubectl get ec2nodeclasses -o json | jq '.items[] | { + name: .metadata.name, + amiFamily: .spec.amiFamily, + amiSelectorTerms: .spec.amiSelectorTerms +}' + +# Check feature gates (Drift) +kubectl get deploy karpenter -n kube-system -o json | jq '.spec.template.spec.containers[0].env[] | select(.name=="FEATURE_GATES")' +``` + +## Check Registry + +| ID | Check | Pass Criteria | Severity | +|----|-------|---------------|----------| +| KARP-01 | Version compatibility | Karpenter release supports target K8s version | Critical | +| KARP-02 | Drift enabled | Feature gate on (default since v0.33) | High | +| KARP-03 | expireAfter set | Not `Never` on any NodePool | High | +| KARP-04 | Disruption budgets | At least 1 node can be disrupted (not `nodes: "0"`) | High | +| KARP-05 | AMI not pinned | amiSelectorTerms not pinned to specific AMI ID | High | +| KARP-06 | AMI family valid | amiFamily not AL2 when target >= 1.33 | Critical | +| KARP-07 | Not self-hosted | Controller pods NOT on Karpenter-managed nodes | Critical | +| KARP-08 | NodeClassRef valid | Every NodePool's nodeClassRef points to existing EC2NodeClass | High | +| KARP-09 | Consolidation interference | Short consolidateAfter + active drift = race condition | Medium | +| KARP-10 | Schedule conflict | Disruption budget schedule doesn't block upgrade window | Medium | +| KARP-11 | ExpireAfter timing | NodeClaim expiry won't trigger during upgrade window | Medium | +| KARP-12 | Drift throughput | Estimated time to replace all nodes vs acceptable window | Low | +| KARP-13 | Controller health | All replicas ready, no crash-looping | Critical | +| KARP-14 | v1alpha5 orphans | No leftover Provisioner CRD from incomplete migration | Medium | + +## KARP-07: Self-Hosted Detection (Critical) + +If Karpenter runs on nodes it manages, it may evict itself during drift-based +replacement, halting all further node rotation. + +```bash +# Check if karpenter pods run on karpenter-managed nodes +KARP_NODES=$(kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}') +for node in $KARP_NODES; do + kubectl get node $node -o jsonpath='{.metadata.labels}' | grep -q "karpenter.sh/nodepool" && echo "FAIL: Karpenter self-hosted on $node" +done +``` + +## KARP-08: Dangling NodeClassRef Detection + +Dangling nodeClassRef prevents Karpenter from launching replacement nodes after +drift fires: + +```bash +# Verify all NodePool nodeClassRefs resolve +NODECLASSES=$(kubectl get ec2nodeclasses -o jsonpath='{.items[*].metadata.name}') +kubectl get nodepools -o json | jq --arg ncs "$NODECLASSES" '.items[] | select(.spec.template.spec.nodeClassRef.name as $ref | ($ncs | split(" ") | index($ref)) == null) | {pool: .metadata.name, danglingRef: .spec.template.spec.nodeClassRef.name}' +``` + +## KARP-09: Consolidation Interference + +Short `consolidateAfter` combined with active drift creates a race condition +where consolidation may terminate nodes that drift is trying to replace. + +Check: +```bash +kubectl get nodepools -o json | jq '.items[] | select(.spec.disruption.consolidateAfter != null and .spec.disruption.consolidateAfter != "Never") | {name: .metadata.name, consolidateAfter: .spec.disruption.consolidateAfter}' +``` + +If consolidateAfter is < 30m and drift is active, flag as WARN. + +## KARP-10: Schedule Conflict + +Disruption budget schedules that overlap with the planned upgrade window can +block Karpenter from replacing nodes: + +```bash +kubectl get nodepools -o json | jq '.items[] | select(.spec.disruption.budgets[]?.schedule != null) | {name: .metadata.name, budgets: .spec.disruption.budgets}' +``` + +## KARP-11: ExpireAfter Timing + +If NodeClaims are close to their expiry time, they may trigger replacement +during the upgrade window causing unexpected churn: + +```bash +kubectl get nodeclaims -o json | jq '.items[] | {name: .metadata.name, created: .metadata.creationTimestamp, expireAfter: .spec.expireAfter}' +``` + +## KARP-13: Controller Health + +```bash +kubectl get deploy karpenter -n kube-system -o json | jq '{ + replicas: .spec.replicas, + ready: .status.readyReplicas, + available: .status.availableReplicas, + conditions: .status.conditions +}' + +# Check for crash loops +kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter -o json | jq '.items[] | { + name: .metadata.name, + ready: .status.containerStatuses[0].ready, + restarts: .status.containerStatuses[0].restartCount, + state: .status.containerStatuses[0].state +}' +``` + +## KARP-14: v1alpha5 Orphan Detection + +After migration from Karpenter < 0.33, leftover Provisioner CRDs may remain: + +```bash +# Check for old Provisioner CRD +kubectl get crd provisioners.karpenter.sh 2>/dev/null && echo "WARN: Legacy Provisioner CRD still exists" + +# Check for leftover Provisioner resources +kubectl get provisioners 2>/dev/null && echo "WARN: Legacy Provisioner resources found" + +# Check for old AWSNodeTemplate CRD +kubectl get crd awsnodetemplates.karpenter.k8s.aws 2>/dev/null && echo "WARN: Legacy AWSNodeTemplate CRD still exists" +``` diff --git a/skills/eks-upgrade-readiness/references/pre-drain-safety.md b/skills/eks-upgrade-readiness/references/pre-drain-safety.md new file mode 100644 index 0000000..20e97c0 --- /dev/null +++ b/skills/eks-upgrade-readiness/references/pre-drain-safety.md @@ -0,0 +1,148 @@ +# Pre-Drain Safety Checks (DRAIN-01 to DRAIN-06) + +Beyond PDBs, node drains can fail or cause damage in 6 additional ways. +Check these BEFORE including drain in the upgrade plan. + +## Check Summary + +| ID | Check | Risk | Severity | +|----|-------|------|----------| +| DRAIN-01 | Bare pods (no ownerReferences) | Not rescheduled after eviction; kubectl drain refuses without --force | High | +| DRAIN-02 | Pods with emptyDir volumes | Data lost on drain (--delete-emptydir-data required) | Medium | +| DRAIN-03 | Custom finalizers on pods | Can hang eviction to timeout if finalizer controller is unhealthy | Medium | +| DRAIN-04 | EBS AZ-pinned PVCs | Cross-AZ reschedule strands the volume (Pending forever) | High | +| DRAIN-05 | Fail-closed webhooks on drain-target nodes | Evicting webhook pods deadlocks all further eviction cluster-wide | Critical | +| DRAIN-06 | CoreDNS SPOF | All CoreDNS replicas on same drain batch = cluster-wide DNS outage | Critical | + +## DRAIN-01: Bare Pods + +Pods without ownerReferences are not managed by a controller and will NOT be +rescheduled after eviction. `kubectl drain` refuses to evict them without +`--force`, which can stall automated node replacement. + +```bash +kubectl get pods -A -o json | jq '.items[] | select(.metadata.ownerReferences == null) | { + ns: .metadata.namespace, + name: .metadata.name, + node: .spec.nodeName +}' +``` + +**Remediation:** Wrap bare pods in a Deployment/Job, or acknowledge data loss +and allow `--force` drain. + +## DRAIN-02: Pods with emptyDir Volumes + +emptyDir volumes are ephemeral — data is lost when the pod is evicted. +`kubectl drain` requires `--delete-emptydir-data` flag to proceed. + +```bash +kubectl get pods -A -o json | jq '.items[] | select(.spec.volumes[]?.emptyDir != null) | { + ns: .metadata.namespace, + name: .metadata.name, + node: .spec.nodeName, + emptyDirVolumes: [.spec.volumes[] | select(.emptyDir != null) | .name] +}' +``` + +**Remediation:** Ensure any important data in emptyDir is either ephemeral +(caches, temp files) or backed by external storage. Flag pods using emptyDir +for actual state (e.g., Prometheus WAL without persistent storage). + +## DRAIN-03: Custom Finalizers on Pods + +Pods with custom finalizers can hang eviction indefinitely if the finalizer +controller is unhealthy or slow: + +```bash +kubectl get pods -A -o json | jq '.items[] | select(.metadata.finalizers != null and (.metadata.finalizers | length > 0)) | { + ns: .metadata.namespace, + name: .metadata.name, + finalizers: .metadata.finalizers +}' +``` + +**Remediation:** Verify finalizer controllers are healthy. Consider removing +non-critical finalizers before upgrade or setting an eviction timeout. + +## DRAIN-04: EBS AZ-Pinned PVCs + +EBS volumes are AZ-bound. If a pod is drained to a node in a different AZ, +the PVC cannot be attached — the pod stays Pending forever. + +```bash +# Find EBS PVCs with zone affinity +kubectl get pv -o json | jq '.items[] | select(.spec.csi.driver == "ebs.csi.aws.com") | { + name: .metadata.name, + zone: .spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[] | select(.key == "topology.ebs.csi.aws.com/zone") | .values[0], + claim: .spec.claimRef.namespace + "/" + .spec.claimRef.name +}' + +# Cross-reference with node group AZ distribution +kubectl get nodes -o json | jq '.items[] | { + name: .metadata.name, + zone: .metadata.labels["topology.kubernetes.io/zone"] +}' +``` + +**Remediation:** Ensure node groups span the same AZs as EBS volumes. For +StatefulSets with EBS, use `topologySpreadConstraints` or node affinity to +keep pods in the same AZ as their volumes. During upgrade, ensure surge +nodes are launched in every AZ that has EBS volumes. + +## DRAIN-05: Fail-Closed Webhooks on Drain-Target Nodes + +If ALL endpoint pods for a `failurePolicy: Fail` webhook are on nodes being +drained simultaneously, evicting those pods causes a cluster-wide deadlock — +no further evictions can proceed because the webhook rejects all API calls. + +```bash +# Find fail-closed webhooks and their backing endpoints +kubectl get validatingwebhookconfigurations -o json | jq '.items[].webhooks[] | select(.failurePolicy == "Fail") | {name: .name, service: .clientConfig.service}' +kubectl get mutatingwebhookconfigurations -o json | jq '.items[].webhooks[] | select(.failurePolicy == "Fail") | {name: .name, service: .clientConfig.service}' + +# For each webhook service, check endpoint pod distribution +# Example for a webhook service named "webhook-svc" in namespace "system": +kubectl get endpoints webhook-svc -n system -o json | jq '.subsets[].addresses[].nodeName' + +# Cross-reference with nodes scheduled for drain +``` + +**Remediation:** +- Ensure webhook pods have anti-affinity to spread across nodes/AZs +- Set PDBs on webhook deployments to prevent all replicas from draining simultaneously +- Consider `failurePolicy: Ignore` for non-critical webhooks during upgrade +- Drain webhook-hosting nodes LAST + +## DRAIN-06: CoreDNS Single Point of Failure + +If all CoreDNS replicas end up on nodes in the same drain batch, the entire +cluster loses DNS resolution — new pods can't resolve services, health checks +fail, and cascading failures follow. + +```bash +# Check CoreDNS pod distribution +kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide + +# Detailed node placement +kubectl get pods -n kube-system -l k8s-app=kube-dns -o json | jq '.items[] | { + name: .metadata.name, + node: .spec.nodeName, + ready: .status.conditions[] | select(.type == "Ready") | .status +}' + +# Check if CoreDNS has topology spread or anti-affinity +kubectl get deploy coredns -n kube-system -o json | jq '{ + replicas: .spec.replicas, + topologySpread: .spec.template.spec.topologySpreadConstraints, + affinity: .spec.template.spec.affinity +}' +``` + +**Remediation:** +- Ensure CoreDNS has at least 2 replicas (ideally 3+) +- Add `topologySpreadConstraints` to spread across AZs +- Set a PDB with `minAvailable: 2` (or appropriate for replica count) +- During node rotation, verify CoreDNS pods are rescheduled FIRST before + proceeding with further drains +- Consider running CoreDNS on dedicated system nodes or Fargate diff --git a/skills/eks-upgrade-readiness/references/pre-flight-checks.yaml b/skills/eks-upgrade-readiness/references/pre-flight-checks.yaml new file mode 100644 index 0000000..8502b48 --- /dev/null +++ b/skills/eks-upgrade-readiness/references/pre-flight-checks.yaml @@ -0,0 +1,199 @@ +# EKS Upgrade Readiness — Pre-Flight Checks +# Version: 2.0.0 +# Purpose: Defines blocking vs warning checks with timeouts and soak periods. +# Blocking checks halt the assessment. Warning checks flag but continue. + +blocking_checks: + - id: cluster_active + description: "Cluster status MUST be ACTIVE" + api: "eks:DescribeCluster → status" + failure_action: "STOP — do not proceed with any assessment" + + - id: cluster_not_mid_upgrade + description: "Cluster MUST NOT have a pending platform update in progress" + api: "eks:ListUpdates (filter status=InProgress)" + failure_action: "STOP — wait for current update to complete" + + - id: version_hop_valid + description: "Target version MUST be exactly current + 1 minor version" + api: "eks:DescribeCluster → version comparison" + failure_action: "STOP — EKS requires sequential minor version upgrades" + + - id: version_skew_gate + description: "No kubelet is newer than the current control plane, and all kubelets meet the target lower bound (N-3 for >=1.28, N-2 for <1.28)" + api: "eks:DescribeCluster, eks:DescribeNodegroup (version), kubectl get nodes" + failure_action: "STOP — upgrade or correct node groups first to meet both skew predicates" + + - id: ami_hard_gate + description: "No AL2 AMI usage when target >= 1.33" + api: "eks:DescribeNodegroup → amiType, kubectl get ec2nodeclasses" + failure_action: "STOP — migrate to AL2023 or Bottlerocket before upgrading" + + - id: subnet_minimum_ips + description: "Cluster subnets MUST have at least 5 available IPs" + api: "ec2:DescribeSubnets" + failure_action: "STOP — insufficient IP capacity for control plane upgrade" + + - id: iam_cluster_role_valid + description: "Cluster IAM role MUST exist with correct trust policy" + api: "iam:GetRole for cluster role ARN" + failure_action: "STOP — cluster role missing or insufficient permissions" + + - id: kms_key_valid + description: "If encryption enabled, KMS key MUST exist and be enabled" + api: "eks:DescribeCluster → encryptionConfig" + condition: "Only if cluster has encryptionConfig" + failure_action: "STOP — KMS key invalid, upgrade will fail silently" + + - id: karpenter_controller_healthy + description: "Karpenter controller deployment healthy (if Karpenter detected)" + api: "kubectl get deploy karpenter -n kube-system" + condition: "Only if Karpenter detected" + failure_action: "STOP — unhealthy Karpenter cannot replace nodes post-upgrade" + + - id: karpenter_not_self_hosted + description: "Karpenter pods NOT on Karpenter-managed nodes" + api: "kubectl get pods karpenter + node labels" + condition: "Only if Karpenter detected" + failure_action: "STOP — self-hosted Karpenter may evict itself during drift" + +warning_checks: + - id: insights_warnings + description: "EKS Insights show warnings (not errors) for target version" + api: "eks:ListInsights" + warning_action: "Present findings, recommend remediation before proceeding" + + - id: pdb_blocking + description: "PDBs with zero allowed disruptions detected" + api: "kubectl get pdb -A" + condition: "Only if kubectl available" + warning_action: "Present specific PDBs, recommend adjustment" + + - id: deprecated_apis + description: "Deprecated API usage detected for target version" + api: "eks:ListInsights (primary) or kubectl" + warning_action: "Present API migration requirements" + + - id: self_managed_addons + description: "Self-managed add-ons detected (not via EKS add-on API)" + api: "kubectl get deploy vs eks:ListAddons" + condition: "Only if kubectl available" + warning_action: "Flag for manual version verification" + + - id: webhooks_fail_policy + description: "Webhooks with failurePolicy=Fail targeting broad API groups" + api: "kubectl get validating/mutatingwebhookconfigurations" + condition: "Only if kubectl available" + warning_action: "Present webhook details, assess compatibility" + + - id: control_plane_logging_disabled + description: "Control plane audit logging not enabled" + api: "eks:DescribeCluster → logging" + warning_action: "Recommend enabling before upgrade for debugging" + + - id: helm_stored_deprecated_apis + description: "Helm stored manifests contain deprecated APIs for target" + api: "kubectl get secrets -l owner=helm,status=deployed" + condition: "Only if kubectl + helm available" + warning_action: "Recommend running helm mapkubeapis before upgrade" + + - id: bare_pods_detected + description: "Pods without ownerReferences found on cluster" + api: "kubectl get pods -A" + condition: "Only if kubectl available" + warning_action: "Flag — these pods won't be rescheduled after drain" + + - id: ebs_az_pinning + description: "EBS PVCs zone-pinned with multi-AZ node groups" + api: "kubectl get pvc,pv + ec2:DescribeSubnets" + condition: "Only if kubectl available" + warning_action: "Flag — cross-AZ reschedule will strand these volumes" + + - id: karpenter_consolidation_active + description: "Karpenter consolidation may race with drift during upgrade" + api: "kubectl get nodepools (consolidateAfter)" + condition: "Only if Karpenter detected" + warning_action: "Recommend pausing consolidation during upgrade window" + + - id: management_plane_unknown + description: "Cannot determine cluster management plane (manual/IaC/ACK/KRO)" + api: "eks:DescribeCluster → tags, kubectl get crd" + warning_action: "Ask operator to confirm management approach" + +# Timeout and soak period guidance for upgrade execution +timeouts: + control_plane_upgrade: + expected_minutes: 25 + alert_at_minutes: 40 + escalate_at_minutes: 60 + note: "CRD-heavy clusters (>200 CRDs) may take 60-90 min" + + node_group_upgrade: + per_node_estimate_seconds: 120 + default_minutes: 60 + note: "Based on node count × per-node-time × (1/maxUnavailable)" + + addon_update: + default_minutes: 15 + verification_wait_seconds: 30 + + node_bootstrap: + default_seconds: 900 + gpu_instance_seconds: 1200 + windows_instance_seconds: 1200 + +soak_periods: + post_control_plane_minutes: 15 + post_first_nodegroup_minutes: 30 + between_nodegroup_batches_minutes: 10 + post_addon_update_seconds: 30 + post_all_upgrades_minutes: 60 + note: "Soak periods allow detection of delayed failure modes (OOM, connection exhaustion, cert expiry)" + +# Rollback awareness +rollback: + control_plane: + available_since: "July 2026" + window_days: 7 + conditions: + - "Cluster was upgraded to current version (not created at it)" + - "Within 7 days of upgrade completion" + - "Single version rollback only (N → N-1)" + - "Target rollback version still supported by EKS" + - "Cluster status is ACTIVE" + - "No features enabled at current version incompatible with N-1" + what_rolls_back: + - "API server and control plane components" + - "Platform version" + - "Auto Mode nodes (automatically)" + what_does_not_roll_back: + - "etcd data" + - "Workloads and their configs" + - "Add-ons (manual downgrade required)" + - "Persistent volumes" + - "Managed node groups (manual)" + - "Self-managed nodes (manual)" + - "Fargate pods (must delete/recreate)" + post_upgrade_check: "eks:ListInsights --filter categories=ROLLBACK_READINESS" + initiate_command: "aws eks update-cluster-version --name --kubernetes-version " + force_flag: "--force (bypasses insight checks but NOT prerequisite validations)" + + addons: + reversibility: "full" + method: "aws eks update-addon --addon-version " + window: "indefinite" + + managed_node_groups: + reversibility: "partial" + note: "Can halt mid-upgrade; completed nodes stay at new version" + method: "Cannot downgrade — must recreate node group at old version if needed" + + karpenter_nodes: + reversibility: "full" + method: "Revert EC2NodeClass amiSelectorTerms, delete drifted NodeClaims" + window: "indefinite" + + self_managed_nodes: + reversibility: "full" + method: "Revert launch template version, terminate new nodes" + window: "indefinite" diff --git a/skills/eks-upgrade-readiness/references/required-check-registry.yaml b/skills/eks-upgrade-readiness/references/required-check-registry.yaml new file mode 100644 index 0000000..fb1edc0 --- /dev/null +++ b/skills/eks-upgrade-readiness/references/required-check-registry.yaml @@ -0,0 +1,475 @@ +# EKS Upgrade Readiness — Required Check Registry +# Version: 2.0.0 +# Purpose: Defines all checks for the upgrade readiness assessment. +# Evidence Completeness = checks_performed / total_applicable_checks +# +# Fields: +# id - Unique identifier +# description - What the check verifies +# category - pre_flight | node_assessment | addon_assessment | +# workload_assessment | infrastructure | networking | +# karpenter | pre_drain | rollback +# required - always | if_kubectl | if_helm | if_karpenter | +# if_fargate | if_encrypted | if_custom_ami | +# if_self_managed_nodes | if_auto_mode | if_post_upgrade +# api - Primary API or method used +# severity - critical | high | medium | low (if check fails) + +checks: + + # ── PRE-FLIGHT ──────────────────────────────────────────────────────────── + + - id: PF-01 + description: "Cluster status is ACTIVE" + category: pre_flight + required: always + api: "eks:DescribeCluster → status" + severity: critical + + - id: PF-02 + description: "Cluster version identified, target determined" + category: pre_flight + required: always + api: "eks:DescribeCluster → version" + severity: critical + + - id: PF-03 + description: "Cluster not mid-upgrade (no pending platform update)" + category: pre_flight + required: always + api: "eks:ListUpdates" + severity: high + + - id: PF-04 + description: "Control plane logging enabled (audit + authenticator)" + category: pre_flight + required: always + api: "eks:DescribeCluster → logging" + severity: medium + + - id: PF-05 + description: "EKS Upgrade Insights queried (primary signal)" + category: pre_flight + required: always + api: "eks:ListInsights, eks:DescribeInsight" + severity: critical + + - id: PF-06 + description: "Cluster IAM role valid with correct trust policy" + category: pre_flight + required: always + api: "iam:GetRole, iam:ListAttachedRolePolicies" + severity: critical + + - id: PF-07 + description: "KMS key valid and accessible (if encryption enabled)" + category: pre_flight + required: if_encrypted + api: "eks:DescribeCluster → encryptionConfig" + severity: critical + + - id: PF-08 + description: "All nodes Ready, no pending CSRs, no crash-looping system pods" + category: pre_flight + required: if_kubectl + api: "kubectl get nodes, kubectl get csr, kubectl get pods -n kube-system" + severity: high + + - id: PF-09 + description: "DNS resolution working (CoreDNS healthy)" + category: pre_flight + required: if_kubectl + api: "kubectl get pods -n kube-system -l k8s-app=kube-dns" + severity: high + + - id: PF-10 + description: "Metrics server responding" + category: pre_flight + required: if_kubectl + api: "kubectl top nodes" + severity: low + + - id: PF-11 + description: "Version skew lower-bound gate — kubelets meet target N-3 (1.28+) or N-2 (<1.28)" + category: pre_flight + required: always + api: "eks:DescribeNodegroup (version), kubectl get nodes" + severity: critical + + - id: PF-12 + description: "Current-state upper-bound gate — no kubelet is newer than the current control plane" + category: pre_flight + required: always + api: "eks:DescribeCluster → version, kubectl get nodes → status.nodeInfo.kubeletVersion" + severity: critical + + - id: PF-13 + description: "Management plane detected (self-managed/ACK/KRO/GitOps/unknown)" + category: pre_flight + required: always + api: "eks:DescribeCluster → tags, kubectl get crd clusters.eks.services.k8s.aws" + severity: medium + + # ── INFRASTRUCTURE ──────────────────────────────────────────────────────── + + - id: INFRA-01 + description: "Subnet and VPC CNI mode-specific surge capacity (node/pod subnets, prefixes, or branch ENIs)" + category: infrastructure + required: always + api: "ec2:DescribeSubnets, aws-node env/config, kubectl get eniconfig" + severity: critical + + - id: INFRA-02 + description: "EC2 vCPU quota headroom for surge nodes" + category: infrastructure + required: always + api: "servicequotas:GetServiceQuota (L-1216C47A)" + severity: high + + - id: INFRA-03 + description: "EBS volume quota headroom" + category: infrastructure + required: always + api: "servicequotas:GetServiceQuota (L-7A658000)" + severity: high + + - id: INFRA-04 + description: "VPC CNI mode detected (standard/prefix-delegation/custom-networking/IPv6/SGP)" + category: networking + required: always + api: "eks:DescribeAddon (vpc-cni), kubectl get ds aws-node env vars" + severity: medium + + - id: INFRA-06 + description: "Mode-specific VPC CNI capacity calculation (prefix /28, ENIConfig pod subnet, or branch-ENI limit)" + category: networking + required: if_kubectl + api: "ec2:DescribeSubnets, kubectl get eniconfig, ec2:DescribeNetworkInterfaces" + severity: critical + + - id: INFRA-05 + description: "Security groups allow required control-plane to node communication" + category: networking + required: always + api: "ec2:DescribeSecurityGroups" + severity: high + + # ── NODE ASSESSMENT ─────────────────────────────────────────────────────── + + - id: NODE-01 + description: "Managed node group inventory (all MNGs described)" + category: node_assessment + required: always + api: "eks:ListNodegroups, eks:DescribeNodegroup" + severity: critical + + - id: NODE-02 + description: "Self-managed ASG inventory" + category: node_assessment + required: if_self_managed_nodes + api: "autoscaling:DescribeAutoScalingGroups (filtered by cluster tag)" + severity: high + + - id: NODE-03 + description: "AMI family detection (AL2, AL2023, Bottlerocket, Windows, Custom)" + category: node_assessment + required: always + api: "eks:DescribeNodegroup → amiType" + severity: critical + + - id: NODE-04 + description: "AL2 hard gate check (AL2 unavailable for target >= 1.33)" + category: node_assessment + required: always + api: "eks:DescribeNodegroup → amiType vs target version" + severity: critical + + - id: NODE-05 + description: "Custom launch template AMI detection" + category: node_assessment + required: always + api: "ec2:DescribeLaunchTemplateVersions → ImageId" + severity: high + + - id: NODE-06 + description: "Fargate profile inventory" + category: node_assessment + required: if_fargate + api: "eks:ListFargateProfiles, eks:DescribeFargateProfile" + severity: medium + + - id: NODE-07 + description: "Auto Mode detection and config" + category: node_assessment + required: if_auto_mode + api: "eks:DescribeCluster → computeConfig" + severity: medium + + - id: NODE-08 + description: "Custom AMI bootstrap method (bootstrap.sh vs nodeadm)" + category: node_assessment + required: if_custom_ami + api: "ec2:DescribeLaunchTemplateVersions → UserData" + severity: high + + # ── ADDON ASSESSMENT ────────────────────────────────────────────────────── + + - id: ADDON-01 + description: "EKS managed addon inventory with current versions" + category: addon_assessment + required: always + api: "eks:ListAddons, eks:DescribeAddon" + severity: critical + + - id: ADDON-02 + description: "Addon compatibility verified against target via live API" + category: addon_assessment + required: always + api: "eks:DescribeAddonVersions" + severity: critical + + - id: ADDON-03 + description: "Self-managed addon detection including core VPC CNI, CoreDNS, and kube-proxy images/config" + category: addon_assessment + required: if_kubectl + api: "kubectl get daemonset aws-node,kube-proxy; deployment coredns; configmaps" + severity: high + + - id: ADDON-04 + description: "Addon upgrade order calculated (dependency chain)" + category: addon_assessment + required: always + api: "derived from ADDON-01 + ADDON-02" + severity: medium + + - id: ADDON-05 + description: "Third-party controller compatibility (LBC, cert-manager, external-dns, etc.)" + category: addon_assessment + required: if_kubectl + api: "kubectl get deploy (image tags) vs compatibility matrix" + severity: high + + # ── WORKLOAD ASSESSMENT ─────────────────────────────────────────────────── + + - id: WKLD-01 + description: "Deprecated API scan (live resources)" + category: workload_assessment + required: if_kubectl + api: "kubectl, kubent, pluto" + severity: high + + - id: WKLD-02 + description: "Helm stored manifest API scan (deployed revision only)" + category: workload_assessment + required: if_helm + api: "kubectl get secrets -l owner=helm,status=deployed" + severity: critical + + - id: WKLD-03 + description: "Third-party CRD API deprecation check (vendor/version-aware)" + category: workload_assessment + required: if_kubectl + api: "kubectl get crds, kubectl get deploy (controller versions)" + severity: high + + - id: WKLD-04 + description: "PDB blocking analysis (maxUnavailable=0 or minAvailable=replicas)" + category: workload_assessment + required: if_kubectl + api: "kubectl get pdb -A" + severity: high + + - id: WKLD-05 + description: "Admission webhook compatibility (failurePolicy=Fail on broad groups)" + category: workload_assessment + required: if_kubectl + api: "kubectl get validatingwebhookconfigurations, mutatingwebhookconfigurations" + severity: high + + - id: WKLD-06 + description: "StatefulSet safety (grace period, PVC retention, single-replica)" + category: workload_assessment + required: if_kubectl + api: "kubectl get statefulsets -A" + severity: medium + + - id: WKLD-07 + description: "Scaled-to-zero workload detection" + category: workload_assessment + required: if_kubectl + api: "kubectl get deploy,sts (replicas=0)" + severity: low + + - id: WKLD-08 + description: "TopologySpreadConstraints on critical workloads" + category: workload_assessment + required: if_kubectl + api: "kubectl get deploy -A (check spec)" + severity: low + + - id: WKLD-09 + description: "Service mesh detection (Istio, App Mesh, Linkerd)" + category: workload_assessment + required: if_kubectl + api: "kubectl get deploy, kubectl get crd" + severity: medium + + # ── KARPENTER ───────────────────────────────────────────────────────────── + + - id: KARP-01 + description: "Karpenter version compatibility with target K8s version" + category: karpenter + required: if_karpenter + api: "kubectl get deploy karpenter -n kube-system (image tag)" + severity: critical + + - id: KARP-02 + description: "Drift feature gate enabled (default on v0.33+)" + category: karpenter + required: if_karpenter + api: "kubectl get deploy karpenter env FEATURE_GATES" + severity: high + + - id: KARP-03 + description: "expireAfter not set to Never on any NodePool" + category: karpenter + required: if_karpenter + api: "kubectl get nodepools" + severity: high + + - id: KARP-04 + description: "Disruption budgets allow at least 1 node disruption" + category: karpenter + required: if_karpenter + api: "kubectl get nodepools (spec.disruption.budgets)" + severity: high + + - id: KARP-05 + description: "EC2NodeClass amiSelectorTerms not pinned to specific AMI ID" + category: karpenter + required: if_karpenter + api: "kubectl get ec2nodeclasses" + severity: high + + - id: KARP-06 + description: "EC2NodeClass amiFamily not AL2 when target >= 1.33" + category: karpenter + required: if_karpenter + api: "kubectl get ec2nodeclasses (spec.amiFamily)" + severity: critical + + - id: KARP-07 + description: "Karpenter controller not self-hosted on Karpenter-managed nodes" + category: karpenter + required: if_karpenter + api: "kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter (nodeSelector)" + severity: critical + + - id: KARP-08 + description: "NodePool nodeClassRef points to existing EC2NodeClass" + category: karpenter + required: if_karpenter + api: "kubectl get nodepools, kubectl get ec2nodeclasses" + severity: high + + - id: KARP-09 + description: "Consolidation interference check (short consolidateAfter + drift)" + category: karpenter + required: if_karpenter + api: "kubectl get nodepools (spec.disruption)" + severity: medium + + - id: KARP-10 + description: "Disruption budget schedule does not conflict with upgrade window" + category: karpenter + required: if_karpenter + api: "kubectl get nodepools (spec.disruption.budgets[].schedule)" + severity: medium + + - id: KARP-11 + description: "NodeClaim expireAfter won't trigger during upgrade window" + category: karpenter + required: if_karpenter + api: "kubectl get nodeclaims" + severity: medium + + - id: KARP-12 + description: "Drift throughput estimation (time to replace all nodes)" + category: karpenter + required: if_karpenter + api: "kubectl get nodes (karpenter-managed count) + budget rate" + severity: low + + - id: KARP-13 + description: "Karpenter controller health (all replicas ready, no crash-loop)" + category: karpenter + required: if_karpenter + api: "kubectl get deploy karpenter -n kube-system" + severity: critical + + - id: KARP-14 + description: "v1alpha5 Provisioner orphan detection (incomplete migration)" + category: karpenter + required: if_karpenter + api: "kubectl get crd provisioners.karpenter.sh" + severity: medium + + # ── PRE-DRAIN SAFETY ────────────────────────────────────────────────────── + + - id: DRAIN-01 + description: "Bare pods (no ownerReferences) on nodes — not rescheduled after eviction" + category: pre_drain + required: if_kubectl + api: "kubectl get pods -A (filter no ownerReferences)" + severity: high + + - id: DRAIN-02 + description: "Pods with emptyDir volumes — data lost on drain" + category: pre_drain + required: if_kubectl + api: "kubectl get pods -A (volume type emptyDir)" + severity: medium + + - id: DRAIN-03 + description: "Pods with custom finalizers — can hang eviction indefinitely" + category: pre_drain + required: if_kubectl + api: "kubectl get pods -A (metadata.finalizers)" + severity: medium + + - id: DRAIN-04 + description: "EBS AZ-pinned PVCs — cross-AZ reschedule strands the volume" + category: pre_drain + required: if_kubectl + api: "kubectl get pvc,pv + ec2:DescribeSubnets (AZ match)" + severity: critical + + - id: DRAIN-05 + description: "Fail-closed webhooks on drain-target nodes — eviction deadlock" + category: pre_drain + required: if_kubectl + api: "kubectl get webhookconfigs + kubectl get endpoints" + severity: critical + + - id: DRAIN-06 + description: "CoreDNS SPOF — all replicas on same drain batch" + category: pre_drain + required: if_kubectl + api: "kubectl get pods -n kube-system -l k8s-app=kube-dns (node distribution)" + severity: critical + + # ── ROLLBACK ────────────────────────────────────────────────────────────── + + - id: ROLL-01 + description: "Pre-upgrade rollback advisory (eligibility conditions and 7-day window documented; not yet verifiable)" + category: rollback + required: always + api: "advisory from upgrade plan" + severity: medium + + - id: ROLL-02 + description: "Post-upgrade ROLLBACK_READINESS insights are complete, fresh, and passing" + category: rollback + required: if_post_upgrade + api: "eks:ListInsights --filter categories=ROLLBACK_READINESS; eks:DescribeInsight" + severity: high diff --git a/skills/eks-upgrade-readiness/references/safety-invariants.md b/skills/eks-upgrade-readiness/references/safety-invariants.md new file mode 100644 index 0000000..a8a1a19 --- /dev/null +++ b/skills/eks-upgrade-readiness/references/safety-invariants.md @@ -0,0 +1,86 @@ +# SAFETY INVARIANTS — Load First, Keep in Context + +> This file defines non-negotiable safety rules for the EKS Upgrade Readiness +> skill. Every rule here is a hard constraint. Violating any Hard Rule is a +> critical defect in the assessment. + +## Knowledge Hierarchy (highest authority wins) + +``` +1. Live API response from THIS session (eks:Describe*, ec2:Describe*, kubectl get) + ▼ overrides +2. Config files in this skill (YAML references, check registry) + ▼ overrides +3. Fetched documentation (EKS docs, upstream K8s release notes) + ▼ overrides +4. Training data / model knowledge +``` + +When sources conflict, the higher-numbered source is WRONG. Live API responses +are ground truth. Never override a live API response with cached or trained +knowledge. + +**Example:** If `DescribeAddonVersions` returns a version list, use ONLY those +versions — never suggest a version remembered from training data. + +## Hard Rules + +| # | Rule | Rationale | +|---|------|-----------| +| H1 | NEVER select an add-on version not returned by `DescribeAddonVersions` in THIS session | Stale version data causes upgrade failures | +| H2 | NEVER execute a mutating API call — this skill is READ-ONLY | All mutations are in the Remediation Playbook for operator approval | +| H3 | NEVER skip a required check from the registry — mark it `SKIPPED` with reason | Evidence Completeness must be accurate | +| H4 | NEVER mark a gate PASS when data is missing or access was denied | UNKNOWN is the only valid verdict for missing data | +| H5 | NEVER produce a READY verdict while any gate is UNKNOWN | Operator must investigate unknowns before proceeding | +| H6 | ALWAYS document rollback classification before recommending any step | Operator must know recovery options | +| H7 | ALWAYS paginate API results to exhaustion — partial results produce LOW confidence | Incomplete data is dangerous | +| H8 | NEVER claim the control plane upgrade is irreversible without checking rollback eligibility | EKS supports 7-day rollback (since July 2026) under specific conditions | +| H9 | NEVER operate on a cluster without first confirming target identity (name, account, region) | Wrong-cluster assessments are useless | +| H10 | NEVER assume node group AMI type without checking `DescribeNodegroup` | Custom AMIs have unpredictable behavior | + +## Operation Classification + +This skill is **read-only by design**. All operations are Tier 3 (Allowed): + +| Tier | Operations | This Skill | +|------|-----------|------------| +| Tier 3: ALLOWED | `describe*`, `list*`, `get*`, `kubectl get/describe` | ✅ All assessment work | +| Tier 2: REVIEW-REQUIRED | `update-addon`, `patch`, `helm upgrade` | ❌ In Remediation Playbook only | +| Tier 1: BLOCKED | `update-cluster-version`, `delete-*`, `drain` | ❌ In Remediation Playbook only | + +## Uncertainty Handling + +``` +Agent encounters unknown condition + │ + ▼ +Can it be verified via live API call? + │ + ├─ Yes → Call API, use response as ground truth + │ + └─ No → Mark gate as UNKNOWN + Report: what was encountered, what was attempted, why uncertain + Overall verdict: CANNOT DETERMINE +``` + +- NEVER guess at version compatibility — verify via API or declare UNKNOWN +- NEVER infer cluster configuration from naming conventions alone +- NEVER assume an add-on is EKS-managed without checking `ListAddons` +- Empty API results ≠ PASS (means "no data" not "no problem") +- Pagination exhausted without `nextToken` = complete. Stopped early = LOW confidence. + +## Scope Restrictions + +- All operations scoped to the cluster(s) the operator specified +- CloudWatch queries (if any) target only `/aws/eks//cluster` +- Cross-cluster operations forbidden unless explicitly requested +- Cross-account operations require explicit account list from operator + +## AccessDenied Protocol + +When any API returns `AccessDeniedException` or `Forbidden`: +1. Log which permission is missing +2. Mark the affected gate as **UNKNOWN** +3. Continue with remaining checks (do not abort entire assessment) +4. Include in report: "Gate X: UNKNOWN — AccessDenied on ``" +5. Overall verdict: CANNOT DETERMINE (UNKNOWN gates exist) diff --git a/skills/eks-upgrade-readiness/references/upgrade-troubleshooting.md b/skills/eks-upgrade-readiness/references/upgrade-troubleshooting.md new file mode 100644 index 0000000..29695f4 --- /dev/null +++ b/skills/eks-upgrade-readiness/references/upgrade-troubleshooting.md @@ -0,0 +1,277 @@ +# EKS Upgrade Troubleshooting + +Common failures during EKS upgrades and their resolutions. + +## Control Plane Upgrade Failures + +### Upgrade stuck in "Updating" for > 60 minutes + +**Causes:** +- Webhook configurations blocking API server startup +- Custom admission controllers not compatible with new version +- Insufficient IAM permissions for EKS service role + +**Resolution:** +1. Check EKS update status: `aws eks describe-update --name --update-id ` +2. Review CloudTrail for EKS API errors +3. If webhook is blocking: the control plane will eventually recover by + skipping the webhook; no user action needed but it causes delays +4. If IAM: verify `AmazonEKSClusterPolicy` is attached to cluster role + +### Control plane upgrade succeeded but kubectl fails + +**Causes:** +- kubeconfig pointing to old endpoint +- aws-auth ConfigMap missing after upgrade (rare) +- Client version too old for new API server + +**Resolution:** +1. Update kubeconfig: `aws eks update-kubeconfig --name ` +2. Verify: `kubectl version` — client should be within one minor of server +3. Check aws-auth: `kubectl get configmap aws-auth -n kube-system` + +## Addon Upgrade Failures + +### Addon update returns "ConfigurationConflict" + +**Cause:** Addon was manually modified outside of EKS addon management. + +**Resolution:** This is an operator-approved mutation. Inspect and back up the +existing managed-addon configuration first, then choose deliberately: + +```bash +# Preserve intentional customer configuration when it is target-compatible. +aws eks update-addon --cluster-name \ + --addon-name --addon-version \ + --resolve-conflicts PRESERVE + +# OVERWRITE replaces conflicting customer configuration with EKS defaults. +# Use only after reviewing the diff, recording configurationValues, and +# confirming the rollback plan. +aws eks update-addon --cluster-name \ + --addon-name --addon-version \ + --resolve-conflicts OVERWRITE +``` + +### CoreDNS not running after upgrade + +**Causes:** +- Corefile incompatible with new version +- Pod scheduling issues (taints, resource limits) + +**Resolution:** +1. Check pods: `kubectl get pods -n kube-system -l k8s-app=kube-dns` +2. Check events: `kubectl describe pod -n kube-system` +3. If Corefile issue: check `kubectl get configmap coredns -n kube-system -o yaml` + +## Node Group Upgrade Failures + +### Nodes not draining (upgrade stuck) + +**Causes:** +- PDB with `maxUnavailable: 0` blocking eviction +- Pod with no controller (standalone pod without owner) +- Local storage preventing eviction (emptyDir with data) +- Finalizers blocking pod deletion + +**Resolution:** +1. Check PDBs: `kubectl get pdb --all-namespaces` +2. Identify blocking pods from node group update events: + ``` + aws eks describe-update --name --update-id --nodegroup-name + ``` +3. Temporarily adjust PDB: `kubectl patch pdb -p '{"spec":{"maxUnavailable":1}}'` +4. For standalone pods: delete manually or add controller + +### New nodes joining but pods not scheduling + +**Causes:** +- Taints on new nodes not tolerated by workloads +- Node labels changed between AMI versions +- Resource requests exceed new node capacity + +**Resolution:** +1. Check node taints: `kubectl describe node | grep Taint` +2. Check pending pods: `kubectl get pods --field-selector=status.phase=Pending` +3. Check events: `kubectl describe pod ` + +### InsufficientInstanceCapacity during node group upgrade + +**Cause:** EC2 cannot launch the required instance type in the AZ. + +**Resolution:** +1. Check which AZ is constrained from the update error +2. Options: + - Wait and retry (capacity may free up) + - Use Capacity Reservations (see `capacity-planning.md`) + - Add alternative instance types to the node group + - Reduce `maxUnavailablePercentage` to lower simultaneous surge + +### Launch template version mismatch + +**Cause:** Custom launch template AMI doesn't match target EKS version. + +**Resolution:** +1. Check LT: `aws ec2 describe-launch-template-versions --launch-template-id ` +2. Update AMI to match target version: + ``` + aws ssm get-parameter --name /aws/service/eks/optimized-ami//amazon-linux-2023/x86_64/standard/recommended/image_id + ``` +3. Create new LT version with correct AMI +4. Update node group to use new LT version + +## Karpenter-Specific Issues + +### Karpenter not launching nodes with new AMI after upgrade + +**Cause:** `amiSelectorTerms` in EC2NodeClass pinned to old version. + +**Resolution:** +1. Check EC2NodeClass: `kubectl get ec2nodeclass -o yaml` +2. Update `amiSelectorTerms` to include new version or use `amiFamily` for auto-discovery +3. Roll nodes: `kubectl delete nodes -l karpenter.sh/nodepool=` + +### Karpenter version incompatible with new EKS version + +**Cause:** Old Karpenter release doesn't support new K8s API version. + +**Resolution:** +1. Check Karpenter compatibility matrix in release notes +2. Upgrade Karpenter BEFORE or alongside control plane upgrade +3. For Karpenter v1.x, check minimum EKS version in docs + +## Rollback Scenarios + +Control-plane rollback is **conditional**, not one-way: EKS makes it available +only to an eligible cluster for seven days after a successful upgrade. During +that window, query `ListInsights` with `ROLLBACK_READINESS` and resolve every +`ERROR` or `UNKNOWN` before the operator attempts rollback. Outside the window, +or when eligibility conditions are not met, the control plane must be fixed +forward. + +| Component | Rollback Possible? | How | +|-----------|-------------------|-----| +| Control plane | CONDITIONAL (7 days after eligible upgrade) | Operator performs rollback only after `ROLLBACK_READINESS` insights PASS; otherwise fix forward or follow the documented forced-rollback procedure | +| Addons | YES | `aws eks update-addon --addon-version ` | +| Managed node groups | PARTIAL | Can halt; completed nodes stay at new version | +| Self-managed nodes | YES | Revert launch template, terminate new nodes | +| Karpenter nodes | YES | Revert EC2NodeClass AMI, delete nodes | + +## Prevention Checklist + +- [ ] Run EKS Upgrade Insights before starting +- [ ] Test upgrade in a non-production cluster first +- [ ] Enable cluster audit logging before upgrade +- [ ] Take Velero backup of critical resources +- [ ] Verify all webhooks are compatible with target version +- [ ] Confirm no pending node group health issues +- [ ] Schedule upgrade during low-traffic window +- [ ] Have rollback plan for addons and node groups + +## Feature-Specific Removal Guidance + +### Dockershim Removal (EKS 1.25) + +EKS Optimized AMI for 1.25+ no longer includes Dockershim. If workloads mount +the Docker socket (`/var/run/docker.sock`), they will break. + +**Detection:** +```bash +# Install and run Detector for Docker Socket (DDS) +kubectl krew install dds +kubectl dds +``` + +**Resolution:** Remove Docker socket dependencies. Use containerd-compatible +alternatives or CRI APIs directly. + +### PodSecurityPolicy Removal (EKS 1.25) + +PSP was removed in Kubernetes 1.25. Clusters using PSP must migrate before +upgrading. + +**Detection:** +```bash +kubectl get psp +# If any PSPs exist, migration is required +``` + +**Migration options:** +1. Pod Security Standards (PSS) with Pod Security Admission (PSA) — built-in +2. Policy-as-code: OPA Gatekeeper or Kyverno + +See [AWS PSP removal FAQ](https://docs.aws.amazon.com/eks/latest/userguide/pod-security-policy-removal-faq.html). + +### In-Tree Storage Driver Deprecation (EKS 1.23) + +The `kubernetes.io/aws-ebs` in-tree provisioner is deprecated. Must use the +EBS CSI driver (`ebs.csi.aws.com`) before upgrading to 1.23+. + +**Detection:** +```bash +kubectl get sc -o jsonpath='{range .items[*]}{.metadata.name}: {.provisioner}{"\n"}{end}' +# Look for kubernetes.io/aws-ebs +kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.csi.driver // "in-tree"}{"\n"}{end}' +``` + +**Resolution:** Install the [Amazon EBS CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) and create new StorageClasses +using `ebs.csi.aws.com`. Existing PVs will be handled by CSI migration (automatic). + +## Useful Upgrade Tools + +| Tool | Purpose | Link | +|------|---------|------| +| kubent | Scan cluster for deprecated APIs | https://github.com/doitintl/kube-no-trouble | +| pluto | Detect deprecated APIs in cluster and Helm charts | https://pluto.docs.fairwinds.com | +| kubectl-convert | Convert manifests between API versions | https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-kubectl-convert-plugin | +| eksup (ClowdHaus) | EKS upgrade guidance CLI | https://clowdhaus.github.io/eksup | +| GoNoGo | Determine upgrade confidence for add-ons | https://github.com/FairwindsOps/GoNoGo | +| DDS | Detect Docker socket dependencies | https://github.com/aws-containers/kubectl-detector-for-docker-socket | +| Velero | Cluster backup before upgrade | https://velero.io | +| AWS Backup | Managed backup for EKS | https://docs.aws.amazon.com/eks/latest/userguide/integration-backup.html | + +## Blue-Green Cluster Strategy + +For very large clusters or when skipping multiple minor versions is required: + +**Benefits:** +- Can jump multiple EKS versions at once +- Able to switch back to old cluster if issues arise +- Creates a fresh cluster with latest configurations + +**Downsides:** +- API endpoint and OIDC change (requires updating all consumers: kubectl, CI/CD, IRSA) +- Two clusters running in parallel (cost, capacity limits) +- Load balancers and external DNS cannot easily span clusters +- Stateful workload migration requires careful planning (data backup + restore) +- More coordination needed if workloads depend on each other + +**When to consider:** +- Cluster is 3+ minor versions behind +- In-place sequential upgrades would take too long or be too risky +- Cluster was created with legacy tooling and needs to be rebuilt with modern IaC +- Compliance requires a clean-state cluster + +### Identity Migration Considerations (IRSA vs. Pod Identity) + +Both mechanisms need work on a new cluster, but the work is different — don't +assume "no IAM changes" means "no identity work": + +| Mechanism | What Must Happen on the New Cluster | Effort | +|-----------|--------------------------------------|--------| +| **IRSA** | Each new cluster has its own OIDC provider ARN. Existing IAM role trust policies must be updated to also trust the new cluster's OIDC provider (a trust policy can list multiple issuers, but is capped at 4096 characters — roles shared across many clusters can hit this limit). | Edit IAM role trust policies | +| **Pod Identity** | The IAM role's trust policy does not change (it trusts the cluster-agnostic `pods.eks.amazonaws.com` service principal). However, associations (service account ↔ role mappings) are stored as an EKS resource scoped to one cluster — each association must be explicitly recreated with `aws eks create-pod-identity-association` on the new cluster. Nothing carries over automatically. | Recreate every association (no IAM edits) | + +Before a blue-green cutover, inventory both: +```bash +# IRSA: service accounts with role-arn annotations +kubectl get sa -A -o json | jq '.items[] | select(.metadata.annotations["eks.amazonaws.com/role-arn"] != null) | {ns: .metadata.namespace, sa: .metadata.name, role: .metadata.annotations["eks.amazonaws.com/role-arn"]}' + +# Pod Identity: existing associations on the source cluster +aws eks list-pod-identity-associations --cluster-name +``` +Every association returned by `list-pod-identity-associations` needs an +equivalent `create-pod-identity-association` call against the new cluster +before cutting workloads over — this is a mutation and belongs in the +Remediation Playbook (Step 14 of SKILL.md), not something the assessment +executes automatically.