Add eks-upgrade-readiness skill

skills/eks-upgrade-readiness/.skilleval.yaml

@@ -0,0 +1,3 @@
+audit:
+  ignore:
+    - STR-016    # README alongside SKILL.md is intentional

skills/eks-upgrade-readiness/CHANGELOG.md

@@ -0,0 +1,178 @@
+# Changelog
+
+## 2.0.0 (continued)
+
+### Final PR review corrections
+
+- Removed the accidental repository-root trigger-test scratch file; the
+  canonical trigger suite remains `evals/eval_queries.json` inside the skill.
+- Reconciled rollback guidance with current EKS behavior: rollback is
+  conditional for seven days after an eligible upgrade; `ROLLBACK_READINESS`
+  insights are post-upgrade only and `ERROR`/`UNKNOWN` blocks normal rollback.
+- Made kubelet skew bidirectional: no kubelet may be newer than the **current**
+  control plane, and the target lower bound remains N-3/N-2. Added PF-12 and
+  an explicit current-control-plane upper-skew eval.
+- Dated `addon-version-matrix.md` and `api-deprecations.md` as static fallback
+  references; live `DescribeAddonVersions` remains the authority. Reframed
+  kube-proxy exact-minor matching as a post-upgrade recommendation, not a
+  hard-coded blocker.
+- Added explicit discovery and compatibility handling for self-managed VPC
+  CNI, CoreDNS, and kube-proxy, including custom Corefile/config inspection.
+- Added Upgrade Insights freshness semantics: a stale `lastRefreshTime` is
+  `UNKNOWN`, not PASS.
+- Added concrete VPC CNI surge-capacity branches for prefix delegation (/28),
+  custom networking/ENIConfig, Security Groups for Pods/branch ENIs, and IPv6.
+- Documented `PRESERVE` versus `OVERWRITE` add-on conflict handling and the
+  risk that `OVERWRITE` discards customer configuration.
+- Made verdict aggregation mutually exclusive: all PASS = READY; WARNs only =
+  READY WITH WARNINGS; FAIL = NOT READY; otherwise UNKNOWN = CANNOT DETERMINE.
+- Corrected AL2 upstream end-of-life to June 30, 2026; added 5 functional
+  evals for current-control-plane upper skew, stale Insights, custom
+  CoreDNS/self-managed core addons, VPC CNI custom configuration, and
+  read-only add-on conflict resolution (24 total).
+
+### Follow-up items addressed from the second review round
+
+Picked off the three lowest-effort items from the "candidate follow-up issue"
+list; the remaining five (quorum-aware stateful drain, alternate CNIs/service
+meshes, hybrid nodes/Auto Mode depth, GPU/Neuron/Windows accelerated compute,
+fleet consistency) are tracked as separate follow-up issues per the
+reviewer's suggestion, not folded into this PR.
+
+1. **Client and CI tooling skew** — Step 2 now checks kubectl (±1 minor per
+   the upstream Kubernetes version skew policy), eksctl, Helm, and Terraform
+   AWS provider versions. WARN-level, not a blocker; no hardcoded version
+   floors since they shift every EKS release.
+
+2. **Pod Identity awareness** — Step 5 explicitly checks the
+   `eks-pod-identity-agent` managed addon like any other addon. New "Identity
+   Migration Considerations" section in `upgrade-troubleshooting.md` contrasts
+   IRSA (trust policy must be updated per new cluster's OIDC provider) with
+   Pod Identity (trust policy unchanged, but associations are scoped per
+   cluster and must be recreated with `create-pod-identity-association`).
+
+3. **Machine-readable output** — Step 17 now emits an optional structured
+   JSON verdict alongside the markdown report, with gate IDs matching
+   `required-check-registry.yaml` prefixes so CI/CD pipelines can gate on
+   specific check categories, not just the overall verdict.
+
+4. Added 3 new eval scenarios (19 total) covering CLI tooling skew, Pod
+   Identity blue-green migration, and machine-readable output requests.
+
+## 2.0.0
+
+Major rewrite addressing PR #48 review feedback. Breaking changes to step
+numbering and report format.
+
+### Must-Fix Items Addressed
+
+1. **EKS Upgrade Insights as primary signal** — Step 3 now explicitly declares
+   Insights as the primary authoritative signal. UNKNOWN verdict (not PASS) when
+   Insights is unavailable or returns no data. Pagination enforced.
+
+2. **Complete data plane inventory** — New Step 6 inventories ALL node
+   populations: Managed Node Groups (with DescribeNodegroup details),
+   self-managed ASGs (via autoscaling API + launch template inspection),
+   Karpenter (NodePools + EC2NodeClasses), Auto Mode, and Fargate profiles.
+   Kubelet version map across all nodes with skew validation.
+
+3. **Live addon API usage** — Step 5 uses `DescribeAddon` + `DescribeAddonVersions`
+   as primary source. Self-managed addon detection via deployment/Helm scan.
+   Static addon-version-matrix.md is now explicitly a fallback-only reference.
+
+4. **AL2→AL2023 comprehensive migration** — New Step 7 covers launch template
+   analysis, custom AMI detection, user data bootstrap differences (bootstrap.sh
+   → nodeadm/NodeConfig), cgroup v2 compatibility, IMDSv2 defaults, yum→dnf.
+
+5. **Mutations separated into Remediation Playbook** — New Step 14 consolidates
+   ALL mutating commands (helm upgrade, rollout restarts, OVERWRITE addon
+   updates, PDB patches, Karpenter annotation, CA pause). Clearly marked as
+   requiring operator approval. Agent never executes these.
+
+6. **Upgrade ordering refined** — New Step 8 explicitly defines pre-upgrade
+   alignment (Karpenter, CA, webhooks may need update BEFORE control plane) vs
+   post-upgrade addon/node-group sequence.
+
+7. **Deterministic test cases** — evals.json expanded from 6 to 16 scenarios
+   covering: N-2/N-3 mixed fleet, version skew violations, missing Insights
+   (AccessDenied), custom bootstrap AL2→AL2023, pagination handling, Karpenter
+   Drift disabled, GitOps/IaC detection, post-upgrade validation, self-managed
+   addon detection, pre-upgrade health failure. Assertions enforce UNKNOWN≠PASS.
+
+### Recommended Additions Addressed
+
+1. **GitOps/IaC version ownership detection** — New Step 12 detects CF/TF/CDK/
+   ArgoCD/Flux/eksctl from cluster/nodegroup tags and routes all remediation
+   through the owning tool. Never suggests direct CLI if IaC-managed.
+
+2. **VPC CNI networking modes awareness** — Step 2 subnet check is now mode-
+   aware: standard, prefix delegation, custom networking, IPv6, Security Groups
+   for Pods. Includes ENIConfig detection and mode-specific capacity gates.
+
+3. **Autoscaler pause during rotation** — New Step 13 checks Karpenter
+   consolidation policy and CA scale-down state. Pause commands in Step 14.
+
+4. **Pre-upgrade cluster health baseline** — New Step 10 validates all nodes
+   Ready, no pending CSRs, no crash-looping system pods, DNS resolution working,
+   metrics-server responding. Any failure blocks the upgrade.
+
+5. **Post-upgrade functional validation** — New Step 15 provides smoke tests:
+   DNS, metrics-server, pod scheduling, LB health, IRSA/Pod Identity, baseline
+   comparison.
+
+### Additional Improvements
+
+- Documented required AWS IAM permissions and Kubernetes RBAC ClusterRole
+- Added AccessDenied handling protocol (mark UNKNOWN, never PASS)
+- Added pagination requirements throughout (ListInsights, ListNodegroups, etc.)
+- Helm scanning now checks deployed revision only (not full history)
+- CRD checks are vendor-aware and version-aware (compare installed version)
+- StatefulSet PVC check corrected (persistentVolumeClaimRetentionPolicy)
+- MNG update algorithm documented (for capacity planning accuracy)
+- Verdict rules formalized: CANNOT DETERMINE when any gate is UNKNOWN
+- Trigger eval expanded to 16 queries (8 positive, 8 negative)
+- Skill expanded from 11 to 17 steps
+- Version bump to 2.0.0
+
+## 1.2.0
+
+- Add Helm manifest scanning — detects deprecated APIs in Helm release
+- Add version-specific removal gates — AL2 AMI unavailability (≥1.33),
+  kube-proxy IPVS deprecation (≥1.35/1.36), unmaintained ingress-nginx
+- Add service quota headroom checks
+- Add StatefulSet safety checks
+- Add more comprehensive Karpenter checks
+- Add scaled-to-zero workload detection
+- Add grading guards with confidence levels
+- Add third-party CRD API deprecation checks
+- Add cost awareness section
+- Add conditional evaluation logic — version-gated checks only when relevant
+- Expand pre-upgrade checklist with new checks
+- Version bump to 1.2.0
+
+## 1.1.0
+
+- Add infrastructure prerequisites check (subnet IPs, IAM role, KMS key)
+- Add Karpenter Drift and node expiry handling
+- Add TopologySpreadConstraints validation
+- Add Fargate pod restart requirement (Step 9)
+- Add feature-specific removals (Dockershim, PodSecurityPolicy, in-tree storage)
+- Add detection tools: kubent, pluto, kubectl-convert, eksup, GoNoGo
+- Add blue-green cluster alternative for large upgrades
+- Add EKS release calendar and auto-upgrade policy context
+- Add EKS Auto Mode awareness
+- Add rollback matrix
+- Expand from 9 steps to 11 steps
+- Align fully with AWS EKS Best Practices Guide cluster-upgrades section
+
+## 1.0.0
+
+- Initial version
+- 9-step upgrade readiness assessment workflow
+- API deprecation analysis with version-specific removal matrix
+- Addon compatibility check against target EKS version
+- Node group version skew and AMI readiness validation
+- Pod Disruption Budget validation for drain safety
+- Capacity planning with surge calculation and reservation guidance
+- Structured upgrade plan generation with rollback gates
+- Reference documents for API deprecations, addon matrix, capacity planning, and troubleshooting

skills/eks-upgrade-readiness/README.md

@@ -0,0 +1,212 @@
+# EKS Upgrade Readiness Skill
+
+A skill for AWS DevOps Agent that performs **read-only** upgrade readiness
+assessments for Amazon EKS clusters, aligned with the
+[AWS EKS Best Practices Guide — Cluster Upgrades](https://docs.aws.amazon.com/eks/latest/best-practices/cluster-upgrades.html).
+
+## Purpose
+
+EKS upgrades can fail or cause downtime when deprecated APIs, incompatible
+addons, version-skewed node groups, or misconfigured PDBs are not caught
+beforehand. This skill systematically checks every dimension the Best Practices
+Guide calls out and produces a READY / NOT READY / READY WITH WARNINGS /
+CANNOT DETERMINE verdict with a prioritized remediation plan.
+
+## Key Capabilities
+
+- **EKS Upgrade Insights (primary signal)** — UPGRADE_READINESS findings from
+  the ListInsights/DescribeInsight APIs
+- **Infrastructure prerequisites** — subnet IP availability (VPC CNI mode-aware),
+  IAM role, KMS key, service quotas
+- **API deprecation analysis** — maps removed APIs to replacements, scans Helm
+  stored manifests (deployed revision), vendor-aware CRD checks
+- **Addon compatibility (live API)** — validates managed addons via
+  DescribeAddonVersions, detects self-managed addons via deployment/Helm scan
+- **Full data plane inventory** — managed node groups, self-managed ASGs,
+  Karpenter (Drift, expiry, EC2NodeClass, budgets), Auto Mode, Fargate profiles,
+  kubelet version map
+- **AL2→AL2023 migration** — launch template analysis, custom AMI detection,
+  bootstrap differences, cgroup v2, IMDSv2
+- **Upgrade ordering** — pre-upgrade alignment (Karpenter/CA/webhooks before CP)
+  vs post-upgrade sequence
+- **PDB and topology spread** — detects drain blockers and availability risks
+- **StatefulSet safety** — grace period, PVC retention policy, single-replica risks
+- **Pre-upgrade health baseline** — node Ready status, pending CSRs, system pod
+  health, DNS/metrics baseline
+- **Capacity planning** — surge calculation, ODCR/FDCR guidance, blue-green
+  alternative, autoscaler pause
+- **GitOps/IaC detection** — routes remediation through owning tool
+  (Terraform/CDK/ArgoCD/Flux/eksctl)
+- **Post-upgrade validation** — DNS, metrics, scheduling, LB health, IRSA smoke tests
+- **Client/CI tooling skew** — kubectl (±1 minor), eksctl, Helm, Terraform provider checks (WARN-level)
+- **Pod Identity awareness** — addon version check plus IRSA-vs-Pod-Identity blue-green migration guidance
+- **Remediation playbook** — all mutations separated, require operator approval
+- **Structured upgrade plan** — ordered execution with rollback gates
+- **Machine-readable output** — optional JSON verdict (per-gate status, confidence,
+  evidence) alongside the markdown report, for CI/CD gating
+
+## Prerequisites
+
+### IAM Permissions
+
+The DevOps Agent role needs read access to EKS, EC2, IAM, and Auto Scaling:
+
+```
+eks:DescribeCluster
+eks:ListClusters
+eks:ListInsights
+eks:DescribeInsight
+eks:ListAddons
+eks:DescribeAddon
+eks:DescribeAddonVersions
+eks:ListNodegroups
+eks:DescribeNodegroup
+eks:ListFargateProfiles
+eks:DescribeFargateProfile
+eks:ListUpdates
+eks:DescribeUpdate
+ec2:DescribeSubnets
+ec2:DescribeInstances
+ec2:DescribeLaunchTemplateVersions
+ec2:DescribeImages
+ec2:DescribeCapacityReservations
+iam:GetRole
+autoscaling:DescribeAutoScalingGroups
+autoscaling:DescribeLaunchConfigurations
+servicequotas:GetServiceQuota
+```
+
+### Kubernetes RBAC (if kubectl access available)
+
+A `ClusterRole` with read-only access to nodes, pods, deployments, statefulsets,
+daemonsets, PDBs, configmaps, secrets (Helm), CRDs, CSRs, Karpenter resources,
+and ENIConfigs. See the "Required Permissions" section in SKILL.md for the
+full `ClusterRole` manifest. `kubectl` access is optional — the assessment
+still runs on AWS APIs alone at lower confidence for CRD/Helm/PDB checks.
+
+### AWS Resources
+
+- One or more Amazon EKS clusters (any supported version)
+- Control plane logging enabled (recommended for post-upgrade debugging)
+
+## Limitations
+
+- **EKS clusters only.** Does not cover EKS Anywhere, Outposts, or Local Zones.
+- **Read-only by design.** The skill produces recommendations; it never executes
+  mutating APIs. All mutations are in the Remediation Playbook (Step 14).
+- **UNKNOWN ≠ PASS.** Missing data or access denial produces UNKNOWN, never
+  PASS. The overall verdict cannot be READY while any gate is UNKNOWN.
+- **Addon version data may lag.** Static reference table is fallback only —
+  always prefer live `describe-addon-versions` API.
+- **Pagination required.** Large clusters with many node groups or addons
+  require exhausting API pagination tokens.
+
+## Agent Types
+
+- **Chat tasks** — ask for upgrade readiness assessments
+- **Evaluation** — periodic upgrade readiness scans
+
+## Uploading to AWS DevOps Agent
+
+**Option A: Import from GitHub (recommended)**
+
+If you have a [GitHub connection configured](https://docs.aws.amazon.com/devopsagent/latest/userguide/connecting-to-cicd-pipelines-connecting-github.html) in your Agent Space, you can import this skill directly from the repository. In the DevOps Agent web app, go to Settings → Add Skill → Import from repository, then
+point to `skills/eks-upgrade-readiness`. See [Importing a skill from a repository](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-devops-agent-skills.html#creating-skills) for full instructions.
+
+> **Note:** You cannot connect the `aws-samples` GitHub organization directly because the GitHub connection setup requires admin rights on the organization. Instead, connect your personal GitHub account and select any repository from it during the connection setup. Once a GitHub connection is established, you can import skills from any public repository, including this one, even if it wasn't selected during the connection setup.
+
+**Option B: Upload as a zip file**
+
+1. Zip the `eks-upgrade-readiness/` directory (only including allowed extensions):
+
+```bash
+cd skills
+zip -r eks-upgrade-readiness.zip eks-upgrade-readiness/ \
+  -i '*.md' '*.json' '*.yaml' '*.yml' \
+  -x '*/README.md' '*/.skilleval.yaml' '*/CHANGELOG.md' '*/evals/*'
+```
+
+2. In the AWS DevOps Agent web app, navigate to the **Skills** page.
+3. Click **Add skill** → **Upload skill**.
+4. Drag and drop the `eks-upgrade-readiness.zip` file (max 6 MB).
+5. Select the agent types: **Chat tasks** and **Evaluation**.
+6. Click **Upload**.
+
+**Option C: Upload via the Asset API**
+
+Use the DevOps Agent Asset API to programmatically manage skills — useful for CI/CD pipelines or automation workflows. Assign to `CHAT` and `EVALUATION` agent types. See [Managing a skill end-to-end](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-managing-assets.html#managing-a-skill-end-to-end) for the full API workflow.
+
+## How to Use This Skill
+
+Describe the task in natural language — you do not need to name the skill.
+
+### Example Prompts
+
+```
+"Is my EKS cluster prod-cluster in us-east-1 ready to upgrade to 1.31?"
+"Check upgrade readiness for all my EKS clusters"
+"What deprecated APIs would break if I upgrade to Kubernetes 1.32?"
+"Plan the upgrade of my cluster from 1.30 to 1.31 including node groups"
+"Are my addons compatible with EKS 1.31?"
+"Will my PDBs block a node group upgrade?"
+"I need to upgrade a 50-node cluster — what capacity do I need?"
+"Compare in-place vs blue-green strategy for my 200-node cluster"
+"My cluster is managed by Terraform — how should I do the upgrade?"
+"We use AL2 with custom bootstrap scripts — what breaks going to 1.33?"
+"We just upgraded to 1.31 — what should we validate?"
+"We use Pod Identity and are planning a blue-green migration — what identity work is needed?"
+"Give me the upgrade readiness result as JSON so I can gate our CI/CD pipeline"
+```
+
+### Modes
+
+| Mode | Trigger | Behavior |
+|------|---------|----------|
+| Full assessment | "upgrade readiness", "ready to upgrade" | All 17 steps, scored report |
+| Targeted check | "deprecated APIs", "addon compatibility", "PDB check" | One dimension, focused |
+| Planning | "upgrade plan", "upgrade runbook" | Execution order with rollback gates |
+| Comparison | "blue-green vs in-place" | Strategy recommendation |
+| Validation | "post-upgrade check", "validate upgrade" | Smoke tests (Step 15) |
+
+## Skill Structure
+
+```
+eks-upgrade-readiness/
+├── SKILL.md                # Main skill instructions (17-step workflow)
+├── README.md               # This file
+├── CHANGELOG.md            # Version history
+├── .skilleval.yaml         # Agent Skill Eval config
+├── evals/
+│   ├── evals.json          # 24 functional evaluation scenarios
+│   └── eval_queries.json   # Trigger tests (positive and negative)
+└── references/
+    ├── safety-invariants.md         # Hard safety rules, knowledge hierarchy, operation classification
+    ├── required-check-registry.yaml # All 60+ checks with IDs, categories, and severity
+    ├── pre-flight-checks.yaml       # Blocking vs warning checks, timeouts, soak periods, rollback conditions
+    ├── api-deprecations.md          # K8s API removal schedule by version
+    ├── addon-version-matrix.md      # EKS addon compatibility (static fallback)
+    ├── capacity-planning.md         # FDCR/ODCR surge capacity guidance
+    ├── upgrade-troubleshooting.md   # Tools, feature removals, blue-green
+    ├── karpenter-checks.md          # Full 14-check Karpenter registry (KARP-01 to KARP-14)
+    ├── pre-drain-safety.md          # DRAIN-01 to DRAIN-06 detection and remediation
+    ├── al2-al2023-migration.md      # AL2→AL2023 migration assessment details
+    └── data-plane-inventory.md      # MNG, self-managed, Karpenter, Auto Mode, Fargate inventory commands
+```
+
+## Safety
+
+This skill operates in **read-only** mode:
+
+- No cluster modifications — upgrade actions are recommendations only
+- No `update-*`, `delete-*`, or `create-*` API calls
+- All mutations isolated in Step 14 Remediation Playbook (operator approval)
+- All findings include evidence and specific remediation steps
+- The operator reviews the report and decides whether to proceed
+- UNKNOWN verdicts prevent false confidence (never marks missing data as PASS)
+
+## Non-production disclaimer
+
+> ⚠️ This skill is sample code, not intended for production use without
+> additional review and testing. Validate in a non-production environment first.
+> Compatibility data and version matrices are point-in-time references — always
+> verify with `aws eks describe-addon-versions` for the latest data.