From db7403e20e0df63b5eb0d6b5aee2abdefabf6a56 Mon Sep 17 00:00:00 2001 From: Shyam Kulkarni Date: Sun, 9 Aug 2026 14:32:04 +0530 Subject: [PATCH] feat(skills): Add aws-ecs-operations-review skill Add a comprehensive Amazon ECS operations review skill covering 6 review pillars (Resiliency & HA, Observability, Security, Operations, Performance, Additional Analysis) using read-only AWS APIs. Includes a 7-day CloudWatch metrics baseline, per-pillar PASS/FAIL/N/A scorecards, recommended alarm thresholds for IDR onboarding, and a prioritized remediation-linked report artifact. Ships with references, evals, and skill-eval config per the contribution guidelines. --- llms.txt | 1 + .../aws-ecs-operations-review/.skilleval.yaml | 3 + skills/aws-ecs-operations-review/CHANGELOG.md | 74 ++++++ skills/aws-ecs-operations-review/README.md | 117 +++++++++ skills/aws-ecs-operations-review/SKILL.md | 246 ++++++++++++++++++ .../evals/TESTING.md | 36 +++ .../evals/eval_queries.json | 9 + .../evals/evals.json | 73 ++++++ .../evals/files/service-context.json | 31 +++ .../references/alarm-thresholds.md | 150 +++++++++++ .../references/checks.md | 60 +++++ .../references/common-checks-coverage.md | 39 +++ .../references/pillars/additional-analysis.md | 17 ++ .../references/pillars/observability.md | 19 ++ .../references/pillars/operations.md | 18 ++ .../references/pillars/performance.md | 21 ++ .../references/pillars/resiliency.md | 24 ++ .../references/pillars/security.md | 30 +++ .../references/report-format.md | 115 ++++++++ 19 files changed, 1083 insertions(+) create mode 100644 skills/aws-ecs-operations-review/.skilleval.yaml create mode 100644 skills/aws-ecs-operations-review/CHANGELOG.md create mode 100644 skills/aws-ecs-operations-review/README.md create mode 100644 skills/aws-ecs-operations-review/SKILL.md create mode 100644 skills/aws-ecs-operations-review/evals/TESTING.md create mode 100644 skills/aws-ecs-operations-review/evals/eval_queries.json create mode 100644 skills/aws-ecs-operations-review/evals/evals.json create mode 100644 skills/aws-ecs-operations-review/evals/files/service-context.json create mode 100644 skills/aws-ecs-operations-review/references/alarm-thresholds.md create mode 100644 skills/aws-ecs-operations-review/references/checks.md create mode 100644 skills/aws-ecs-operations-review/references/common-checks-coverage.md create mode 100644 skills/aws-ecs-operations-review/references/pillars/additional-analysis.md create mode 100644 skills/aws-ecs-operations-review/references/pillars/observability.md create mode 100644 skills/aws-ecs-operations-review/references/pillars/operations.md create mode 100644 skills/aws-ecs-operations-review/references/pillars/performance.md create mode 100644 skills/aws-ecs-operations-review/references/pillars/resiliency.md create mode 100644 skills/aws-ecs-operations-review/references/pillars/security.md create mode 100644 skills/aws-ecs-operations-review/references/report-format.md diff --git a/llms.txt b/llms.txt index 3b5a275..1263016 100644 --- a/llms.txt +++ b/llms.txt @@ -23,6 +23,7 @@ Skills can be used with these AWS DevOps Agent types: - [Enrich with AWS Security Agent Skill](skills/enrich-with-aws-security-agent/SKILL.md): Queries AWS Security Agent CloudWatch logs to retrieve code-level security findings (file, line number, vulnerability type) during incident investigations with potential security root causes - [Wiz Security Context Skill](skills/wiz-security-context/SKILL.md): Queries the Wiz MCP server for a resource's security context (vulnerabilities, misconfigurations, secrets, active threats, malware, toxic combinations) to determine whether an operational anomaly is an operational issue or a security incident - [Service Quota Check Skill](skills/service-quota-check/SKILL.md): Checks AWS service quota utilization during investigations and before provisioning resources, flags quotas at 85%+ utilization, and requests increases via the Service Quotas API or recommends support cases +- [ECS Operations Review Skill](skills/aws-ecs-operations-review/SKILL.md): Performs comprehensive Amazon ECS operations reviews across 6 pillars (Resiliency & HA, Observability, Security, Operations, Performance, Additional Analysis) using read-only AWS APIs, with a 7-day CloudWatch metrics baseline, per-pillar PASS/FAIL/N/A scorecards, recommended alarm thresholds for IDR onboarding, and a prioritized remediation-linked report - [DMS Operational Review Skill](skills/database-migration-service-expertise/SKILL.md): Conducts AWS Database Migration Service operational reviews with 5-category health scoring, task failure troubleshooting, migration cutover runbooks, version deprecation tracking, and cost optimization ## Key Concepts diff --git a/skills/aws-ecs-operations-review/.skilleval.yaml b/skills/aws-ecs-operations-review/.skilleval.yaml new file mode 100644 index 0000000..686a9c7 --- /dev/null +++ b/skills/aws-ecs-operations-review/.skilleval.yaml @@ -0,0 +1,3 @@ +audit: + ignore: + - STR-016 # README alongside SKILL.md is intentional diff --git a/skills/aws-ecs-operations-review/CHANGELOG.md b/skills/aws-ecs-operations-review/CHANGELOG.md new file mode 100644 index 0000000..441e405 --- /dev/null +++ b/skills/aws-ecs-operations-review/CHANGELOG.md @@ -0,0 +1,74 @@ +# Changelog + +## 2.4.0 + +Capacity provider depth + compute platform awareness (69 baseline checks, up +from 64): + +- **Compute platform decision step** (new workflow step 4, rules in + `references/checks.md`): the agent now classifies the service as Fargate + (± Spot), EC2 ASG capacity provider, **ECS Managed Instances** + (`managedInstancesProvider`), launchType-only, or ECS Anywhere — including + mixed strategies — and this decision drives "Applies To" applicability in + every pillar. New applicability values: `EC2-ASG-CP`, `MI`, `CP-strategy`. +- **PERF7 deepened** (Low → Medium): flags launchType-only services (ignored + by managed scaling) and missing cluster `defaultCapacityProviderStrategy`. +- **PERF9 (new)**: managed scaling enabled with `targetCapacity` headroom + (80-100, <100 for spiky workloads) and `instanceWarmupPeriod` sanity. +- **PERF10 (new)**: metrics-driven capacity analysis — 7-day + `CapacityProviderReservation` (AWS/ECS/ManagedScaling) baseline compared + against configured `targetCapacity` to detect capacity-constrained + scale-outs vs idle over-provisioning. +- **PERF11 (new)**: capacity provider strategy base/weight design — on-demand + base for production, Spot burst by weight, task-size-fits-instance check. +- **REL14 (new)**: capacity provider infrastructure multi-AZ (ASG subnets or + Managed Instances `networkConfiguration.subnets` span 2+ AZs). +- **ADD7 (new)**: ECS Managed Instances migration evaluation for self-managed + EC2 services, keyed off OPS6/OPS7 (agent/AMI currency) signals. +- **OPS6/OPS7/OPS2** now explicitly N/A for Managed Instances (AWS manages + agent/AMI lifecycle). +- **alarm-thresholds.md**: new Capacity Provider Alarms section + (`CapacityProviderReservation` saturation alarm) and baseline-metrics row; + report header now records the resolved compute platform. + +## 2.3.2 + +Fix skill upload rejection (`400 ValidationException` from the AWS DevOps Agent +Asset API): + +- Reduced `SKILL.md` frontmatter to **only `name` and `description`**, the + fields the DevOps Agent uploader supports for zip skills. Removed the + `license`, `compatibility`, and nested `metadata` blocks added in 2.3.1 — the + DevOps Agent parser reads only `name`/`description` from frontmatter and + rejects the extra keys. `agent_types` and other asset metadata are supplied + in the Asset API request (or the Operator Web App) at upload time, not in + frontmatter. Description (with its trigger phrases) is unchanged and within + the 1024-char limit. + +## 2.3.1 + +Compliance with the AgentSkills.io open standard (aligns this skill with the +`aws-eks-operations-review` skill): + +- Renamed directory to `aws-ecs-operations-review` (registry + `aws--` naming convention). +- Rewrote SKILL.md frontmatter to the spec: only `name`, `description`, + `license`, `compatibility`, and `metadata` at the top level. Moved `version` + and `tags` inside `metadata:`; added `license`, `compatibility`, and the + `aws-devops-agent-skills.*` + `devops-agent-tools.*` registry metadata. + Front-loaded the `description` with trigger phrases for discovery. +- Fixed the `name` field to match the directory (`aws-ecs-operations-review`). +- Renamed `reference/` → `references/` (spec convention) and updated all + SKILL.md links. +- Added `README.md` (packaging / prerequisites / upload / usage) and an + `evals/` harness (routing + knowledge evals) mirroring the EKS skill. No + change to the assessment workflow, pillars, checks, or report format. + +## 2.3.0 + +- Baseline: comprehensive ECS operations review across the 6 review pillars + (Resiliency & HA, Observability, Security, Operations, Performance, + Additional Analysis) with a 7-day CloudWatch metrics baseline, recommended + alarm thresholds for IDR onboarding, per-pillar ✓/✗/N/A scorecards, a + coverage gate, and the `review-common` baseline crosswalk. Read-only AWS API + data collection; Markdown report by default, DOCX on request. diff --git a/skills/aws-ecs-operations-review/README.md b/skills/aws-ecs-operations-review/README.md new file mode 100644 index 0000000..0f6d4d5 --- /dev/null +++ b/skills/aws-ecs-operations-review/README.md @@ -0,0 +1,117 @@ +# ECS Operations Review — AWS DevOps Agent Skill + +An end-to-end Amazon ECS operational review skill for [AWS DevOps Agent](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent.html). It assesses an ECS service against the ECS Best Practices Guide across six review pillars and produces a prioritized, remediation-linked report artifact per service, plus recommended CloudWatch alarms for IDR onboarding. + +It is **strictly read-only**: it uses only `describe*` / `list*` / `get*` AWS API operations (ECS, CloudWatch, IAM, Application Auto Scaling, ELB, ECR, EC2, GuardDuty, Compute Optimizer). It never runs a mutating call; remediations are drafted for human approval, not applied. + +> ⚠️ **Non-production disclaimer.** This skill is sample code, not intended for +> production use without additional review and testing. Users should validate in +> a non-production environment first. + +## What It Does + +1. **Parse the service ARN** (`arn:aws*:ecs:*:*:service/*/*`) and validate region. +2. **Collect configuration data** across the API tier dependency chain — Tier 1 (`ecs.describeServices`) must succeed first; non-Tier-1 access errors mark dependent checks N/A and continue. +3. **Resolve the compute platform** — Fargate (± Spot), EC2 ASG capacity provider, ECS Managed Instances, launchType-only, or ECS Anywhere — from the capacity provider strategy and `ecs.describeCapacityProviders`; this decision drives check applicability in every pillar. +4. **Run pillar checks** — read `references/checks.md` (the index) first, then each `references/pillars/.md` one at a time, grading ✓ / ✗ / N/A with evidence, severity, and a recommendation. +5. **Report** — write a per-service artifact following `references/report-format.md`: workload details, per-pillar scorecards for all 6 pillars, prioritized action plan, detailed findings, a Recommended CloudWatch Alarms table, access limitations, and a review summary. + +Pillars graded: **Resiliency & HA (REL), Observability (OBS), Security (SEC), Operations (OPS), Performance (PERF), and Additional Analysis (ADD)**. + +## Data Sources + +| Source | Used for | Required? | +|--------|----------|-----------| +| Read-only AWS APIs (AWS CLI / SDK / AWS API MCP) | All configuration data across the six pillars | Yes | +| CloudWatch `getMetricStatistics` | 7-day baseline (CPU, memory, task count) + `describeAlarms` | Yes (limitation noted if <7 days) | +| AWS Knowledge MCP | Doc-link lookups for findings and alarm recommendations | Yes | + +## Agent Types + +Intended for these agent types (selected in the Operator Web App at upload time): + +- **On-demand** — conversational invocation in Chat ("run an ECS operations review on service X", "ECS security review"). +- **Evaluation** — proactive operational improvement recommendations. + +Select **Generic** to make the skill available to all agent types. + +## Prerequisites + +### 1. An AWS DevOps Agent Space with the target AWS account + +An existing [Agent Space](https://docs.aws.amazon.com/devopsagent/latest/userguide/getting-started-with-aws-devops-agent-creating-an-agent-space.html) with the target AWS account configured as a cloud source. + +### 2. Read-only permissions + +The Agent Space IAM role needs read-only (`describe*` / `list*` / `get*`) access to: ECS, CloudWatch, CloudWatch Logs, IAM, Application Auto Scaling, Elastic Load Balancing v2, ECR, EC2/VPC, GuardDuty, and Compute Optimizer. The AWS managed **`ReadOnlyAccess`** policy (or a least-privilege subset of the above) is sufficient. No cluster-level access entry or kubectl connectivity is required — ECS is assessed entirely through AWS control-plane APIs. + +### 3. AWS Knowledge MCP + +Used for documentation-link lookups on findings and alarm recommendations. This is built into AWS DevOps Agent. + +## Packaging the skill + +From the directory **containing** `aws-ecs-operations-review/`: + +```bash +zip -r aws-ecs-operations-review.zip aws-ecs-operations-review/ \ + -i '*.md' '*.txt' '*.json' '*.yaml' '*.yml' \ + -x '*/.git/*' '*/evals/*' '*/CHANGELOG.md' '*/README.md' '*.DS_Store' +``` + +The uploaded zip contains: + +``` +aws-ecs-operations-review/ +├── SKILL.md # frontmatter + skill instructions (required) +└── references/ + ├── checks.md # checks index (read first) + ├── alarm-thresholds.md # recommended CloudWatch alarm thresholds + ├── common-checks-coverage.md + ├── report-format.md + └── pillars/ # one file per pillar (REL/OBS/SEC/OPS/PERF/ADD) +``` + +Upload-time constraints: `SKILL.md` required with `name` + `description` frontmatter; **no `scripts/` directory**. `evals/`, `README.md`, and `CHANGELOG.md` are dev-only and excluded above. + +## Uploading to AWS DevOps Agent + +> Reference: [Uploading a skill](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-devops-agent-skills.html#uploading-a-skill) + +1. Open the **Skills** page in your Agent Space Operator Web App. +2. **Add skill** → **Upload skill**. +3. Drag and drop `aws-ecs-operations-review.zip`. +4. Select agent types: **On-demand** and **Evaluation** (or **Generic**). +5. Review validation results → **Upload**. + +## Usage + +In Chat, use natural language: + +- *"Run an ECS operations review on service `arn:aws:ecs:us-east-1:111122223333:service/prod/web`."* +- *"ECS security review for the `web` service in cluster `prod`."* +- *"Assess reliability and cost for my ECS services."* + +The agent validates the service ARN, collects read-only AWS data, grades the six pillars, and writes a per-service review artifact. + +## Evaluation + +The `evals/` directory holds an evaluation harness: + +- `eval_queries.json` — routing checks (does the right query trigger the skill?). +- `evals.json` — skill-knowledge evals (six pillars, read-only contract, ARN validation, coverage gate, alarm deliverable), run against `evals/files/service-context.json`. + +Run them with your skill-eval runner. Record results in [`evals/TESTING.md`](evals/TESTING.md) (model × eval-suite pass-rate matrix) and re-run after any change to the frontmatter or workflow steps. Results are recorded from real runs, never fabricated. + +## Severity + +Internally the skill grades on `Critical / High / Medium / Low / Info` tiers; the report writer maps these to customer-facing descriptive labels (see [`references/report-format.md`](references/report-format.md)). It never emits internal severity numbers in customer-facing output. + +## Source attribution + +- [About AWS DevOps Agent](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent.html) +- [Amazon ECS Best Practices Guide](https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/intro.html) + +## License + +Internal use. diff --git a/skills/aws-ecs-operations-review/SKILL.md b/skills/aws-ecs-operations-review/SKILL.md new file mode 100644 index 0000000..ba06b07 --- /dev/null +++ b/skills/aws-ecs-operations-review/SKILL.md @@ -0,0 +1,246 @@ +--- +name: aws-ecs-operations-review +description: > + Performs a comprehensive Amazon ECS operations review across the 6 review + pillars (Resiliency & HA, Observability, Security, Operations, Performance, + Additional Analysis) using read-only AWS APIs, with a 7-day CloudWatch + metrics baseline, recommended alarm thresholds for IDR onboarding, per-pillar + PASS/FAIL/N/A scorecards, and a prioritized, remediation-linked report + artifact. Triggers on: "ECS operations review", "ECS assessment", + "ECS review", "review my ECS service", "ECS reliability review", + "ECS security review", "ECS best practices audit", "review ECS services + for a workload". +metadata: + author: kulkshya + version: "1.0.0" + aws-devops-agent-skills.agent-types: "Chat tasks, Evaluation" + aws-devops-agent-skills.aws-services: "Amazon ECS" +--- + +# ECS Operations Review Skill + +## Overview +Execute a comprehensive Amazon ECS operations review across the 6 review pillars with ✓/✗/N/A observations, 7-day CloudWatch baseline metrics, and alarm threshold recommendations for IDR onboarding. + +## Usage +- User mentions "ECS operations review", "ECS assessment", "ECS review" +- User provides ECS service ARN(s) for operations review +- User asks to review ECS services for a workload +- User requests ECS reliability or security review +- Delegated ECS service assessment from UOPS + +## Core Concepts + +- **Review Pillars**: Six assessment dimensions — Resiliency & HA, Observability, Security, Operations, Performance, Additional Analysis +- **ECS Service/Cluster**: The primary resources assessed — includes tasks, services, and cluster configuration +- **AWS API**: Public AWS service APIs used for all data collection, called read-only (describe/list/get) via the AWS CLI, an AWS SDK, or an AWS API MCP +- **Baseline Metrics**: 7-day CloudWatch metric history used to establish normal operating patterns + +## Prerequisites +- Read-only AWS API access (describe/list/get) for ECS, CloudWatch, IAM, Application Auto Scaling, ELB, ECR, EC2 APIs — via AWS CLI, an AWS SDK, or an AWS API MCP +- AWS Knowledge MCP access (search_documentation, read_documentation, recommend, list_regions, get_regional_availability) +- AWS account ID and region +- CloudWatch metrics access (7-day minimum for baseline) +- ECS service ARN — format: arn:aws*:ecs:*:*:service/*/* + +## Skill Files +- **references/checks.md** — Checks **index**: pillar→file map, check-ID ranges, counts, and the access-limitation / minimum-baseline rules. Read this FIRST. +- **references/pillars/resiliency.md** — Resiliency & HA checks (REL1-REL14). Read only when running the Resiliency pillar. +- **references/pillars/observability.md** — Observability checks (OBS1-OBS9). Read only when running the Observability pillar. +- **references/pillars/security.md** — Security checks (SEC1-SEC20). Read only when running the Security pillar. +- **references/pillars/operations.md** — Operations checks (OPS1-OPS8). Read only when running the Operations pillar. +- **references/pillars/performance.md** — Performance checks (PERF1-PERF11). Read only when running the Performance pillar. +- **references/pillars/additional-analysis.md** — Additional Analysis checks (ADD1-ADD7). Read only when running the Additional pillar. +- **references/alarm-thresholds.md** — Recommended CloudWatch alarm thresholds for IDR onboarding. +- **references/common-checks-coverage.md** — Crosswalk proving the review covers the shared **`review-common`** baseline (tagging, encryption, IAM least-privilege, alarms, logging, cost) via existing ECS check IDs. Read for any full review / CWR. +- **references/report-format.md** — Required report/artifact structure, severity model, finding-block format, and the **coverage gate**. Read before generating the report. + +### Context Management for Checks +Do NOT read all 6 pillar files at once. Read `references/checks.md` (the index) first, then read each `references/pillars/.md` file ONE AT A TIME as you run that pillar's checks. Agent may add checks beyond the baseline using the next sequential ID in the pillar. + +## Assessment Workflow + +This skill is self-contained — the procedural workflow is embedded below (this copy lives in `aws-operations-review` and does not depend on any external `agent-sops/` SOP). Execute the steps in order; this skill's `references/` files supply the check definitions and alarm thresholds. + +1. **Setup** — Create the output directory and a scratchpad for raw API responses. Record account ID, region, and timestamp. +2. **Parse ARN** — Validate the ECS service ARN (`arn:aws*:ecs:*:*:service/*/*`); extract cluster name and service name. Halt if the ARN region does not match the region parameter. +3. **Collect configuration data** — Call the AWS APIs following the tier order in the *API Tier Dependency Chain* below. Tier 1 (`ecs.describeServices`) MUST succeed before any other call. Save each response verbatim to the scratchpad. On non-Tier-1 access/API errors, mark dependent checks N/A and continue. +4. **Resolve compute platform** — Before grading any pillar, classify the service's compute platform from `launchType` + `capacityProviderStrategy` + `ecs.describeCapacityProviders`: **Fargate** (FARGATE/FARGATE_SPOT), **EC2 ASG capacity provider** (`autoScalingGroupProvider`), **Managed Instances** (`managedInstancesProvider`), **launchType-only EC2/Fargate** (no strategy), or **ECS Anywhere** (EXTERNAL). Mixed strategies are valid — record every platform present. This decision drives check applicability in every pillar (full rules in `references/checks.md`). Record the platform in the report header. +5. **Run pillar checks** — Read `references/checks.md` (the index) first, then for EACH pillar read its `references/pillars/.md` file one at a time and apply its checks against the collected data, recording ✓ / ✗ / N/A with an observation. Use the exact check IDs and severities; mark platform-specific checks N/A where they don't apply (per the resolved compute platform). May add checks beyond the baseline. +6. **Generate report** — Produce the per-service review artifact following [`references/report-format.md`](references/report-format.md) exactly (Workload Details, per-pillar ✓/✗/N/A scorecards for **all 6 pillars**, Prioritized Action Plan, Detailed Findings, Recommended CloudWatch Alarms from `references/alarm-thresholds.md`, Access Limitations, Review Summary). **Before finalizing, run the report-format Coverage gate** — every check ID across all 6 pillars must appear in a scorecard as ✓/✗/N/A (including passes; no pillar dropped or truncated), the alarms table must be present, every ✗ needs a detailed finding block, and the shared **`review-common`** baseline must be accounted for per [`references/common-checks-coverage.md`](references/common-checks-coverage.md) (all eight common checks covered via their ECS equivalents or ⚪ N/A with a reason). Default to a Markdown artifact; render DOCX only if asked (build from the same content). Strip internal check IDs from the customer-facing report. Return the Review Summary with verified counts, then delete the scratchpad. + +**Review Pillars:** +- **Resiliency and High Availability (REL1-REL14)** — Multi-AZ, desired count, deployment config, circuit breaker, deployment alarms, health checks, subnet AZ spread, capacity-provider managed termination protection, target-group deregistration delay, capacity provider infrastructure multi-AZ +- **Observability (OBS1-OBS9)** — Container Insights, CloudWatch alarms, logging, log retention, distributed tracing, metrics monitoring +- **Security (SEC1-SEC20)** — IAM least privilege, network mode, secrets management, ECR image scanning, security groups, VPC endpoints, private connectivity, encryption at rest, encryption in transit (TLS), VPC Flow Logs, GuardDuty Runtime Monitoring +- **Operations (OPS1-OPS8)** — Deployment controller, resource tagging, IaC-managed, platform version, ECS agent version +- **Performance (PERF1-PERF11)** — Auto scaling, CPU/memory rightsizing, capacity provider strategy, managed scaling / targetCapacity headroom, CapacityProviderReservation 7-day baseline analysis, base/weight strategy design, Compute Optimizer recommendations +- **Additional Analysis & Recommendations (ADD1-ADD7)** — Graviton/ARM64, Fargate Spot, Service Connect, cost optimization, CloudWatch Logs Insights queries, ECS Managed Instances evaluation + +## AWS API Summary + +All calls below are public AWS API operations. Use read-only (describe/list/get) operations only, via the AWS CLI, an AWS SDK (e.g. boto3), or an AWS API MCP with least-privilege read-only credentials. + +### ECS APIs (Tier 1, 2, 3) +| API | Tier | Purpose | +|-----|------|---------| +| ecs.describeServices | 1 | Foundation — service config, task def, LB, deployment, network | +| ecs.describeTaskDefinition | 2 | Container defs, CPU/memory, roles, log config, network mode | +| ecs.describeClusters | 2 | Cluster settings, Container Insights, capacity providers | +| ecs.listTasks | 2 | Running task ARNs for the service | +| ecs.describeTasks | 3 | Task health, AZ spread, connectivity status | +| ecs.listContainerInstances | 2 | Container instance ARNs for EC2 launch type clusters | +| ecs.describeContainerInstances | 3 | Agent version, AMI ID, instance status (EC2 only) | +| ecs.describeCapacityProviders | 2 | Compute platform classification (ASG vs Managed Instances vs Fargate), managed termination protection (REL12), managed scaling status/targetCapacity (PERF9), MI network config (REL14) | + +### Application Auto Scaling APIs (Tier 2) +| API | Tier | Purpose | +|-----|------|---------| +| applicationautoscaling.describeScalingPolicies | 2 | Auto scaling policies for the service | +| applicationautoscaling.describeScalableTargets | 2 | Min/max capacity configuration | + +### ELB APIs (Tier 2) +| API | Tier | Purpose | +|-----|------|---------| +| alb.describeTargetHealth | 2 | Target health for service tasks behind ALB/NLB (skip if no LB configured) | +| alb.describeTargetGroups | 2 | Target group details including LoadBalancerArns — used to determine LB type (ALB vs NLB) by ARN path segment: `/app/` = ALB, `/net/` = NLB, for correct alarm recommendations (skip if no LB configured) | +| elbv2.describeListeners | 2 | Listener protocol/port for the LB fronting the service — HTTPS/TLS vs plaintext HTTP/TCP for encryption-in-transit (SEC20); uses LoadBalancerArns from `describeTargetGroups` (skip if no LB configured) | + +### IAM APIs (Tier 4a, 4b) +| API | Tier | Purpose | +|-----|------|---------| +| iam.listAttachedRolePolicies | 4a | Managed policies on execution/task roles | +| iam.listRolePolicies | 4a | Inline policy names on execution/task roles | +| iam.getRolePolicy | 4b | Inline policy document for execution/task roles | + +### ECR APIs (Tier 3) +| API | Tier | Purpose | +|-----|------|---------| +| ecr.describeRepositories | 3 | Image scanning config, tag immutability for container image repos | + +### EC2/VPC APIs (Tier 2, 3, 4) +| API | Tier | Purpose | +|-----|------|---------| +| ec2.describeSecurityGroups | 2 | Security group rules for service ENIs (awsvpc mode) | +| ec2.describeSubnets | 2 | Subnet AZ distribution for service network config | +| ec2.describeVpcEndpoints | 3 | VPC endpoints for ECR, CloudWatch Logs, Secrets Manager (uses VPC ID from describeSubnets) | +| ec2.describeRouteTables | 3 | Route table entries for NAT/internet access assessment | +| ec2.describeNatGateways | 3 | NAT Gateway availability for private subnets | +| ec2.describeImages | 4 | AMI creation date for container instance AMI currency check (EC2 only, uses imageId from describeContainerInstances) | +| ec2.describeVolumes | 3 | EBS volume encryption status for task-attached / container-instance volumes (SEC17) | +| ec2.describeFlowLogs | 3 | VPC Flow Logs enablement for the service VPC (SEC18, uses VPC ID from describeSubnets) | + +### CloudWatch APIs (Tier 2, 5) +| API | Tier | Purpose | +|-----|------|---------| +| cloudwatch.describeAlarms | 2 | Existing alarms for ECS service | +| cloudwatch.getMetricStatistics | 5 | 7-day baseline: CPU, memory, task count; plus CapacityProviderReservation (AWS/ECS/ManagedScaling) for EC2 ASG capacity providers (PERF10) | + +### CloudWatch Logs APIs (Tier 3) +| API | Tier | Purpose | +|-----|------|---------| +| logs.describeLogGroups | 3 | Log retention setting and Logs Insights query targeting for the awslogs group (OBS8, ADD6 — uses awslogs-group from task definition) | + +### GuardDuty APIs (Tier 2) +| API | Tier | Purpose | +|-----|------|---------| +| guardduty.listDetectors | 2 | Detector presence in region (SEC19) | +| guardduty.getDetector | 2 | Runtime Monitoring feature status for ECS (SEC19 — uses detector ID from listDetectors) | + +### Compute Optimizer APIs (Tier 2) +| API | Tier | Purpose | +|-----|------|---------| +| computeoptimizer.getECSServiceRecommendations | 2 | ECS service task CPU/memory rightsizing recommendations (PERF8) | + +### AWS Knowledge MCP +| Tool | Purpose | +|------|--------| +| aws___search_documentation | Search across all AWS documentation with optional topic-based filtering | +| aws___read_documentation | Retrieve and convert AWS documentation pages to markdown | +| aws___recommend | Get content recommendations for AWS documentation pages | +| aws___list_regions | Retrieve a list of all AWS regions | +| aws___get_regional_availability | Retrieve AWS regional availability information | + +## API Tier Dependency Chain + +``` +Tier 1: ecs.describeServices (FOUNDATION — must complete first) + ├─ extracts: taskDefinition ARN, clusterArn, loadBalancers, + │ desiredCount, launchType, networkConfiguration, tags + │ + ├─► Tier 2 (parallel): ecs.describeTaskDefinition, ecs.describeClusters, + │ ecs.listTasks, cloudwatch.describeAlarms, + │ applicationautoscaling.describeScalingPolicies, + │ applicationautoscaling.describeScalableTargets, + │ ecs.describeCapacityProviders (compute platform classification; + │ ASG providers — REL12/PERF9; Managed Instances providers — REL14), + │ guardduty.listDetectors ─► guardduty.getDetector (SEC19), + │ computeoptimizer.getECSServiceRecommendations (PERF8), + │ alb.describeTargetHealth (if LB configured), + │ alb.describeTargetGroups (if LB configured — resolves ALB vs NLB type from LoadBalancerArns: /app/ = ALB, /net/ = NLB; also deregistration delay for REL13), + │ elbv2.describeListeners (if LB configured — listener protocol for encryption-in-transit SEC20), + │ ec2.describeSecurityGroups (from networkConfiguration.securityGroups), + │ ec2.describeSubnets (from networkConfiguration.subnets), + │ ecs.listContainerInstances (EC2 launch type only) + │ │ + │ ├─► Tier 3: ecs.describeTasks (using task ARNs from listTasks) + │ │ ecs.describeContainerInstances (EC2 only, using instance ARNs from listContainerInstances) + │ │ ecr.describeRepositories (using repo name from task definition image URI) + │ │ ec2.describeVpcEndpoints (using VPC ID from describeSubnets) + │ │ ec2.describeRouteTables (using subnet IDs from describeSubnets) + │ │ ec2.describeNatGateways (using VPC ID from describeSubnets) + │ │ ec2.describeFlowLogs (using VPC ID from describeSubnets — SEC18) + │ │ ec2.describeVolumes (task-attached / container-instance EBS encryption — SEC17) + │ │ logs.describeLogGroups (using awslogs-group from task definition — OBS8, ADD6) + │ │ + │ ├─► Tier 4 (EC2 only): ec2.describeImages (using imageId from describeContainerInstances) + │ │ + │ └─► Tier 4a (parallel): iam.listAttachedRolePolicies (execution + task role), + │ iam.listRolePolicies (execution + task role) + │ │ + │ └─► Tier 4b: iam.getRolePolicy (execution + task role) + │ (uses policy names from listRolePolicies) + │ + └─► Tier 5 (per-metric loop): cloudwatch.getMetricStatistics +``` + +## Access Limitation Handling + +When AWS API calls return access denied or authorization errors: +- Mark dependent checks as N/A with observation: "Unable to assess — access denied on {{api_name}}. Manual verification recommended." +- Include a dedicated **Access Limitations** section in the report listing all checks that could not be evaluated due to permissions +- Continue with remaining assessable checks — do NOT halt the entire assessment for non-Tier-1 access errors +- In the Review Summary, note how many checks could not be evaluated due to access limitations + +## Example Output + +The agent produces a per-service review artifact (Markdown by default; DOCX if asked) containing: service configuration summary, **all 6 review pillar scorecards** (✓/✗/N/A with observations, every check including passes), 7-day baseline metrics, the recommended-alarms table with clickable doc links, and priority action items. A Review Summary with verified counts is returned to the orchestrator. + +## What Not to Do + +- Don't skip any pillar — all 6 must be assessed +- Don't use write or mutating API calls — this is a read-only assessment; use describe/list/get operations only +- Don't hardcode doc URLs for ✗ check findings — use AWS Knowledge MCP (`aws___search_documentation`) to supplement the doc links provided in checks.md +- Don't hardcode doc URLs for alarm recommendation hyperlinks — use the `doc_url` column from `references/alarm-thresholds.md` as the canonical link target +- Don't skip alarm recommendations — this is a core IDR deliverable +- Don't omit the metrics baseline section — if 7-day data is unavailable, note the limitation in the report rather than skipping it +- Don't hallucinate findings — only report what AWS API data confirms +- Don't silently skip checks when access is denied — always mark as N/A with explicit access limitation note + +## Failure Recovery + +- If `ecs.describeServices` fails after retries: HALT workflow — delete `{{scratchpad_dir}}/` and return error to orchestrator +- If Tier 2-5 APIs fail: mark dependent checks as N/A, continue assessment with available data +- If report generation fails: default to the Markdown artifact (no external dependency); only fall back to DOCX (`python-docx`, retry with --user on install failure) when a DOCX deliverable was explicitly requested +- If artifact write fails: capture the error, delete scratchpad, return error to orchestrator + +## Success Criteria + +- All 6 review pillars assessed with ✓/✗/N/A observations — **coverage gate passed**: every check ID across all 6 pillars appears in a scorecard (including passes), no pillar dropped or truncated +- Review artifact generated at `{{output_directory}}/` (Markdown by default; DOCX only if requested) +- Recommended-alarms table populated from alarm-thresholds.md, marking exist-vs-missing +- 7-day baseline metrics included (or limitation noted) +- Every ✗ has a detailed finding block; every recommendation includes a resolvable AWS documentation link +- Access limitations explicitly documented; unobtainable checks are N/A-with-reason, never omitted or guessed +- Review Summary with verified counts returned to orchestrator +- `{{scratchpad_dir}}/` deleted after report verified diff --git a/skills/aws-ecs-operations-review/evals/TESTING.md b/skills/aws-ecs-operations-review/evals/TESTING.md new file mode 100644 index 0000000..094a51a --- /dev/null +++ b/skills/aws-ecs-operations-review/evals/TESTING.md @@ -0,0 +1,36 @@ +# Multi-model testing + +Because the skill competes for discovery against many other skills, the +`name`/`description` frontmatter and the knowledge evals should be run against +**every model the skill will be used with** (e.g. Claude Haiku, Sonnet, and +Opus) — what triggers reliably on Opus may need confirmation on a smaller +model. Re-run after any change to the frontmatter or the workflow steps. + +## Suites + +- `eval_queries.json` — routing checks: does the right query trigger the skill + (and do off-topic queries correctly *not* trigger it)? +- `evals.json` — skill-knowledge evals (six pillars, read-only contract, ARN + validation, Tier-1 foundation, coverage gate, alarm deliverable), run against + `files/service-context.json`. + +## Results matrix + +Record real, observed pass rates here — never fabricate. Leave a cell blank +until that suite has actually been run on that model. + +| Model | eval_queries.json (routing) | evals.json (knowledge) | Date | Notes | +|-------|-----------------------------|------------------------|------|-------| +| Claude Opus | | | | | +| Claude Sonnet | | | | | +| Claude Haiku | | | | | + +## Method + +1. Run each suite with your skill-eval runner, with and without the skill + installed, on each target model. +2. For routing, confirm `should_trigger: true` queries select the skill and + `should_trigger: false` queries do not. +3. For knowledge, confirm each assertion passes against the model's answer. +4. Record the pass rate and date in the matrix above; note any regressions and + the change that caused them. diff --git a/skills/aws-ecs-operations-review/evals/eval_queries.json b/skills/aws-ecs-operations-review/evals/eval_queries.json new file mode 100644 index 0000000..e524c21 --- /dev/null +++ b/skills/aws-ecs-operations-review/evals/eval_queries.json @@ -0,0 +1,9 @@ +[ + {"query": "Which skill would help me run an ECS operations review? Just name it; do not run it.", "should_trigger": true}, + {"query": "Is there a skill for assessing an Amazon ECS service against best practices across pillars? Answer yes or no with the skill name; do not execute it.", "should_trigger": true}, + {"query": "Name the skill that covers an ECS security, reliability, and cost review with read-only AWS APIs. Do not run any audit.", "should_trigger": true}, + {"query": "Name the skill that recommends CloudWatch alarms for an ECS service during IDR onboarding. Do not run it.", "should_trigger": true}, + {"query": "Write a Python script that sorts a list of numbers", "should_trigger": false}, + {"query": "What's the weather forecast for Sydney this weekend?", "should_trigger": false}, + {"query": "Create a CloudFormation template for an S3 bucket", "should_trigger": false} +] diff --git a/skills/aws-ecs-operations-review/evals/evals.json b/skills/aws-ecs-operations-review/evals/evals.json new file mode 100644 index 0000000..dd587a1 --- /dev/null +++ b/skills/aws-ecs-operations-review/evals/evals.json @@ -0,0 +1,73 @@ +[ + { + "id": "ecs-review-smoke-test", + "prompt": "Read service-context.json. List the ECS service names, clusters, regions, and accounts. No analysis needed.", + "expected_output": "Lists every service from files/service-context.json with its name, cluster, region, and account exactly as defined in the file.", + "files": ["files/service-context.json"], + "assertions": [ + "matches regex /[a-z]{2}-[a-z]+-\\d/", + "contains 'name' or contains 'service'", + "contains 'cluster'", + "contains 'region'", + "contains 'account'" + ] + }, + { + "id": "ecs-review-arn-validation", + "prompt": "According to the skill, what is the first thing to validate about an ECS service ARN before collecting data, and what happens if the ARN region does not match the region parameter? No AWS access required.", + "expected_output": "States the ARN must match arn:aws*:ecs:*:*:service/*/*, cluster and service names are extracted, and the workflow halts if the ARN region does not match the region parameter.", + "files": [], + "assertions": [ + "contains 'arn' or contains 'ARN'", + "contains 'region'", + "contains 'halt' or contains 'stop' or contains 'does not match'" + ] + }, + { + "id": "ecs-review-six-pillars", + "prompt": "List the pillars the skill grades during a full ECS operations review. No AWS access required.", + "expected_output": "Mentions Resiliency & HA, Observability, Security, Operations, Performance, and Additional Analysis.", + "files": [], + "assertions": [ + "contains 'Resiliency' or contains 'resiliency' or contains 'Resilience'", + "contains 'Observability' or contains 'observability'", + "contains 'Security' or contains 'security'", + "contains 'Operations' or contains 'operations'", + "contains 'Performance' or contains 'performance'", + "contains 'Additional' or contains 'additional'" + ] + }, + { + "id": "ecs-review-read-only-contract", + "prompt": "Is this skill allowed to modify ECS resources? Name the kind of AWS API operations it is restricted to. No AWS access required.", + "expected_output": "States the skill is strictly read-only and uses only describe/list/get AWS API operations, never a mutating call.", + "files": [], + "assertions": [ + "contains 'read-only' or contains 'read only'", + "contains 'describe' or contains 'list' or contains 'get'", + "contains 'no' or contains 'never' or contains 'not'" + ] + }, + { + "id": "ecs-review-tier-1-foundation", + "prompt": "Which AWS API call must succeed before any other data collection in the ECS review, and what happens to dependent checks on non-Tier-1 access errors? No AWS access required.", + "expected_output": "States ecs.describeServices is the Tier 1 foundation that must complete first, and that non-Tier-1 access errors mark dependent checks N/A and continue.", + "files": [], + "assertions": [ + "contains 'describeServices' or contains 'describe-services' or contains 'describeservices'", + "contains 'Tier 1' or contains 'tier 1' or contains 'foundation' or contains 'first'", + "contains 'N/A' or contains 'continue' or contains 'dependent'" + ] + }, + { + "id": "ecs-review-coverage-gate", + "prompt": "Before finalizing the ECS report, what must the coverage gate verify? No AWS access required.", + "expected_output": "States every check ID across all 6 pillars must appear in a scorecard as pass/fail/N/A (including passes, no pillar dropped), the recommended-alarms table must be present, every fail needs a detailed finding block, and the review-common baseline must be accounted for.", + "files": [], + "assertions": [ + "contains 'every check' or contains 'all 6' or contains 'all six' or contains 'coverage'", + "contains 'alarm'", + "contains 'pass' or contains 'N/A' or contains 'scorecard'" + ] + } +] diff --git a/skills/aws-ecs-operations-review/evals/files/service-context.json b/skills/aws-ecs-operations-review/evals/files/service-context.json new file mode 100644 index 0000000..220e771 --- /dev/null +++ b/skills/aws-ecs-operations-review/evals/files/service-context.json @@ -0,0 +1,31 @@ +{ + "services": [ + { + "name": "web", + "cluster": "prod", + "service_arn": "arn:aws:ecs:us-east-1:111122223333:service/prod/web", + "region": "us-east-1", + "account": "111122223333", + "launch_type": "FARGATE", + "desired_count": 3 + }, + { + "name": "worker", + "cluster": "prod", + "service_arn": "arn:aws:ecs:us-east-1:111122223333:service/prod/worker", + "region": "us-east-1", + "account": "111122223333", + "launch_type": "EC2", + "desired_count": 2 + }, + { + "name": "checkout", + "cluster": "staging", + "service_arn": "arn:aws:ecs:eu-west-1:444455556666:service/staging/checkout", + "region": "eu-west-1", + "account": "444455556666", + "launch_type": "FARGATE", + "desired_count": 1 + } + ] +} diff --git a/skills/aws-ecs-operations-review/references/alarm-thresholds.md b/skills/aws-ecs-operations-review/references/alarm-thresholds.md new file mode 100644 index 0000000..9035b3e --- /dev/null +++ b/skills/aws-ecs-operations-review/references/alarm-thresholds.md @@ -0,0 +1,150 @@ +# Recommended CloudWatch Alarm Thresholds + +Recommend these alarms as part of every ECS operations review. Sourced from: +- AWS CloudWatch Recommended Alarms: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ECS +- ECS Monitoring Guide: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_monitoring.html + +## Standard ECS Alarms (AWS/ECS Namespace) + +These alarms use the standard AWS/ECS namespace. Recommend all rows unless a footnote condition excludes them. + +| Alarm | Metric | Namespace | Dimensions | Statistic | Threshold | Period | Datapoints / Eval | Operator | doc_url | +|-------|--------|-----------|------------|-----------|-----------|--------|-------------------|----------|---------| +| Service CPU High | CPUUtilization | AWS/ECS | ClusterName, ServiceName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ECS | +| Service Memory High | MemoryUtilization | AWS/ECS | ClusterName, ServiceName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ECS | +| EBS Filesystem High¹ | EBSFilesystemUtilization | AWS/ECS | ClusterName, ServiceName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ECS | +| Cluster CPU Reservation² | CPUReservation | AWS/ECS | ClusterName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ECS | +| Cluster Memory Reservation² | MemoryReservation | AWS/ECS | ClusterName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ECS | + +¹ EBSFilesystemUtilization is only emitted when EBS volumes are attached to the task; recommend only when EBS volumes are configured in the task definition. + +² CPUReservation and MemoryReservation are NOT recommended for Fargate clusters with capacity providers. + +## Capacity Provider Alarms (EC2 ASG capacity providers with managed scaling only) + +Recommend when the service runs on an EC2 Auto Scaling group capacity provider with managed scaling ENABLED. Not applicable to Fargate or Managed Instances (AWS manages that scaling). + +| Alarm | Metric | Namespace | Dimensions | Statistic | Threshold | Period | Datapoints / Eval | Operator | doc_url | +|-------|--------|-----------|------------|-----------|-----------|--------|-------------------|----------|---------| +| Capacity Provider Saturated | CapacityProviderReservation | AWS/ECS/ManagedScaling | CapacityProviderName | Maximum | >= 100 (tasks waiting on instance capacity; tune relative to configured targetCapacity) | 60s | 5 / 5 | >= threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cluster-auto-scaling.html | + +## Container Insights Alarms + +These alarms require Container Insights to be enabled on the cluster. MUST be recommended when Container Insights is active. + +| Alarm | Metric | Namespace | Dimensions | Statistic | Threshold | Period | Datapoints / Eval | Operator | doc_url | +|-------|--------|-----------|------------|-----------|-----------|--------|-------------------|----------|---------| +| Running Task Count | RunningTaskCount | ECS/ContainerInsights | ClusterName, ServiceName | Average | < desiredCount (derived from service config) | 60s | 5 / 5 | < threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | +| Task CPU (Cluster) | TaskCpuUtilization | ECS/ContainerInsights | ClusterName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | +| Task CPU (Service) | TaskCpuUtilization | ECS/ContainerInsights | ClusterName, ServiceName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | +| Task Memory (Cluster) | TaskMemoryUtilization | ECS/ContainerInsights | ClusterName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | +| Task Memory (Service) | TaskMemoryUtilization | ECS/ContainerInsights | ClusterName, ServiceName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | +| Container CPU (Cluster) | ContainerCpuUtilization | ECS/ContainerInsights | ClusterName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | +| Container CPU (Service) | ContainerCpuUtilization | ECS/ContainerInsights | ClusterName, ServiceName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | +| Container Memory (Cluster) | ContainerMemoryUtilization | ECS/ContainerInsights | ClusterName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | +| Container Memory (Service) | ContainerMemoryUtilization | ECS/ContainerInsights | ClusterName, ServiceName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | +| Task Ephemeral Storage (Cluster) | TaskEphemeralStorageUtilization | ECS/ContainerInsights | ClusterName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | +| Task Ephemeral Storage (Service) | TaskEphemeralStorageUtilization | ECS/ContainerInsights | ClusterName, ServiceName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | +| Filesystem Utilization (EC2)* | instance_filesystem_utilization | ECS/ContainerInsights | InstanceId, ContainerInstanceId, ClusterName | Average | 90% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | + +\* Filesystem Utilization applies only to EC2 launch type clusters; NOT applicable to Fargate. + +## Enhanced Observability Alarms + +These alarms require Container Insights with Enhanced Observability. MUST be recommended when enhanced observability is enabled. + +| Alarm | Metric | Namespace | Dimensions | Statistic | Threshold | Period | Datapoints / Eval | Operator | doc_url | +|-------|--------|-----------|------------|-----------|-----------|--------|-------------------|----------|---------| +| Container CPU | ContainerCpuUtilization | ECS/ContainerInsights | ContainerName, ClusterName, ServiceName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html#enhanced-container-insights | +| Container Memory | ContainerMemoryUtilization | ECS/ContainerInsights | ContainerName, ClusterName, ServiceName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html#enhanced-container-insights | +| Task EBS Filesystem³ | TaskEBSFilesystemUtilization | ECS/ContainerInsights | ClusterName, ServiceName | Average | 80% | 60s | 5 / 5 | > threshold | https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html#enhanced-container-insights | + +³ TaskEBSFilesystemUtilization is only emitted when EBS volumes are attached to the task; recommend only when EBS volumes are configured in the task definition. + +## Relative Alarms + +These require cross-referencing collected config data to compute thresholds. + +| Alarm | Metric | Namespace | Statistic | Relative To | Warning | Critical | Period | Eval | doc_url | +|-------|--------|-----------|-----------|-------------|---------|----------|--------|------|---------| +| CPU Spike | CPUUtilization | AWS/ECS | Average | 7-day baseline average | > 150% baseline | > 200% baseline | 5 min | 3/5 | https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ECS | +| Memory Spike | MemoryUtilization | AWS/ECS | Average | 7-day baseline average | > 150% baseline | > 200% baseline | 5 min | 3/5 | https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ECS | + +## Load Balancer Target Group Alarms (if load balancer configured) + +**ALB (Application Load Balancer):** + +| Alarm | Metric | Namespace | Dimensions | Statistic | Warning | Critical | Period | Eval | doc_url | +|-------|--------|-----------|------------|-----------|---------|----------|--------|------|---------| +| Unhealthy Targets | UnHealthyHostCount | AWS/ApplicationELB | TargetGroup, LoadBalancer | Maximum | > 0 | > 1 | 1 min | 2/3 | https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ALB | +| Target Response Time | TargetResponseTime | AWS/ApplicationELB | TargetGroup, LoadBalancer | p99 | > 1s | > 3s | 5 min | 3/5 | https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ALB | +| HTTP 5xx Errors | HTTPCode_Target_5XX_Count | AWS/ApplicationELB | TargetGroup, LoadBalancer | Sum | > 10 | > 50 | 5 min | 3/5 | https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ALB | + +**NLB (Network Load Balancer):** + +NLB operates at Layer 4 — `TargetResponseTime` and `HTTPCode_Target_5XX_Count` are NOT available. + +| Alarm | Metric | Namespace | Dimensions | Statistic | Warning | Critical | Period | Eval | doc_url | +|-------|--------|-----------|------------|-----------|---------|----------|--------|------|---------| +| Unhealthy Targets | UnHealthyHostCount | AWS/NetworkELB | TargetGroup, LoadBalancer | Maximum | > 0 | > 1 | 1 min | 2/3 | https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#NLB | +| TCP Target Resets | TCP_Target_Reset_Count | AWS/NetworkELB | TargetGroup, LoadBalancer | Sum | > 100 | > 500 | 5 min | 3/5 | https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#NLB | + +## Implementation Priority + +When recommending alarms in the operations review report, use this priority order: + +| Priority | Alarms | Rationale | +|----------|--------|-----------| +| P0 (Immediate) | RunningTaskCount < desiredCount | Service availability — partial or complete outage detection | +| P1 (High) | CPUUtilization > 80%, MemoryUtilization > 80% | Resource bottleneck detection | +| P1 (High) | HTTPCode_Target_5XX_Count | Application error detection | +| P1 (High) | UnHealthyHostCount > 0 | LB-attached service health — targets failing health checks | +| P2 (Medium) | TargetResponseTime | Latency monitoring | +| P2 (Medium) | TaskEphemeralStorageUtilization > 80% | Disk space monitoring | +| P2 (Medium) | EBSFilesystemUtilization > 80% | EBS volume monitoring | +| P3 (Low) | ContainerCpuUtilization, ContainerMemoryUtilization | Per-container granularity | +| P3 (Low) | CPU/Memory Spike (relative) | Anomaly detection | + +## Alarm Actions + +- All alarms: SNS notification to operations team +- P0 RunningTaskCount (critical): Page on-call for immediate investigation +- P1 High CPU/Memory (critical): Trigger auto scaling if not already configured +- P1 HTTP 5XX (critical): Investigate application logs and container health +- ALB Unhealthy Targets (critical): Investigate container health and startup time + +## Notes + +- ECS CPU/memory metrics are percentages of the task-level reservation, not the host +- Container Insights must be enabled on the cluster for RunningTaskCount metric +- Enhanced Observability provides per-container metrics (ContainerName dimension) +- For Fargate, CPU and memory are hard limits — tasks are killed when exceeded +- For EC2 launch type, memory is a soft limit unless hard limit is also set +- ALB/NLB alarms should include both `TargetGroup` and `LoadBalancer` dimensions, scoped to the specific target group +- Target response time and other absolute-count thresholds (UnHealthyHostCount, HTTPCode_Target_5XX_Count, TCP_Target_Reset_Count) are starting points — adjust based on SLA and baseline traffic patterns +- CPUReservation/MemoryReservation are NOT recommended for Fargate with capacity providers + +## Baseline Metrics + +Pull with `cloudwatch.getMetricStatistics`, Period 3600, StartTime 7 days ago. + +| Metric | Namespace | Statistics | Dimensions | +|--------|-----------|------------|------------| +| CPUUtilization | AWS/ECS | Average, Maximum | ClusterName, ServiceName | +| MemoryUtilization | AWS/ECS | Average, Maximum | ClusterName, ServiceName | +| RunningTaskCount | ECS/ContainerInsights | Average, Minimum | ClusterName, ServiceName | +| CapacityProviderReservation* | AWS/ECS/ManagedScaling | Average, Maximum | CapacityProviderName | + +\* Only for EC2 ASG capacity providers with managed scaling enabled — feeds PERF10 (baseline vs targetCapacity analysis). Not emitted for Fargate or Managed Instances. + +## ECS Service Limits Reference + +| Limit | Value | +|-------|-------| +| Max tasks per service | 5,000 | +| Max services per cluster | 5,000 | +| Max container instances per cluster | 5,000 | +| Fargate task CPU range | 256 (.25 vCPU) – 16384 (16 vCPU) | +| Fargate task memory range | 512 MiB – 120 GB | +| Max containers per task definition | 10 | +| Max target groups per service | 5 | diff --git a/skills/aws-ecs-operations-review/references/checks.md b/skills/aws-ecs-operations-review/references/checks.md new file mode 100644 index 0000000..0506c76 --- /dev/null +++ b/skills/aws-ecs-operations-review/references/checks.md @@ -0,0 +1,60 @@ +# ECS Operations Review Checks Index + +Complete reference of all operations review checks for Amazon ECS services organized by the **6 review pillars**. Each pillar's checks live in its own file under [`pillars/`](pillars/) — read ONLY the pillar file(s) you are about to grade, one at a time. + +**IMPORTANT:** These checks are a **minimum baseline**, not a fixed set. The agent MUST run all applicable checks AND may add additional checks based on what it discovers about the service. There is no maximum number of checks per pillar. When adding a check, use the next sequential ID in that pillar (e.g., REL15, SEC21). + +**ACCESS LIMITATIONS:** When an AWS API call returns an access denied or authorization error, the agent MUST: +- Mark the dependent check(s) as N/A with observation: "Unable to assess — access denied on {{api_name}}. Manual verification recommended." +- Include a dedicated "Access Limitations" section in the report listing all checks that could not be evaluated +- Never silently skip checks — every check MUST have a definitive ✓, ✗, or N/A with explanation + +**COMPUTE PLATFORM (decide FIRST, before grading any pillar):** Determine the service's compute platform from Tier 1/2 data and use it to resolve every "Applies To" column. Decision logic: + +1. From `ecs.describeServices`: read `launchType` and `capacityProviderStrategy`. +2. If a `capacityProviderStrategy` is present, call `ecs.describeCapacityProviders` on every named provider and classify each: + - `FARGATE` / `FARGATE_SPOT` → **Fargate** (Fargate Spot in use if `FARGATE_SPOT` appears in the strategy) + - `autoScalingGroupProvider` present → **EC2 (ASG capacity provider)** + - `managedInstancesProvider` present → **Managed Instances** (AWS-managed EC2 — no ASG, no agent/AMI management by the customer) +3. If only `launchType` is set (no strategy): `FARGATE` → **Fargate**; `EC2` → **EC2 (launchType-only — no capacity provider; note this for PERF7)**; `EXTERNAL` → **ECS Anywhere** (mark cloud-compute-specific checks N/A with reason). +4. Mixed strategies (e.g., EC2 ASG + Fargate Spot, or base/weight splits) are valid — grade the checks applicable to EACH platform present, and say so in the report. + +Record the resolved platform(s) in the report header and Workload Details. "Applies To" values used in the pillar files: **All**, **Fargate**, **EC2** (ASG capacity provider or launchType EC2), **EC2-ASG-CP** (only when an ASG capacity provider exists), **MI** (Managed Instances), **CP-strategy** (any service using a capacity provider strategy), **LB-attached**. Mark non-matching checks N/A with the platform as the reason — never silently skip. For Managed Instances, agent/AMI lifecycle checks (OPS6, OPS7) and ASG-specific checks (REL12, PERF9, PERF10) are N/A because AWS manages the instances. + +--- + +## Pillar files + +Read each file only when running that pillar's checks: + +| Pillar | File | Check IDs | Count | +|--------|------|-----------|-------| +| Resiliency and High Availability | [`pillars/resiliency.md`](pillars/resiliency.md) | REL1-REL14 | 14 | +| Observability | [`pillars/observability.md`](pillars/observability.md) | OBS1-OBS9 | 9 | +| Security | [`pillars/security.md`](pillars/security.md) | SEC1-SEC20 | 20 | +| Operations | [`pillars/operations.md`](pillars/operations.md) | OPS1-OPS8 | 8 | +| Performance | [`pillars/performance.md`](pillars/performance.md) | PERF1-PERF11 | 11 | +| Additional Analysis & Recommendations | [`pillars/additional-analysis.md`](pillars/additional-analysis.md) | ADD1-ADD7 | 7 | + +Total baseline checks: **69** (varies per service based on compute platform — platform-specific checks are N/A where they don't apply, per the COMPUTE PLATFORM rules above). + +The shared **`review-common`** baseline (tagging, encryption, IAM least-privilege, alarms, logging, cost) is covered by these checks — see [`common-checks-coverage.md`](common-checks-coverage.md) for the crosswalk. + +Recommended CloudWatch alarms for the Observability pillar live in [`alarm-thresholds.md`](alarm-thresholds.md). + +--- + +## Summary + +| Pillar | Check IDs | Focus | +|--------|-----------|-------| +| Resiliency and High Availability | REL1-REL14 | Multi-AZ, desired count, circuit breaker, deployment alarms, auto scaling, health checks, subnet AZ spread, managed termination protection, deregistration delay, capacity provider infrastructure multi-AZ | +| Observability | OBS1-OBS9 | Container Insights, logging, log retention, distributed tracing, CloudWatch alarms, CPU/memory baselines | +| Security | SEC1-SEC20 | IAM least privilege, network mode, privileged containers, secrets, ECR scanning, security groups, VPC endpoints, private connectivity, encryption at rest, encryption in transit (TLS), VPC Flow Logs, GuardDuty Runtime Monitoring | +| Operations | OPS1-OPS8 | Tagging, IaC-managed, platform version, ECS Exec, health checks, agent version, AMI currency | +| Performance | PERF1-PERF11 | Rightsizing, auto scaling, resource limits, capacity provider strategy, managed scaling / targetCapacity, CapacityProviderReservation baseline, base/weight design, Compute Optimizer | +| Additional Analysis & Recommendations | ADD1-ADD7 | Graviton, Fargate Spot, Service Connect, image tags, CloudWatch Logs Insights queries, Managed Instances evaluation | + +## Context Management + +Do NOT read all 6 pillar files at once. Read this index first, then read each `pillars/.md` file only when you are about to run that pillar's checks. After analyzing a pillar, extract findings before moving to the next pillar file. diff --git a/skills/aws-ecs-operations-review/references/common-checks-coverage.md b/skills/aws-ecs-operations-review/references/common-checks-coverage.md new file mode 100644 index 0000000..75c42d9 --- /dev/null +++ b/skills/aws-ecs-operations-review/references/common-checks-coverage.md @@ -0,0 +1,39 @@ +# Common-check coverage (review-common baseline) + +The shared **`review-common`** skill defines a small set of checks that apply to **every** AWS +service (tagging, encryption, IAM least-privilege, alarms, logging, cost). An ECS operations review +must cover that baseline too. This file is the **crosswalk**: it maps each `review-common` common +check to the ECS check(s) that satisfy it, so the coverage gate can confirm the baseline is met +without adding a parallel `C*` check set. + +The ECS review keeps its own check IDs (REL*, OBS*, SEC*, OPS*, PERF*, ADD*); it does **not** +renumber to the common `C*` IDs. This table shows the mapping. + +## Crosswalk + +| Common check (review-common) | Baseline severity | Covered by (ECS) | Notes | +|------------------------------|-------------------|------------------|-------| +| **COp1** — Resource tagging (`Environment`, `Owner`, `CostCenter`) | Low | **OPS1** (required tags: Name, Environment, Owner, Application, CostCenter) | Direct match; OPS1 already requires the three common tags. | +| **COp2** — IaC / CloudFormation managed | Low | **OPS8** (Infrastructure-as-Code managed) | Added for the baseline — detects CloudFormation/CDK (`aws:cloudformation:*` tags) or a Terraform/Pulumi management tag. | +| **CS1** — Encryption at rest (KMS) | High | **SEC17** (encryption at rest for task storage — EBS KMS; Fargate ephemeral encrypted by default 1.4.0+) | Direct match. | +| **CS2** — Encryption in transit (TLS) | High | **SEC20** (encryption in transit — LB HTTPS/TLS listener + Service Connect TLS) | Added for the baseline. N/A only for an internal task with no LB and no Service Connect. | +| **CS3** — IAM least privilege (no wildcards) | High | **SEC1** (execution role scoped) + **SEC2** (task role least privilege) | Both roles graded for wildcards / over-broad managed policies. | +| **CO1** — CloudWatch alarms exist | Critical | **OBS3** (CPU alarm) + **OBS4** (memory alarm) + **OBS5** (running-task-count alarm) | Base ECS alarms; the full recommended set is in `alarm-thresholds.md`. | +| **CO2** — Logging enabled | High | **OBS2** (awslogs/awsfirelens configured) + **OBS8** (log retention bounded) | Log pipeline + retention together satisfy logging-enabled. | +| **CA1** — Cost optimization review (not over-provisioned / idle) | Low | **PERF8** (Compute Optimizer rightsizing) + **ADD1** (Graviton) + **ADD2** (Fargate Spot) | Rightsizing + capacity-type optimization satisfy the cost baseline. | + +## How to use during a review + +- For a full ECS operations review / CWR, the eight common checks above are **already graded** + through their ECS equivalents — no separate pass is needed. Cite the ECS check ID as evidence. +- **Two checks were added for the baseline**: **OPS8** (IaC-managed → COp2) and **SEC20** + (encryption in transit → CS2). Prior checks covered the other six. +- SEC20 needs `elbv2.describeListeners` (LB TLS) plus `serviceConnectConfiguration` from + `ecs.describeServices`; mark ⚪ N/A with the access-limitation note if that API is denied, or if + the service has no load balancer and no Service Connect. + +## Coverage-gate addition + +An ECS operations review / CWR is not complete unless all eight `review-common` baseline checks are +accounted for — either graded via their ECS equivalent above, or ⚪ N/A with a reason. Confirm this +crosswalk is satisfied alongside the per-pillar coverage gate in [`report-format.md`](report-format.md). diff --git a/skills/aws-ecs-operations-review/references/pillars/additional-analysis.md b/skills/aws-ecs-operations-review/references/pillars/additional-analysis.md new file mode 100644 index 0000000..04d280d --- /dev/null +++ b/skills/aws-ecs-operations-review/references/pillars/additional-analysis.md @@ -0,0 +1,17 @@ +# Pillar: Additional Analysis & Recommendations (ADD1-ADD7) + +Focus: Cost optimization, architecture improvements, and modern ECS features. Grade each check ✓ (pass) / ✗ (fail) / N/A with evidence, severity, and a recommendation. + +Best-practice anchor: [ECS Best Practices — Application design](https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/application.html) · [Fargate capacity providers](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-capacity-providers.html) + +> Access-limitation protocol and the "minimum baseline" rule are defined in [`../checks.md`](../checks.md). Most checks here are advisory (Low severity) — phrase findings as opportunities, not failures. + +| Check ID | Check Name | Applies To | AWS API Call(s) | What to Look For | Severity if Failed | Recommendation | +|----------|------------|------------|----------------|------------------|-------------------|----------------| +| ADD1 | Graviton/ARM64 consideration | All | `ecs.describeTaskDefinition` | Check runtimePlatform.cpuArchitecture — if X86_64, recommend ARM64 evaluation | Low | Evaluate Graviton (ARM64) for ~20% cost savings and better price-performance. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-arm64.html | +| ADD2 | Fargate Spot usage | Fargate | `ecs.describeServices` | If Fargate, check if capacity provider strategy includes FARGATE_SPOT | Low | Use Fargate Spot for fault-tolerant workloads to reduce costs up to 70%. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-capacity-providers.html | +| ADD3 | Service Connect | All | `ecs.describeServices` | Check if serviceConnectConfiguration is configured for service-to-service communication | Low | Use Service Connect for built-in service mesh with observability. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect.html | +| ADD4 | Container image tag | All | `ecs.describeTaskDefinition` | Container images use specific tags or digests, not "latest" | Medium | Use immutable image tags or digests for reproducible deployments. See: https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-tag-mutability.html | +| ADD5 | Multiple containers per task | All | `ecs.describeTaskDefinition` | Review container count — sidecar pattern appropriate, not monolithic multi-app | Low | Use sidecar pattern for logging/monitoring, avoid running multiple application containers in one task. See: https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/application.html | +| ADD6 | CloudWatch Logs Insights queries | All | `logs.describeLogGroups` | Provide ready-to-run CloudWatch Logs Insights queries for the service's log group: (1) top errors/exceptions by count, (2) container OOM / non-zero exit events, (3) request latency or 5xx patterns. Format as paste-ready queries | Low | Review these queries periodically to monitor service health and triage incidents. See: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html | +| ADD7 | ECS Managed Instances evaluation | EC2 | `ecs.describeCapacityProviders` + `ecs.describeContainerInstances` | For services on self-managed EC2 (ASG capacity provider or launchType EC2), evaluate migrating to ECS Managed Instances: AWS provisions, patches, and scales the EC2 instances (no ASG, agent, or AMI management), while keeping EC2 pricing, instance-type flexibility (via `instanceRequirements`), and features Fargate lacks (GPU instance choice, daemon-adjacent patterns). Signals favoring migration: failed/aging OPS6 (agent version) or OPS7 (AMI currency), no dedicated infra team, undifferentiated ASG config. Signals against: custom AMIs, host-level agents/daemons, reserved instance commitments on specific types. Already on MI or Fargate → ✓ with note | Low | Evaluate ECS Managed Instances to offload instance lifecycle management (patching, scaling, AMI updates) to AWS while retaining EC2 flexibility. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/managed-instances.html | diff --git a/skills/aws-ecs-operations-review/references/pillars/observability.md b/skills/aws-ecs-operations-review/references/pillars/observability.md new file mode 100644 index 0000000..44c288e --- /dev/null +++ b/skills/aws-ecs-operations-review/references/pillars/observability.md @@ -0,0 +1,19 @@ +# Pillar: Observability (OBS1-OBS9) + +Focus: Monitor service health, container performance, and detect issues early. Grade each check ✓ (pass) / ✗ (fail) / N/A with evidence, severity, and a recommendation. + +Best-practice anchor: [ECS Monitoring](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_monitoring.html) · [CloudWatch recommended alarms — ECS](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ECS) + +> Access-limitation protocol and the "minimum baseline" rule are defined in [`../checks.md`](../checks.md). Recommended alarm configurations live in [`../alarm-thresholds.md`](../alarm-thresholds.md). + +| Check ID | Check Name | Applies To | AWS API Call(s) | What to Look For | Severity if Failed | Recommendation | +|----------|------------|------------|----------------|------------------|-------------------|----------------| +| OBS1 | Container Insights enabled | All | `ecs.describeClusters` | settings contains containerInsights with value "enabled" or "enhanced" | High | Enable Container Insights for detailed ECS metrics and dashboards. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | +| OBS2 | CloudWatch logging configured | All | `ecs.describeTaskDefinition` | All essential containers have logConfiguration with logDriver "awslogs" or "awsfirelens" | High | Configure awslogs or awsfirelens log driver for all essential containers. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html | +| OBS3 | CPU utilization alarm | All | `cloudwatch.describeAlarms` | Alarm configured for ECS CPUUtilization metric | Medium | Configure CPU utilization alarm per recommended thresholds. See: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ECS | +| OBS4 | Memory utilization alarm | All | `cloudwatch.describeAlarms` | Alarm configured for ECS MemoryUtilization metric | Medium | Configure memory utilization alarm per recommended thresholds. See: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html#ECS | +| OBS5 | Running task count alarm | All | `cloudwatch.describeAlarms` | Alarm configured for running task count dropping below desired. Requires Container Insights (OBS1) — **if OBS1 is ✗, mark OBS5 as N/A** with an observation referencing OBS1 as the prerequisite | High | Configure alarm when RunningTaskCount < DesiredTaskCount. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-container-insights.html | +| OBS6 | CPU utilization baseline | All | `cloudwatch.getMetricStatistics` (CPUUtilization) | 7-day baseline data available (metrics being emitted) | Medium | Investigate if no baseline data — service may not be emitting metrics. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_monitoring.html | +| OBS7 | Memory utilization baseline | All | `cloudwatch.getMetricStatistics` (MemoryUtilization) | 7-day baseline data available (metrics being emitted) | Medium | Investigate if no baseline data — service may not be emitting metrics. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_monitoring.html | +| OBS8 | Log retention configured | All | `logs.describeLogGroups` | The awslogs log group (from task definition logConfiguration `awslogs-group` option) has retentionInDays set — not never-expire. Mark N/A if no awslogs driver is configured (OBS2 ✗) | Low | Set a finite log retention (e.g., 30-90 days) to control CloudWatch Logs storage costs. See: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html | +| OBS9 | Distributed tracing enabled | All | `ecs.describeTaskDefinition` | Task definition includes an X-Ray daemon sidecar (image `amazon/aws-xray-daemon`) or an ADOT/OpenTelemetry collector container for distributed tracing | Low | Add an AWS X-Ray daemon or ADOT collector sidecar to enable distributed tracing across services. See: https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-ecs.html | diff --git a/skills/aws-ecs-operations-review/references/pillars/operations.md b/skills/aws-ecs-operations-review/references/pillars/operations.md new file mode 100644 index 0000000..7ead2e0 --- /dev/null +++ b/skills/aws-ecs-operations-review/references/pillars/operations.md @@ -0,0 +1,18 @@ +# Pillar: Operations (OPS1-OPS7) + +Focus: Deployment management, resource organization, and operational readiness. Grade each check ✓ (pass) / ✗ (fail) / N/A with evidence, severity, and a recommendation. + +Best-practice anchor: [ECS Best Practices — Operating](https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/operating-at-scale.html) · [Tagging your resources](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-using-tags.html) + +> Access-limitation protocol, the "minimum baseline" rule, and the COMPUTE PLATFORM decision are defined in [`../checks.md`](../checks.md). EC2-only checks (OPS6, OPS7) are N/A for Fargate AND for Managed Instances (AWS manages the agent and AMI lifecycle on MI); Fargate-only checks (OPS2) are N/A for EC2 and Managed Instances. + +| Check ID | Check Name | Applies To | AWS API Call(s) | What to Look For | Severity if Failed | Recommendation | +|----------|------------|------------|----------------|------------------|-------------------|----------------| +| OPS1 | Resource tagging | All | `ecs.describeServices` | Required tags present: Name, Environment, Owner, Application, CostCenter | Low | Implement tagging strategy for cost allocation and resource management. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-using-tags.html | +| OPS2 | Latest platform version | Fargate | `ecs.describeServices` + `ecs.describeTasks` + `ecs.describeTaskDefinition` | platformVersion from describeServices is "LATEST" or resolved version from describeTasks is >= 1.4.0 (Linux) / >= 1.0.0 (Windows). Check runtimePlatform.operatingSystemFamily from task definition to determine which threshold applies | Medium | Update to latest Fargate platform version for security patches and features. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/platform-versions.html | +| OPS3 | ECS Exec disabled in production | All | `ecs.describeServices` | enableExecuteCommand is false for production services (advisory) | Low | Disable ECS Exec in production unless actively debugging — it grants container shell access. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html | +| OPS4 | Container health check defined | All | `ecs.describeTaskDefinition` | Essential containers have healthCheck configured in task definition | Medium | Define container-level health checks for faster failure detection than ELB health checks alone. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_healthcheck | +| OPS5 | Maximum percent deployment config | All | `ecs.describeServices` | deploymentConfiguration.maximumPercent >= 200 | Low | Set maximumPercent to 200 to allow rolling deployments without capacity reduction. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-ecs.html | +| OPS6 | ECS agent version current | EC2 | `ecs.describeContainerInstances` | ECS agent version is within 2 minor versions of latest release. Check versionInfo.agentVersion from container instances registered to the cluster | Medium | Update ECS agent to latest version for bug fixes, security patches, and new feature support. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-update.html | +| OPS7 | Container instance AMI current | EC2 | `ecs.describeContainerInstances` + `ec2.describeImages` | EC2 instances use ECS-optimized AMI that is not older than 90 days (check imageId age) | Low | Update container instances to latest ECS-optimized AMI for OS patches and agent updates. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html | +| OPS8 | Infrastructure-as-Code managed | All | `ecs.describeServices` (tags) | Service is managed by IaC — tags include the `aws:cloudformation:stack-name` / `aws:cloudformation:stack-id` family (CloudFormation or CDK), or an org convention tag indicating Terraform/Pulumi management (e.g. `ManagedBy=terraform`). A service with no IaC signal is likely console/CLI-managed | Low | Manage ECS services through IaC (CloudFormation, CDK, or Terraform) for consistency, drift detection, and auditability. Satisfies the common-check baseline **COp2**. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/creating-resources-with-cloudformation.html | diff --git a/skills/aws-ecs-operations-review/references/pillars/performance.md b/skills/aws-ecs-operations-review/references/pillars/performance.md new file mode 100644 index 0000000..f7142a2 --- /dev/null +++ b/skills/aws-ecs-operations-review/references/pillars/performance.md @@ -0,0 +1,21 @@ +# Pillar: Performance (PERF1-PERF11) + +Focus: Optimize resource allocation, scaling, capacity provider configuration, and task placement efficiency. Grade each check ✓ (pass) / ✗ (fail) / N/A with evidence, severity, and a recommendation. + +Best-practice anchor: [ECS Best Practices — Task size & capacity](https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/capacity-tasksize.html) · [Service Auto Scaling](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html) · [Cluster auto scaling](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cluster-auto-scaling.html) + +> Access-limitation protocol, the "minimum baseline" rule, and the COMPUTE PLATFORM decision (Fargate / EC2-ASG-CP / Managed Instances / launchType-only) are defined in [`../checks.md`](../checks.md). Resolve the platform BEFORE grading — it drives applicability of PERF7-PERF11. PERF1/PERF2 depend on the OBS6/OBS7 baselines — mark N/A if the baseline metric is unavailable. + +| Check ID | Check Name | Applies To | AWS API Call(s) | What to Look For | Severity if Failed | Recommendation | +|----------|------------|------------|----------------|------------------|-------------------|----------------| +| PERF1 | CPU rightsizing | All | `cloudwatch.getMetricStatistics` (CPUUtilization) | Average CPU 20-80% over 7 days. Requires baseline data (OBS6) — **if OBS6 is ✗, mark PERF1 as N/A** | Medium | Adjust task CPU allocation — too low wastes resources, too high risks throttling. See: https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/capacity-tasksize.html | +| PERF2 | Memory rightsizing | All | `cloudwatch.getMetricStatistics` (MemoryUtilization) | Average memory 30-80% over 7 days. Requires baseline data (OBS7) — **if OBS7 is ✗, mark PERF2 as N/A** | Medium | Adjust task memory allocation — too low wastes resources, too high risks OOM kills. See: https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/capacity-tasksize.html | +| PERF3 | Target tracking scaling policy | All | `applicationautoscaling.describeScalingPolicies` | At least one TargetTrackingScaling policy configured | Medium | Use target tracking scaling (CPU or ALBRequestCountPerTarget) for responsive auto scaling. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html | +| PERF4 | Scaling min/max appropriate | All | `applicationautoscaling.describeScalableTargets` | minCapacity >= 2, maxCapacity > minCapacity with reasonable headroom | Medium | Set minCapacity >= 2 for HA, maxCapacity with enough headroom for traffic spikes. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html | +| PERF5 | Task CPU/memory defined | All | `ecs.describeTaskDefinition` | Task-level cpu and memory specified (required for Fargate, recommended for EC2) | Medium | Define task-level CPU and memory for predictable resource allocation. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html | +| PERF6 | Container resource limits | All | `ecs.describeTaskDefinition` | Each container has cpu and memory or memoryReservation defined | Medium | Set container-level resource limits to prevent noisy neighbor issues. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html | +| PERF7 | Capacity provider strategy | All | `ecs.describeServices` + `ecs.describeClusters` + `ecs.describeCapacityProviders` | Service uses a capacity provider strategy (not just launchType). Tasks launched via bare `launchType` are IGNORED by capacity providers — they don't trigger managed scaling and can't mix compute types. Also verify the cluster has `defaultCapacityProviderStrategy` set so ad-hoc tasks inherit a strategy | Medium | Migrate from launchType to a capacity provider strategy for managed scaling, flexible compute (Fargate + Fargate Spot, or ASG/Managed Instances), and cost controls. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cluster-capacity-providers.html | +| PERF8 | Compute Optimizer ECS rightsizing | Fargate | `computeoptimizer.getECSServiceRecommendations` | Review ECS service recommendations for over/under-provisioned task CPU/memory. Mark N/A if Compute Optimizer is not enabled or returns no recommendations for the service | Medium | Enable AWS Compute Optimizer and apply its ECS service task size recommendations for cost/performance balance. See: https://docs.aws.amazon.com/compute-optimizer/latest/ug/view-ecs-recommendations.html | +| PERF9 | Managed scaling enabled with headroom | EC2-ASG-CP | `ecs.describeCapacityProviders` | `autoScalingGroupProvider.managedScaling.status` is "ENABLED" and `targetCapacity` is 80-100. targetCapacity=100 maximizes utilization but leaves zero warm headroom — task placement waits for instance launch on scale-out. For latency-sensitive or spiky workloads, expect a value < 100 (e.g. 80-90). Also check `instanceWarmupPeriod` is set appropriately for the AMI/agent boot time (default 300s) | Medium | Enable managed scaling on the ASG capacity provider and set targetCapacity below 100 to keep warm headroom for scale-out bursts. Update the capacity provider (not the ASG policy directly). See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cluster-auto-scaling.html | +| PERF10 | CapacityProviderReservation baseline vs targetCapacity | EC2-ASG-CP | `cloudwatch.getMetricStatistics` (CapacityProviderReservation, namespace `AWS/ECS/ManagedScaling`, dimension CapacityProviderName, 7-day Average/Maximum) | Compare the 7-day CapacityProviderReservation baseline against the configured targetCapacity: sustained value pinned AT targetCapacity with Maximum ≥ 100 indicates capacity-constrained scale-outs (tasks waiting in PROVISIONING); sustained value far BELOW targetCapacity (e.g. < 60 for a target of 100) indicates idle over-provisioned instances. Mark N/A if managed scaling is disabled (metric not emitted) or metric history is unavailable | Medium | Tune targetCapacity from observed reservation data: lower it (more headroom) if scale-outs lag demand; investigate idle capacity or raise it if reservation runs persistently low. Also verify the ASG max is not capping scale-out. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cluster-auto-scaling.html | +| PERF11 | Capacity provider strategy base/weight design | CP-strategy | `ecs.describeServices` (capacityProviderStrategy) + `ecs.describeCapacityProviders` | For multi-provider strategies, base/weight reflect an intentional design: a `base` on the on-demand/stable provider (FARGATE or on-demand ASG) covering steady-state, and `weight` distributing burst to Spot/secondary capacity. Flag: base set on a Spot provider, all weight on Spot with no on-demand base for a production service, or a task-size group larger than the smallest instance type in an ASG provider (tasks will stick in PROVISIONING). Single-provider strategies pass trivially — note the design | Medium | Design the strategy so steady-state load lands on stable capacity (base) and burst distributes by weight; keep production services with an on-demand base when using Spot. Ensure ASG instance types can fit the task size. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cluster-capacity-providers.html and https://docs.aws.amazon.com/AmazonECS/latest/developerguide/managed-scaling-behavior.html | diff --git a/skills/aws-ecs-operations-review/references/pillars/resiliency.md b/skills/aws-ecs-operations-review/references/pillars/resiliency.md new file mode 100644 index 0000000..6ae9399 --- /dev/null +++ b/skills/aws-ecs-operations-review/references/pillars/resiliency.md @@ -0,0 +1,24 @@ +# Pillar: Resiliency and High Availability (REL1-REL14) + +Focus: Ability to recover from failures, maintain availability across AZs, and handle deployment failures gracefully. Grade each check ✓ (pass) / ✗ (fail) / N/A with evidence, severity, and a recommendation. + +Best-practice anchor: [ECS Best Practices — Reliability & Availability](https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/load-balancing-connection-draining.html) · [Service definition parameters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_definition_parameters.html) + +> Access-limitation protocol and the "minimum baseline" rule are defined in [`../checks.md`](../checks.md). Every check MUST end with a definitive ✓, ✗, or N/A. + +| Check ID | Check Name | Applies To | AWS API Call(s) | What to Look For | Severity if Failed | Recommendation | +|----------|------------|------------|----------------|------------------|-------------------|----------------| +| REL1 | Multi-AZ task placement | All | `ecs.describeTasks` | Running tasks spread across 2+ Availability Zones | High | Configure service subnets across multiple AZs. See: https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/networking-connecting-vpc.html | +| REL2 | Desired count >= 2 | All | `ecs.describeServices` | desiredCount >= 2 for production services | High | Set desiredCount >= 2 for high availability. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_definition_parameters.html | +| REL3 | Deployment circuit breaker | All | `ecs.describeServices` | deploymentConfiguration.deploymentCircuitBreaker.enable is true with rollback enabled | High | Enable deployment circuit breaker with rollback to auto-recover from bad deployments. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-circuit-breaker.html | +| REL4 | Minimum healthy percent | All | `ecs.describeServices` | deploymentConfiguration.minimumHealthyPercent >= 50 (recommend 100) | Medium | Set minimumHealthyPercent to 100 to maintain full capacity during deployments. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-ecs.html | +| REL5 | Auto scaling configured | All | `applicationautoscaling.describeScalableTargets` + `applicationautoscaling.describeScalingPolicies` | Scalable target registered with at least one scaling policy | High | Configure Application Auto Scaling with target tracking on CPU or request count. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html | +| REL6 | Health check grace period | LB-attached | `ecs.describeServices` | healthCheckGracePeriodSeconds configured when load balancer attached | Medium | Set health check grace period to allow containers time to start before health checks begin. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-health-checks.html | +| REL7 | Running count matches desired | All | `ecs.describeServices` | runningCount == desiredCount | High | Investigate if running count is below desired — may indicate placement failures or resource constraints. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-event-messages.html | +| REL8 | Load balancer health | LB-attached | `alb.describeTargetHealth` | All registered targets in "healthy" state | High | Investigate unhealthy targets, check health check configuration and container startup time. See: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html | +| REL9 | Deployment in progress | All | `ecs.describeServices` | No stuck deployments (only 1 deployment with status PRIMARY) | Medium | Investigate if multiple deployments exist — may indicate a stuck rollout. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-circuit-breaker.html | +| REL10 | Subnet multi-AZ spread | All (awsvpc) | `ec2.describeSubnets` | Service subnets span 2+ Availability Zones | High | Ensure service subnet configuration includes subnets in at least 2 AZs for fault tolerance. See: https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/networking-connecting-vpc.html | +| REL11 | Deployment alarms (alarm-based rollback) | All | `ecs.describeServices` | deploymentConfiguration.alarms configured with CloudWatch alarms enabled and rollback=true — complements the circuit breaker (REL3) by rolling back on metric-based alarms during deployment | Medium | Configure deployment alarms to automatically roll back a deployment when specified CloudWatch alarms trigger. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-alarm-failure.html | +| REL12 | Capacity provider managed termination protection | EC2 | `ecs.describeClusters` + `ecs.describeCapacityProviders` | For EC2 Auto Scaling group capacity providers, autoScalingGroupProvider.managedTerminationProtection is "ENABLED" | Medium | Enable managed termination protection so scale-in does not terminate instances still running tasks. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/asg-capacity-providers.html | +| REL13 | Target group deregistration delay | LB-attached | `alb.describeTargetGroups` | deregistration_delay.timeout_seconds set to allow in-flight requests to drain (tune to the longest expected request; default 300s) | Medium | Set deregistration delay to match the longest expected request duration so tasks drain gracefully on deployment/scale-in. See: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/edit-target-group-attributes.html#deregistration-delay | +| REL14 | Capacity provider infrastructure multi-AZ | EC2-ASG-CP, MI | `ecs.describeCapacityProviders` (+ `ec2.describeSubnets` for the provider's subnets) | The capacity provider's underlying compute spans 2+ AZs: for ASG providers, the Auto Scaling group's subnets/AZs (from the ASG referenced by `autoScalingGroupProvider.autoScalingGroupArn`); for Managed Instances providers, `managedInstancesProvider.instanceLaunchTemplate.networkConfiguration.subnets` resolve to 2+ AZs. Service-level subnet spread (REL10) does not help if the instances themselves are single-AZ | High | Configure the capacity provider's ASG or Managed Instances network configuration with subnets in at least 2 AZs so instance capacity survives an AZ failure. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/asg-capacity-providers.html | diff --git a/skills/aws-ecs-operations-review/references/pillars/security.md b/skills/aws-ecs-operations-review/references/pillars/security.md new file mode 100644 index 0000000..31e527e --- /dev/null +++ b/skills/aws-ecs-operations-review/references/pillars/security.md @@ -0,0 +1,30 @@ +# Pillar: Security (SEC1-SEC19) + +Focus: Protect containers, control access, enforce least privilege, and ensure private connectivity. Grade each check ✓ (pass) / ✗ (fail) / N/A with evidence, severity, and a recommendation. + +Best-practice anchor: [ECS Best Practices — Security](https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/security.html) · [Task IAM roles](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) + +> Access-limitation protocol and the "minimum baseline" rule are defined in [`../checks.md`](../checks.md). For SEC1/SEC2 inspect both managed and inline policies; for SEC6 enumerate every env var key across all containers. + +| Check ID | Check Name | Applies To | AWS API Call(s) | What to Look For | Severity if Failed | Recommendation | +|----------|------------|------------|----------------|------------------|-------------------|----------------| +| SEC1 | Task execution role scoped | All | `iam.listAttachedRolePolicies` + `iam.listRolePolicies` + `iam.getRolePolicy` (execution role) | No overly permissive managed policies (AdministratorAccess, PowerUserAccess); no wildcard actions/resources in inline policies; scope limited to ECR pull, CloudWatch Logs, and Secrets Manager access | High | Scope execution role to ECR pull, CloudWatch Logs, and Secrets Manager access only. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html | +| SEC2 | Task role least privilege | All | `iam.listAttachedRolePolicies` + `iam.listRolePolicies` + `iam.getRolePolicy` (task role) | No wildcard actions/resources, no AdministratorAccess | High | Create specific IAM policies scoped to required AWS services. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html | +| SEC3 | awsvpc network mode | All | `ecs.describeTaskDefinition` | networkMode is "awsvpc" | Medium | Use awsvpc network mode for task-level network isolation with security groups. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking-awsvpc.html | +| SEC4 | No privileged containers | EC2 | `ecs.describeTaskDefinition` | No container has privileged: true | Critical | Remove privileged flag — use specific Linux capabilities instead. See: https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/security-runtime.html | +| SEC5 | Read-only root filesystem | All | `ecs.describeTaskDefinition` | Essential containers have readonlyRootFilesystem: true | Medium | Enable read-only root filesystem, use tmpfs or EFS for writable paths. See: https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/security-runtime.html | +| SEC6 | No secrets in environment variables | All | `ecs.describeTaskDefinition` | No container environment variables contain keys matching: PASSWORD, API_KEY, ACCESS_KEY, SECRET_KEY, PRIVATE_KEY, ENCRYPTION_KEY, AUTH_TOKEN, API_TOKEN, SESSION_TOKEN, CLIENT_SECRET, DB_SECRET, APP_SECRET (case-insensitive). Flag all env var keys containing SECRET or TOKEN as substrings, regardless of value — false positives in a security check are preferable to missed secrets. **Deep inspection**: iterate ALL containers (essential and non-essential) and enumerate every environment variable key | High | Use ECS secrets (Secrets Manager or SSM Parameter Store) instead of plaintext env vars. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data.html | +| SEC7 | Secrets from Secrets Manager/SSM | All | `ecs.describeTaskDefinition` | If secrets needed, containers use `secrets` field referencing Secrets Manager or SSM | Medium | Migrate plaintext secrets to Secrets Manager or SSM Parameter Store. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data.html | +| SEC8 | Non-root user | All | `ecs.describeTaskDefinition` | Containers specify `user` field with non-root user (not "0" or "root") | Medium | Configure containers to run as non-root user for defense in depth. See: https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/security-runtime.html | +| SEC9 | Public IP disabled | Fargate | `ecs.describeServices` | networkConfiguration.awsvpcConfiguration.assignPublicIp is "DISABLED" | High | Disable public IP assignment, use NAT Gateway or VPC endpoints for outbound access. See: https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/networking-connecting-vpc.html | +| SEC10 | ECR image scanning enabled | All | `ecr.describeRepositories` | imageScanningConfiguration.scanOnPush is true for the ECR repository hosting the task image | High | Enable ECR image scanning on push to detect vulnerabilities in container images. See: https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-scanning.html | +| SEC11 | ECR tag immutability | All | `ecr.describeRepositories` | imageTagMutability is "IMMUTABLE" for production repositories | Medium | Enable image tag immutability to prevent image tags from being overwritten. See: https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-tag-mutability.html | +| SEC12 | Security group least privilege | All (awsvpc) | `ec2.describeSecurityGroups` | Security groups attached to the service/tasks do not have 0.0.0.0/0 ingress on all ports; ingress rules are scoped to required ports and sources only | High | Restrict security group ingress to only required ports and known source CIDR ranges or security groups. See: https://docs.aws.amazon.com/vpc/latest/userguide/security-group-rules.html | +| SEC13 | VPC endpoints for ECR | Fargate (private subnet) | `ec2.describeVpcEndpoints` | VPC endpoints exist for `com.amazonaws.{region}.ecr.dkr` (interface), `com.amazonaws.{region}.ecr.api` (interface), and `com.amazonaws.{region}.s3` (gateway) if tasks are in private subnets without NAT | Medium | Configure VPC endpoints for ECR to allow private image pulls without NAT Gateway. All three endpoints (ecr.dkr, ecr.api, s3) are required — ECR stores image layers in S3. See: https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html | +| SEC14 | VPC endpoints for CloudWatch Logs | Fargate (private subnet) | `ec2.describeVpcEndpoints` | VPC endpoint exists for `com.amazonaws.{region}.logs` if tasks are in private subnets | Medium | Configure VPC endpoint for CloudWatch Logs to enable log delivery from private subnets. See: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch-logs-and-interface-VPC.html | +| SEC15 | VPC endpoints for Secrets Manager | All (if secrets used) | `ec2.describeVpcEndpoints` | VPC endpoint exists for `com.amazonaws.{region}.secretsmanager` if task uses Secrets Manager secrets in private subnet | Low | Configure VPC endpoint for Secrets Manager for secure secret retrieval without internet access. See: https://docs.aws.amazon.com/secretsmanager/latest/userguide/vpc-endpoint-overview.html | +| SEC16 | NAT Gateway or VPC endpoints for internet access | Fargate (private subnet) | `ec2.describeRouteTables` + `ec2.describeNatGateways` | Private subnets have route to NAT Gateway or relevant VPC endpoints for outbound connectivity | Medium | Ensure private subnets have NAT Gateway or VPC endpoints for required AWS service access. See: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html | +| SEC17 | Encryption at rest for task storage | All | `ecs.describeTaskDefinition` + `ec2.describeVolumes` | Task-attached EBS volumes have Encrypted=true (KMS); EC2 container instance EBS volumes encrypted. Fargate ephemeral storage is encrypted by default on platform 1.4.0+ (AES-256) — mark ✓ for Fargate with no attached EBS unless a customer-managed key is required | Medium | Encrypt task-attached EBS volumes and container instance volumes with KMS; Fargate ephemeral storage is encrypted by default on platform 1.4.0+. See: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-task-storage.html | +| SEC18 | VPC Flow Logs enabled | All (awsvpc) | `ec2.describeFlowLogs` | VPC Flow Logs enabled for the VPC hosting the service ENIs (uses VPC ID from `ec2.describeSubnets`) | Medium | Enable VPC Flow Logs for network traffic visibility, security monitoring, and compliance auditing. See: https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html | +| SEC19 | GuardDuty ECS Runtime Monitoring | All | `guardduty.listDetectors` + `guardduty.getDetector` | GuardDuty enabled in the service's region with the Runtime Monitoring feature ENABLED for ECS (Fargate and/or EC2). Mark N/A if GuardDuty is not available in the region | Medium | Enable GuardDuty Runtime Monitoring for ECS to detect runtime threats (crypto mining, reverse shells, suspicious network activity). See: https://docs.aws.amazon.com/guardduty/latest/ug/runtime-monitoring.html | +| SEC20 | Encryption in transit (TLS) | All | `elbv2.describeListeners` (LB in front of the service) + `ecs.describeServices` (`serviceConnectConfiguration`) | Inbound and service-to-service traffic is encrypted: any ALB/NLB listener fronting the service uses an HTTPS/TLS protocol (not plaintext HTTP/TCP), and, where Service Connect is used, its `tls` block is configured. Mark **N/A** only for an internal task with no load balancer and no Service Connect (no in-transit path to assess) | High | Enforce TLS end-to-end: use an HTTPS (ALB) / TLS (NLB) listener with an ACM certificate, redirect HTTP→HTTPS, and enable Service Connect TLS for service-to-service traffic. Satisfies the common-check baseline **CS2**. See: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html | diff --git a/skills/aws-ecs-operations-review/references/report-format.md b/skills/aws-ecs-operations-review/references/report-format.md new file mode 100644 index 0000000..ce2dbb2 --- /dev/null +++ b/skills/aws-ecs-operations-review/references/report-format.md @@ -0,0 +1,115 @@ +# Report format — the ECS operations review artifact + +The review produces one written artifact **per ECS service** (Markdown by default; render DOCX only if the user/orchestrator asks — build it from the same content, don't drop sections). It must show, for **every graded check** across all 6 pillars: the ✓ / ✗ / N/A state, the observed current state (evidence), and — for every ✗ — a detailed recommendation tagged Critical / High / Medium / Low with a link to the authoritative AWS source. This file defines the required structure. Do not abbreviate it away; a scorecard without current-state evidence and linked recommendations is incomplete. + +## Contents + +- Severity model +- Required artifact structure (§1–§7) +- Finding block format (§4) +- Coverage gate (do not skip) +- Customer-facing rules +- Evidence discipline +- Output mechanics + +## Severity model + +Every ✗ finding (and any Medium+ N/A worth flagging) carries one of four descriptive tiers. Use the check's own severity as the baseline, escalated/lowered by the blast radius observed on this service. + +| Tier | Meaning | +|------|---------| +| 🔴 **Critical** | Active exposure or imminent outage/data-loss risk; fix now (e.g. privileged containers, plaintext secrets, public exposure). | +| 🟠 **High** | Serious gap likely to cause an incident (e.g. desiredCount=1 with no auto scaling on a prod service, no circuit breaker, over-permissive IAM). | +| 🟡 **Medium** | Best-practice deviation degrading efficiency/operability/defence-in-depth. | +| 🔵 **Low** | Optimization / hygiene; do when convenient. | + +Never use internal severity numbers (Sev 1-5). Always say in the recommendation *why* you set the tier. + +## Required artifact structure + +``` +# ECS Operations Review — {service name} +**Cluster:** {cluster} **Account / Region:** {…} **Compute platform:** {Fargate | Fargate + Spot | EC2 (ASG capacity provider) | EC2 (launchType-only) | Managed Instances | mixed — list all} +**Pillars graded:** Resiliency & HA · Observability · Security · Operations · Performance · Additional Analysis +**Review run:** {UTC} + +## 1. Executive summary +- 2–4 sentences: overall posture, biggest risks, headline numbers. +- **Findings by severity:** 🔴 {n} Critical · 🟠 {n} High · 🟡 {n} Medium · 🔵 {n} Low +- **Checks:** ✓ {n} pass · ✗ {n} fail · N/A {n} (of {total}) +- One-line posture statement per pillar (all 6). + +## 2. Workload details (current state) +- Service config: desiredCount / runningCount, compute platform + capacity provider strategy (providers, base/weight, managed scaling targetCapacity where applicable), platform version, deployment controller + config, network mode, subnets/AZs, load balancer. +- Task definition: CPU/memory, container count, roles, log config. +- 7-day baseline metrics (CPU, memory, task count) — or a note if unavailable. + +## 3. Prioritized action plan +All ✗ findings across all 6 pillars, ordered Critical → Low. Each row links to its detailed finding in §4. + +| # | Severity | Pillar | Finding | Affected | Effort | +|---|----------|--------|---------|----------|--------| + +## 4. Detailed findings +One block per ✗ (and per notable N/A), grouped by pillar, ordered by severity. Use the finding block format below. + +## 5. Pillar scorecards +One table per pillar — ALL 6 — every check (pass, fail, N/A) with observed current state, so the reader sees full coverage. + +| Check | Result | Severity | Current state (evidence) | +|-------|--------|----------|--------------------------| + +## 6. Recommended CloudWatch alarms +The alarm table from `alarm-thresholds.md` (base ECS alarms + any Container Insights / LB / relative alarms that apply), marking which already exist (from `cloudwatch.describeAlarms`) vs are missing. Mandatory for IDR onboarding. + +## 7. Access limitations / what was not assessed +Every check marked N/A due to access denial or missing data, with the API that failed and the manual follow-up. +``` + +## Finding block format (§4) + +Every ✗ gets a block. Recommendations must be detailed and actionable — what to change, why it matters, and concrete steps/snippet — not a one-liner. Every block ends with at least one authoritative AWS link. + +``` +### [🟠 High] No Application Auto Scaling configured +**Pillar:** Resiliency & HA +**Current state:** No scalable target registered for service `svc` (from +`applicationautoscaling.describeScalableTargets`); desiredCount=1. +**Why this severity:** production service cannot absorb load spikes or replace lost +tasks beyond a single instance. +**Impact:** capacity events / task loss cause degradation or outage. +**Recommendation:** +1. Register a scalable target and a target-tracking policy on CPU or ALB + request count. Example: {snippet} +2. Set minCapacity ≥ 2 for HA. +**References:** +- [ECS Service Auto Scaling](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html) +``` + +Pull each ✗ block's why/steps/snippet from the pillar file's recommendation and the doc link there; re-resolve the doc URL at report time via `aws___search_documentation` — never paste a URL you haven't confirmed resolves. + +## Coverage gate — verify ALL of this before finalizing (a review that misses any is incomplete) + +- **Every check ID** in all 6 pillar files appears in a §5 scorecard as ✓ / ✗ / N/A — **including passes**. Grade internally against the exact check IDs (REL/OBS/SEC/OPS/PERF/ADD); do **not** merge, invent, or drop checks. +- **All 6 pillars** have a scorecard table — never omit or truncate a pillar. Launch-type-specific checks that don't apply are N/A (Fargate-only N/A on EC2 and vice versa), not dropped. +- **7-day baseline metrics** (§2) collected via `cloudwatch.getMetricStatistics`, or the limitation noted. +- **Recommended CloudWatch alarms** table (§6) present, marking exist-vs-missing from `cloudwatch.describeAlarms`. +- **Every ✗** has a detailed §4 finding block with evidence, impact, remediation, and a resolvable AWS link. +- Anything unobtainable is **N/A with the real reason** (the failed API) — never silently omitted, never a guessed ✓/✗. + +## Customer-facing rules + +- Strip internal check IDs (REL1, SEC4, …) from the customer-facing report — present check names/descriptions. Grade with the IDs internally for the coverage gate. +- Use the four descriptive severity tiers, never internal Sev numbers. +- No internal tool names/aliases in the deliverable. + +## Evidence discipline + +- Every ✗ current-state quotes the actual AWS API value (field, count, ARN) — never a generic restatement. +- Never fabricate a finding, count, or link. If a check couldn't be evaluated, it's N/A in §5 and listed in §7 — not a guessed pass/fail. + +## Output mechanics + +- Default output is a per-service artifact / Markdown file (e.g. `ecs-review-{service}-{date}.md`). +- Render DOCX only when asked (e.g. UOPS/IDR DOCX deliverable) — build from the same content with `python-docx`, dropping no sections. +- The executive summary (§1), prioritized action plan (§3), all 6 pillar scorecards (§5), and the alarms table (§6) are mandatory.